[megatron] support glm4 dense #7177
Conversation
Summary of ChangesHello @Jintao-Huang, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request integrates support for GLM-4 dense models into the Megatron framework. It introduces specialized transformer layer configurations, including custom attention and MLP blocks with unique post-layer normalization, to accurately represent the GLM-4 architecture. The changes also update the project's documentation to list the newly supported GLM-4 variants and include a new test to verify their proper alignment and functionality. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
There was a problem hiding this comment.
Code Review
This pull request adds support for GLM-4 dense models in Megatron. The changes include adding the necessary model type constants, registering the new model, and implementing the GLM-4 specific architecture with custom self-attention and MLP layers. The documentation and tests are also updated accordingly. My review focuses on the new implementation in swift/megatron/model/gpt/glm4.py, where I've identified a few issues related to a misnamed function, an unused variable, and a potentially harmful monkey patch. I've provided suggestions to correct these for better code quality and robustness.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request adds support for dense GLM-4 models to the Megatron framework. The changes include defining the GLM-4 model architecture, updating model constants, modifying the HuggingFace config conversion logic, and adding a new test case. The documentation has also been updated to reflect the new model support. Overall, the changes are well-structured and seem correct. However, I've identified a significant issue in swift/megatron/model/gpt/glm4.py where a global monkey patch is applied. This is a risky practice that can lead to side effects and should be removed, especially since the correct local configuration seems to be already in place. Please see my specific comment for details.
No description provided.