Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

[ckpt] fix: run converter_hf_to_mcore with --test will raise an AttributeError #2010

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jun 13, 2025
Merged

[ckpt] fix: run converter_hf_to_mcore with --test will raise an AttributeError #2010

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
90060af
[megatron] fix: replace ref_state_dict with ref_data
lixiaoguang12 Jun 13, 2025
3a1837b
try to enable test in converter
ETOgaosion Jun 13, 2025
0b3cf34
fix sharded_tensor shape
lixiaoguang12 Jun 13, 2025
03951d8
fix shardedTensor shape
lixiaoguang12 Jun 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions 4 .github/workflows/checkpoint_converter.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,11 +63,11 @@ jobs:
- name: Running Huggingface to Megatron dist_ckpt converter (Qwen/Qwen2.5-0.5B)
run: |
ray stop --force
python scripts/converter_hf_to_mcore.py --hf_model_path=${HOME}/models/Qwen/Qwen2.5-0.5B --output_path checkpoints/Qwen/Qwen2.5-0.5B
python scripts/converter_hf_to_mcore.py --hf_model_path=${HOME}/models/Qwen/Qwen2.5-0.5B --output_path checkpoints/Qwen/Qwen2.5-0.5B --test
- name: Running Huggingface to Megatron dist_ckpt converter (deepseek-ai/deepseek-coder-1.3b-instruct)
run: |
ray stop --force
python scripts/converter_hf_to_mcore.py --hf_model_path=${HOME}/models/deepseek-ai/deepseek-coder-1.3b-instruct --output_path checkpoints/deepseek-ai/deepseek-coder-1.3b-instruct
python scripts/converter_hf_to_mcore.py --hf_model_path=${HOME}/models/deepseek-ai/deepseek-coder-1.3b-instruct --output_path checkpoints/deepseek-ai/deepseek-coder-1.3b-instruct --test
- name: Clean up
run: |
rm -rf checkpoints
Expand Down
15 changes: 12 additions & 3 deletions 15 scripts/converter_hf_to_mcore.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
from megatron.core.dist_checkpointing.serialization import StrictHandling
from megatron.core.models.gpt.gpt_model import ModelType
from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
from megatron.core.dist_checkpointing.mapping import ShardedTensor
from transformers import AutoConfig

from verl.models.mcore import hf_to_mcore_config
Expand Down Expand Up @@ -74,8 +75,12 @@ def test_conversion(megatron_model_provider, tfconfig, output_path, model):
continue
dut_data = dut_state_dict[name].data
if name in ref_state_dict:
ref_data = ref_state_dict[name].data
assert dut_data.shape == ref_state_dict.shape, f"{name=} {dut_data.shape=} {ref_data.shape=}"
ref_data = ref_state_dict[name]
if isinstance(ref_data, ShardedTensor):
ref_data = ref_data.data.view(ref_data.local_shape)
else:
ref_data = ref_data.data
assert dut_data.shape == ref_data.shape, f"{name=} {dut_data.shape=} {ref_data.shape=}"
assert (dut_data == ref_data).all(), f"{name} is not equal"
print(f"{name} is equal")
else:
Expand All @@ -84,7 +89,11 @@ def test_conversion(megatron_model_provider, tfconfig, output_path, model):
if ref_state_dict[name] is None:
print(f"[Warning] {name} is none in ref_state_dict")
continue
ref_data = ref_state_dict[name].data
ref_data = ref_state_dict[name]
if isinstance(ref_data, ShardedTensor):
ref_data = ref_data.data.view(ref_data.local_shape)
else:
ref_data = ref_data.data
if name in dut_state_dict:
dut_data = dut_state_dict[name].data
assert dut_data.shape == ref_data.shape, f"{name=} {dut_data.shape=} {ref_data.shape=}"
Expand Down
Morty Proxy This is a proxified and sanitized view of the page, visit original site.