Compare Qwen3.5 GPU test against the real official checkpoint#723
Conversation
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
67b849a to
6304ba9
Compare
|
I'm seeing this qwen 3.5 test as GPU version of So I think it makes sense to do the GPU test with a real model instead of a tiny test config model |
6304ba9 to
267b7f9
Compare
267b7f9 to
15b88e4
Compare
The Qwen3.5 GPU comparison test previously built a synthetic random-weight Qwen3_5TextConfig. Restore a real-checkpoint comparison: load the official Qwen/Qwen3.5-0.8B weights and tokenizer and compare logits on a real prompt. To keep the GPU job from downloading the gated checkpoint in-job (which can exceed the task timeout), the job mounts Weka and points HF_HOME at a checkpoint cache there, so from_pretrained reads from the local mount. Local runs fall back to a normal download, skipping the test if the checkpoint can't be obtained. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
15b88e4 to
b5dacd8
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e331cc5681
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| --env 'GOOGLE_APPLICATION_CREDENTIALS=/.google/credentials.json' \ | ||
| --env 'TOKENIZERS_PARALLELISM=false' \ | ||
| --env 'CUBLAS_WORKSPACE_CONFIG=:16:8' \ | ||
| --env 'HF_HOME=/weka/oe-training-default/test-fixtures/hf' \ |
There was a problem hiding this comment.
Force cache-only loading in GPU CI
When this job runs with HF_HOME pointed at the Weka cache, from_pretrained() is still allowed to contact the Hub and download any missing or newer snapshot files. In the CI scenario this change is trying to protect (--task-timeout 10m and a ~GB Qwen checkpoint), a cache miss or model update can spend the whole Gantry task downloading instead of failing/skipping quickly. Set the Beaker task offline here (for example HF_HUB_OFFLINE=1) or pass local_files_only=True when using the Weka cache so the test only reads the mounted checkpoint in CI while local runs can still download normally.
Useful? React with 👍 / 👎.
e331cc5 to
de645e0
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: de645e0b7c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| hf_model = _load_pretrained( | ||
| transformers.Qwen3_5ForCausalLM, | ||
| torch_dtype=torch.float32, | ||
| attn_implementation="eager", | ||
| ) |
There was a problem hiding this comment.
Load the text-only Qwen3.5 weights
When this runs against the official Qwen/Qwen3.5-0.8B cache, the repo is a multimodal checkpoint: its text weights are stored under model.language_model.*, while the concrete Qwen3_5ForCausalLM class instantiates a text-only model whose keys are model.*. Loading this class directly from the multimodal repo therefore either gets turned into a skip by _load_pretrained or leaves the text model initialized without the official weights, so the new CI comparison no longer validates the real checkpoint. Use the auto text-only path or explicitly load/remap the text_config/model.language_model weights before comparing.
Useful? React with 👍 / 👎.
The Qwen3.5 GPU comparison test previously built a synthetic random-weight
Qwen3_5TextConfig. This restores a real-checkpoint comparison: it loads the officialQwen/Qwen3.5-0.8Bweights and tokenizer and compares logits on a real prompt.The test just calls
from_pretrained("Qwen/Qwen3.5-0.8B")as-is (skipping if the checkpoint can't be obtained). To keep the GPU job from downloading the gated checkpoint in-job (which can exceed the 10-minute task timeout), the gantry run mounts Weka (oe-training-default) and pointsHF_HOMEat a checkpoint cache there, so the weights are read from the local mount.Required data (already done) populate the cache once from a Weka-mounted machine (with
HF_TOKEN):🤖 Generated with Claude Code