Do not treat an unreachable Hub as a missing model in 16bit merges#950
Do not treat an unreachable Hub as a missing model in 16bit merges#950danielhanchen wants to merge 1 commit intomainunslothai/unsloth-zoo:mainfrom fix-silent-noop-merge-exportunslothai/unsloth-zoo:fix-silent-noop-merge-exportCopy head branch name to clipboard
Conversation
`check_hf_model_exists` wrapped its `HfFileSystem(...).ls(...)` in a bare `except:` returning False, which mapped a 429, a 5xx, a DNS or proxy failure, a read timeout and `HF_HUB_OFFLINE` all onto "this repo does not exist". `determine_base_model_source` then fell through every priority and returned `(None, ...)`, and `merge_and_overwrite_lora` answered a bare `warnings.warn` plus `return None`. The user visible result was that `save_pretrained_merged` returned None and created no output directory at all, with the only signal a UserWarning that scrolls past in a notebook. The user believed the 16bit export had succeeded, and the cost was a whole training run. Telling the two cases apart is possible because HfFileSystem already separates them. `hf_file_system._raise_file_not_found`, which produces a plain FileNotFoundError, is reached only after `_repo_and_revision_exists` catches RepositoryNotFoundError, RevisionNotFoundError or HFValidationError, while every transport failure propagates out of `ls` untouched. Confirmed empirically against huggingface_hub 1.24.0: a missing repo raises FileNotFoundError, and HF_HUB_OFFLINE raises OfflineModeIsEnabled. Narrow the except to genuinely absent or inaccessible repos and re-raise everything else as a RuntimeError that names connectivity or rate limiting as the cause. Turn the silent `return None` in the merge into a RuntimeError that states plainly that nothing was written. This is a behaviour change: a caller that currently swallows the warning now sees an exception. That is the point. A None return that writes no files is never useful to a caller, and staying silent here costs a training run.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 96ed9577b5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| except Exception as e: | ||
| raise RuntimeError( |
There was a problem hiding this comment.
Skip the Hub probe when the base model is local
When model_name is a valid local directory such as base or ./base, HfFileSystem.ls can raise ValueError/HfUriError because the path is not a Hub repo ID, and this new handler turns that into a RuntimeError. determine_base_model_source calls this function before check_local_model_exists (lines 4699–4700), so the merge never reaches its documented local-model priorities; absolute local paths are likewise blocked whenever the Hub is offline or rate-limited. Check for the local source before probing the Hub, or allow this exception to propagate only after local resolution fails.
Useful? React with 👍 / 👎.
What breaks
save_pretrained_merged(..., save_method = "merged_16bit")can returnNone, create no output directory at all, and report the problem only through aUserWarning:In a notebook that scrolls past. The user believes the 16bit export succeeded, and finds out later that nothing was written. The trigger is not a missing model, it is a Hub that was briefly unreachable: a 429, a 5xx, a DNS or proxy failure, a read timeout, or
HF_HUB_OFFLINE.Root cause
unsloth_zoo/saving_utils.py:check_hf_model_existsused a bareexcept::That maps every possible failure onto "this repo does not exist". From there:
determine_base_model_source(saving_utils.py:4697) calls it, getsFalse, falls through all five priorities and returns(None, False, "", False, None).merge_and_overwrite_lora(saving_utils.py:2925) seesfinal_model_name is Noneand answerswarnings.warn(...)plusreturn None.No directory is created, no exception is raised, and the return value is
None.Why the two cases can be told apart
HfFileSystemalready separates them, which is what makes a narrow fix possible rather than a guess.hf_file_system._raise_file_not_found(which raises a plainFileNotFoundError) is reached only after_repo_and_revision_existscatchesRepositoryNotFoundError,RevisionNotFoundErrororHFValidationError. Every other failure propagates out oflsuntouched.Confirmed empirically against huggingface_hub 1.24.0:
HfFileSystem.lsraisesbuiltins.FileNotFoundError: ... (repository not found)HF_HUB_OFFLINE=1huggingface_hub.errors.OfflineModeIsEnabledWorth flagging because it is not the obvious answer: a genuinely missing repo surfaces as a plain
FileNotFoundError, not asRepositoryNotFoundError. fsspec converts it. SoFileNotFoundErrorhas to be in the "absent" set, and including it does not swallow transport errors, because transport errors never reach that conversion.The fix
Narrow the except to genuinely absent or inaccessible repos, and re-raise everything else as a
RuntimeErrorthat names the real cause:and turn the silent
return Noneinmerge_and_overwrite_lorainto aRuntimeErrorthat states plainly that nothing was written.huggingface_hub>=0.34.0is already a hard dependency, so all four error classes are guaranteed to import.Behaviour change, stated explicitly
This turns a warning into an exception, and that is a behaviour change. Callers that currently swallow the
UserWarningwill now see aRuntimeErrorwhere they previously sawNone.The argument for doing it anyway: a
Nonereturn that writes no files is never useful to a caller. There is no recovery a caller can perform with it that it could not perform with an exception, and the current behaviour converts an infrastructure blip into a silently lost training run. Failing loudly is strictly more informative here.The honest cost: any code path that calls
save_pretrained_mergedinside atry/exceptthat only expected aNonereturn, or that checks the return value rather than catching, will now raise instead. A script that looped over many models and tolerated individual failures by testing forNonewill now stop at the first one. That is a real migration cost for a small number of callers, and it is the reason this is a separate PR rather than folded into something else.Note also that a gated repo still returns
Falserather than raising. That preserves today's behaviour for gated models, where falling through to a local copy is usually what the user wants.Measured before and after
New regression test, run against the same tree with and without the fix:
The 7 that pass both ways are the controls: the absent-repo cases that must keep returning
False, and the success paths.Existing suites, same tree with and without the fix, identical either way:
Tests added
tests/test_check_hf_model_exists_transport_errors.py. Every test monkeypatchesHfFileSystem.ls, so nothing touches the network.OfflineModeIsEnabled) must raiseRuntimeError, name connectivity or rate limiting, and keep the original exception as__cause__.False, since that single step is what makes the export silently no-op.FileNotFoundError,RepositoryNotFoundError,RevisionNotFoundError,GatedRepoError) must still returnFalse.True, absent ->False).determine_base_model_sourcepropagates the transport error instead of producing the(None, ...)that caused the silent no-op, and still returns(None, ...)for a genuinely absent repo with no local copy.