ENG-4782: prime train models — show FFT models by default#807
ENG-4782: prime train models — show FFT models by default#807JannikSt wants to merge 8 commits intomainPrimeIntellect-ai/prime:mainfrom feature/eng-4782-prime-train-fft-modelsPrimeIntellect-ai/prime:feature/eng-4782-prime-train-fft-modelsCopy head branch name to clipboard
Conversation
- adds AvailableFFTModel / list_available_fft_models to HostedTrainingClient - prime train models fetches /training/available-fft-models by default and renders a second Rich table when the endpoint returns any results - JSON output gains available_fft_models when non-empty (backwards compatible) - --fft-only flag suppresses the LoRA section - graceful fallback on 401/403/404 so older backends still render LoRA - picks up an incidental uv.lock re-sort from pre-commit uv-run
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5b6eef3831
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| except (NotFoundError, UnauthorizedError): | ||
| return [] |
There was a problem hiding this comment.
Preserve auth errors for FFT-only model lists
When prime train models --fft-only is used, the command skips the LoRA /rft/models call, so this unconditional 401 swallow turns an invalid/expired API key into an empty FFT list and exits successfully with “No FFT models available.” That defeats the caller-side fft_only re-raise path and can make scripts treat an auth failure as a valid empty result; let auth errors propagate in the FFT-only path or only apply this fallback when the LoRA section is being rendered.
Useful? React with 👍 / 👎.
- stop swallowing UnauthorizedError in list_available_fft_models so prime train models --fft-only surfaces auth failures instead of a misleading 'No FFT models available' (flagged by cursor + codex) - only NotFoundError is caught now; the command layer still handles the default-mode 'don't crash the LoRA table' fallback - drop cache_synced_at from the FFT pydantic model + Rich table + JSON output (per user feedback, freshness is an implementation detail the picker already gates on)
|
@codex review |
|
Codex Review: Didn't find any major issues. Chef's kiss. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
- rename LoRA table title from 'Hosted Training - Models' to 'Hosted Training — LoRA' (em-dash matches the inference command's 'Prime Inference — Models' style, and the label is more accurate now that FFT is a second section) - rename FFT title to 'Hosted Training — Full Finetuning' for consistency - left-justify title + caption on both tables so the two sections stack cleanly instead of floating with centered text - add a blank line between the LoRA and FFT tables so the two sections don't run into each other - rename 'Available On' column to 'Cluster(s)' to match the 'GPU Type(s)' pluralization pattern
Cluster names ('research', etc.) aren't actionable for users
dispatching FFT runs — the picker takes gpu_type, not cluster
name. Hiding the column tightens the table and removes noise
that surprised users in the review pass.
JSON output still carries cluster_id/cluster_name so scripts
that need cluster-level fan-out can still get it.
Bugbot caught: AvailableFFTModelsResponse.model_validate raises pydantic.ValidationError on a schema-drifted backend payload. The command layer catches APIError only, so a bad FFT response would have killed the LoRA output too. Wrap model_validate and re-raise as APIError so the command's existing fallback swallows it in default mode (LoRA still renders) and surfaces it with --fft-only (matching the auth-error behavior).
|
@codex review |
|
Codex Review: Didn't find any major issues. Hooray! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Scoped strictly to the lines this PR added — preexisting typing.Dict / typing.List / typing.Optional usages in the same files (create_run, delete_run, build_payload_from_toml, older tests, etc.) are left untouched to avoid entangling the FFT diff with a repo-wide cleanup.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 7e01e99. Configure here.
Bugbot caught: LoRA-empty + FFT-populated printed 'No models available for Hosted Training' above the FFT table, which reads as a global failure even though dispatchable FFT models are present. Now only show the banner when both sections are empty.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b5ae42f8e9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if output == "json": | ||
| output_data_as_json({"models": [m.model_dump() for m in models]}, console) | ||
| payload: dict[str, Any] = {"models": [m.model_dump() for m in models]} | ||
| if fft_models: | ||
| payload["available_fft_models"] = [m.model_dump() for m in fft_models] |
There was a problem hiding this comment.
Include the FFT key for empty --fft-only JSON output
When --fft-only --output json is used and the FFT endpoint returns no models (including the intentionally swallowed 404 case), this condition omits available_fft_models and emits only {"models": []} even though the LoRA request was suppressed. Scripts using the flag specifically to consume .available_fft_models[] therefore receive a missing/null field on a successful empty result; include available_fft_models: [] whenever fft_only is set.
Useful? React with 👍 / 👎.
Codex caught: --fft-only --output json with no FFT models emitted
{"models": []}, so scripts consuming .available_fft_models[]
would see a missing key on a successful empty result. With
--fft-only we now:
- drop the misleading models key (LoRA wasn't fetched)
- always include available_fft_models (empty allowed)
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep it up! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Extends
prime train modelsso it also surfaces models pre-cached on FFT dispatch clusters. LoRA output is unchanged; the FFT table only appears when the new/training/available-fft-modelsendpoint returns results.list_available_fft_modelsonHostedTrainingClient+AvailableFFTModel/FFTModelClusterInfoschemas--output jsongainsavailable_fft_modelswhen non-empty (backwards compatible; old keymodelsuntouched)--fft-onlyflag suppresses the LoRA section for scripts that only care about FFTLinear: ENG-4782
Note
Low Risk
CLI presentation and optional API client behavior only; LoRA paths stay unchanged with explicit backward-compatible JSON and 404 handling.
Overview
prime train modelsnow lists full finetuning (FFT) models that are warm on dispatch clusters, in addition to the existing LoRA catalog.A new
HostedTrainingClient.list_available_fft_modelscall hits/training/available-fft-models, with Pydantic types that omit backend-only fields likecache_synced_at. 404 is treated as no FFT data for backward compatibility; parse failures becomeAPIErrorso the command can fall back without breaking LoRA output.The CLI shows a second table (Hosted Training — Full Finetuning) with model names and collapsed GPU types, refactors LoRA rendering into helpers, and adds
--fft-onlyfor scripts. JSON keeps the existingmodelskey; addsavailable_fft_modelsonly when non-empty (always present under--fft-only). FFT fetch errors are best-effort in default mode but surface when--fft-onlyis set.Reviewed by Cursor Bugbot for commit e0ac729. Bugbot is set up for automated code reviews on this repo. Configure here.