ci(showcase): fetch Git LFS in every workflow that serves integration assets#6160
Draft
jpr5 wants to merge 1 commit into
mainCopilotKit/CopilotKit:mainfrom
fix/showcase-demo-files-lfs-fleetwideCopilotKit/CopilotKit:fix/showcase-demo-files-lfs-fleetwideCopy head branch name to clipboard
Draft
ci(showcase): fetch Git LFS in every workflow that serves integration assets#6160jpr5 wants to merge 1 commit intomainCopilotKit/CopilotKit:mainfrom fix/showcase-demo-files-lfs-fleetwideCopilotKit/CopilotKit:fix/showcase-demo-files-lfs-fleetwideCopy head branch name to clipboard
jpr5 wants to merge 1 commit into
mainCopilotKit/CopilotKit:mainfrom
fix/showcase-demo-files-lfs-fleetwideCopilotKit/CopilotKit:fix/showcase-demo-files-lfs-fleetwideCopy head branch name to clipboard
Conversation
Contributor
Production Digest-Pinning AuditAll 41 services digest-pinned. Run 2026-07-24 16:08:58 PDT — 0 finding(s). |
… assets Demo assets stay uniformly LFS-tracked. The build is the single place this gets resolved, so every workflow that checks out and then serves or bakes integration `public/` assets now fetches LFS objects. `showcase_build.yml` (the workflow that builds and pushes the deployed images) has hardcoded `lfs: true` since 7bde1ee, so the deploy path was already correct. Two paths were not: - `showcase_build_check.yml` read `lfs: ${{ matrix.service.lfs }}`, which was `false` for every integration, so the pre-merge build check baked pointer stubs into the images it built. Build-only, so nothing deployed, but it meant PR CI could never catch a broken demo asset. - `test_e2e-showcase-on-demand.yml` passed no `lfs` input at all. That job runs `next dev` on the integration package and drives it with Playwright, serving `public/` straight from the working tree, so `multimodal.spec.ts` would click "Try with sample image", receive a ~130-byte pointer stub as `image/png`, and fail on the magic-byte guard — with nothing wrong in the code under test. Nine integrations ship that spec. Both now use a uniform `lfs: true`. Also deletes the per-slot `"lfs"` field from both `ALL_SERVICES` definitions. After the above there were no consumers left: `showcase_build.yml` ignored it in favour of a hardcoded `true`, and `showcase_build_check.yml` was its only reader. Leaving a dead flag that reads as authoritative is how this was mis-set in the first place, and deleting it means a new integration cannot be added with LFS fetching silently off — there is no flag to forget. No asset or .gitattributes changes: LFS is used as intended and a committed pointer remains the correct on-disk state for an LFS-tracked file.
jpr5
force-pushed
the
fix/showcase-demo-files-lfs-fleetwide
branch
from
July 24, 2026 23:15
db776cb to
b36d3b9
Compare
Contributor
|
Size Change: 0 B Total Size: 1.19 MB ℹ️ View Unchanged
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Correction first: the deploy path was already fixed
This PR was opened to migrate ~12 integrations' demo assets out of LFS. That approach was dropped — the assets stay uniformly LFS-tracked and the build is the single place this gets resolved.
While establishing scope, one premise turned out to be false and it matters for how this gets prioritised:
showcase_build.ymldoes not read the per-slot"lfs"flag.ALL_SERVICESdoes carry"lfs":falsefor every integration, which reads as though integration images ship pointer stubs. They do not. The image-buildingbuildjob (lines 418–604, containing thedepot/build-push-actionthat pushes the deployed image) hardcodeslfs: true, added 2026-06-07 in7bde1eef3aa— "ci(showcase): pull Git LFS assets during build so demo images ship real binaries" — for exactly this bug. The only reader of the flag wasshowcase_build_check.yml.Evidence, from a real CI run on
main(30116731237, job89559665519,build (built-in-agent, …), 2026-07-24) — notebuilt-in-agentis"lfs":falseinALL_SERVICES:And confirmed end-to-end against what prod and staging actually serve — all 10 pointer-committed integrations, both assets:
So: zero integration images ship stubs, and there are no broken multimodal cells from this cause. The catalogued blast radius is not expanded. The
agno/mastrafailures areHTTP 404from files being absent, a different root cause.Two consequences worth flagging:
agno/mastra's assets — as plain LFS-tracked files, no overrides —showcase_build.ymlalready fetches them, so their images get the real bytes on the next build.What this PR actually changes
Two paths genuinely did not fetch LFS:
showcase_build.ymllfs: true(hardcoded)showcase_build_check.ymllfs: ${{ matrix.service.lfs }}→ falselfs: truetest_e2e-showcase-on-demand.ymllfsinputlfs: trueshowcase_capture-previews.ymllfs: truetest_showcase-frontend-matrix.ymllfs: trueshowcase_validate.ymltest_e2e-showcase-on-demand.ymlwas the meaningful gap. It runsnpx next devon the integration package and drives it with Playwright, servingpublic/straight from the working tree — so with no LFS fetch,multimodal.spec.tsclicks "Try with sample image", gets a ~130-byte pointer stub served asimage/png, and fails the magic-byte guard insample-attachment-buttons.tsxwith nothing wrong in the code under test. Nine of the ten affected integrations ship that spec (all butms-agent-harness-dotnet).Making it not recur
The per-slot
"lfs"field is deleted from bothALL_SERVICESdefinitions. After the change above it had no consumers —showcase_build.ymlignored it in favour of a hardcodedtrue, andshowcase_build_check.ymlwas its only reader.Deleting it rather than setting it to
trueis deliberate: a dead flag that reads as authoritative is how this got mis-set in the first place, and it is what made the state so hard to reason about here. With no flag, integration #21 cannot be added with LFS fetching silently off — there is nothing to set and nothing to forget. Both JSON blobs were re-parsed after the edit (28 and 26 services, zerolfskeys).Residual risk, stated plainly: every fetch site is now a literal
lfs: true, so correctness rests on a new workflow that serves integration assets remembering to include it. That is a smaller and more visible surface than a per-service flag, but it is not structurally enforced. A CI guard asserting that any workflow checking outshowcase/integrations/*passeslfs: truewould close it; not in this PR.Verified affected list — 10 integrations, not 12
Pointer-committed
public/demo-files/sample.{png,pdf}onmain, established from the committed tree (local LFS smudge makes the working tree lie —ls -lshows 10083 B for a 130 B pointer):ag2,built-in-agent,claude-sdk-python,claude-sdk-typescript,crewai-crews,langroid,llamaindex,ms-agent-harness-dotnet,pydantic-ai,spring-aiSizes via the LFS-unaware
git/treesAPI:sample.png130 B,sample.pdf129 B (real: 10083 B / 2486 B). Note the contents API is LFS-aware and reports the resolved 10083 B for pointers too, so it cannot be used to tell them apart.The "12" figure conflated two root causes: these 10 have pointer stubs (correct LFS state, harmless), while
agno/mastrahad the files absent (HTTP 404, genuinely broken, #6157).Already carrying per-integration overrides, untouched here:
google-adk,langgraph-python,langgraph-fastapi,langgraph-typescript,strands,strands-typescript,ms-agent-dotnet,ms-agent-python. Consolidating those onto the uniform LFS convention is a separate follow-up.Bandwidth cost
Negligible for the assets themselves —
sample.png10083 B +sample.pdf2486 B ≈ 12.3 KB per integration.But
actions/checkoutwithlfs: truefetches every LFS object in the tree, not just demo files: 203 objects, ~495 MB (largest are multi-MB GIFs underexamples/andassets/, up to 65 MB). That is the real cost driver, andshowcase_build.ymlhas already been paying it for all ~20 integration builds since 2026-06-07 — this PR adds it to the build-check and on-demand-e2e paths. Worth a follow-up look at whether those two need a scopedgit lfs pull --includeinstead of a blanket fetch; flagging rather than solving it here, since it is a pre-existing cost profile.Proof
Local Docker builds were deliberately NOT used as evidence. A local build smudges LFS from the local checkout and passes regardless of what CI does, so it proves nothing. All evidence above is CI-side or from live prod/staging:
git lfs fetchin the deploy build job (quoted above).git/treesAPI.actionlinton all three edited workflows: 24 findings before, 24 after — none introduced (all pre-existing shellcheck info/style plus the customdepot-*runner label). YAML re-parsed and bothALL_SERVICESblobs re-validated.CI-side before/after for the
build_checkpathSame integration (
crewai-crews, one of the ten pointer-committed ones), from real CI logs:BEFORE — run
30132714792, job89610715113(branch predating this change):No
git lfs install, noFetching LFS objectsgroup. The image was built from a pointer stub.AFTER — run
30133303667, job89612372095(this PR):Build Check on this PR: 26 build jobs, 25 success, 0 failure (one still finishing at time of writing).
Prod + staging multimodal cell state
Read from PocketBase (
showcase-pocketbase-production/pocketbase-staging-eec0), not from dashboard chip colours.e2e:*/multimodald5:*/multimodalRed
d5cells:agno,built-in-agent¹,crewai-crews,llamaindex,mastra,ms-agent-python(¹ prod only — green on staging, i.e. flapping).None of these are asset failures. Every one reports the same signature:
Not
is a Git LFS pointer, not the real asset, and notCould not fetch sample … HTTP 404. A scan of 500 prod status records found zero occurrences of any LFS-pointer,sample.png/sample.pdffetch, or magic-byte error.The correlation is absent in both directions:
ag2,claude-sdk-python,claude-sdk-typescript,langroid,ms-agent-harness-dotnet,pydantic-ai,spring-aims-agent-python(has its own override, real 10083 B / 2486 B assets)google-adk,langgraph-*,strands*,ms-agent-dotnetSo pointer-committed status does not predict cell state, which is what you'd expect once the deploy path fetches LFS.
Reported separately and loudly: four of those reds —
built-in-agent,crewai-crews,llamaindex,ms-agent-python— are long-standing genuine failures withfail_countbetween 1713 and 2008 on prod, unrelated to this PR and unrelated toagno/mastra's missing files. They share thewaitForTurnComplete … dom-missingturn-timeout signature and need their own investigation. Staging shows the same set atfail_count=61.