fix: make local workflow eval portable and Creator-compatible#230
fix: make local workflow eval portable and Creator-compatible#230mouyase wants to merge 8 commits intorpamis:masterrpamis/comet:masterfrom mouyase:fix/eval-python-only-identitymouyase/comet:fix/eval-python-only-identityCopy head branch name to clipboard
Conversation
Reviewer's GuideEnsures all local eval task Docker images install the Claude Code CLI used by the local runner, and adds a regression test to enforce this, along with a version/changelog bump. Flow diagram for enforcing Claude Code installation in local task Docker imagesflowchart LR
TaskDockerfile[Task Dockerfile]
ClaudeInstallStep[RUN npm install -g @anthropic-ai/claude-code]
RegressionTest[test_validation_scripts_py]
TaskImage[Built task Docker image]
LocalEvaluator[Local evaluator]
ClaudeRunner[Claude runner inside image]
ClaudeCodeCLI[Claude Code CLI]
TaskDockerfile --> ClaudeInstallStep
ClaudeInstallStep --> TaskImage
RegressionTest --> TaskDockerfile
RegressionTest -->|requires ClaudeInstallStep| TaskDockerfile
TaskImage --> LocalEvaluator
LocalEvaluator -->|starts| ClaudeRunner
ClaudeRunner -->|uses| ClaudeCodeCLI
File-Level Changes
Assessment against linked issues
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughLocal evaluation images install Claude Code and run as a non-root user. Validator scaffolding supports scripts launched from ChangesLocal evaluation runtime and workflow updates
Estimated code review effort: 3 (Moderate) | ~30 minutes 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Hey - I've found 1 issue, and left some high level feedback:
- The new
test_claude_eval_task_images_install_claude_coderelies on a simple text search; consider tightening it (e.g., matching the actual install command or a specific label) so a commented-out line or unrelated mention of@anthropic-ai/claude-codedoesn’t cause false positives. - You duplicated the apt/node/npm + Claude Code install sequence across four Dockerfiles; factoring this into a common base image or shared include would make future changes to the install process easier to maintain.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The new `test_claude_eval_task_images_install_claude_code` relies on a simple text search; consider tightening it (e.g., matching the actual install command or a specific label) so a commented-out line or unrelated mention of `@anthropic-ai/claude-code` doesn’t cause false positives.
- You duplicated the apt/node/npm + Claude Code install sequence across four Dockerfiles; factoring this into a common base image or shared include would make future changes to the install process easier to maintain.
## Individual Comments
### Comment 1
<location path="eval/local/tasks/authoring-skill-smoke/environment/Dockerfile" line_range="9-10" />
<code_context>
+ ca-certificates nodejs npm && \
+ rm -rf /var/lib/apt/lists/*
+
+RUN npm config set registry https://registry.npmmirror.com && \
+ npm install -g @anthropic-ai/claude-code@latest && \
+ npm config delete registry
</code_context>
<issue_to_address>
**suggestion (bug_risk):** Consider pinning the `@anthropic-ai/claude-code` version instead of using `@latest` for reproducible builds.
Using `@latest` allows rebuilds to silently pick up behavior or breaking changes from `@anthropic-ai/claude-code`. Pinning a specific version (or at least a major/minor range) keeps the image reproducible and safer to rebuild over time.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| RUN npm config set registry https://registry.npmmirror.com && \ | ||
| npm install -g @anthropic-ai/claude-code@latest && \ |
There was a problem hiding this comment.
suggestion (bug_risk): Consider pinning the @anthropic-ai/claude-code version instead of using @latest for reproducible builds.
Using @latest allows rebuilds to silently pick up behavior or breaking changes from @anthropic-ai/claude-code. Pinning a specific version (or at least a major/minor range) keeps the image reproducible and safer to rebuild over time.
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@eval/local/tasks/authoring-skill-smoke/environment/Dockerfile`:
- Around line 9-11: Replace the mutable `@latest` Claude Code install with one
exact reviewed version, preferably sourced from a shared environment or
build-time variable. Apply this change in
eval/local/tasks/authoring-skill-smoke/environment/Dockerfile lines 9-11,
eval/local/tasks/generic-skill-smoke/environment/Dockerfile lines 9-11,
eval/local/tasks/workflow-overlay-contract/environment/Dockerfile lines 11-13,
and eval/local/tasks/workflow-route-conformance/environment/Dockerfile lines
9-11, preserving the existing registry setup and cleanup.
In `@eval/local/tests/tasks/test_validation_scripts.py`:
- Around line 183-191: Update test_claude_eval_task_images_install_claude_code
to require at least one discovered Dockerfile, then validate each file for a
non-comment RUN instruction that executes npm install -g
`@anthropic-ai/claude-code` with any permitted version or arguments. Do not accept
package-name text appearing only in comments or unrelated Dockerfile content,
and preserve reporting of missing Dockerfile paths.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 224397d2-d8b6-4b41-abf6-556b9bc586cb
📒 Files selected for processing (7)
CHANGELOG.mdeval/local/tasks/authoring-skill-smoke/environment/Dockerfileeval/local/tasks/generic-skill-smoke/environment/Dockerfileeval/local/tasks/workflow-overlay-contract/environment/Dockerfileeval/local/tasks/workflow-route-conformance/environment/Dockerfileeval/local/tests/tasks/test_validation_scripts.pypackage.json
| def test_claude_eval_task_images_install_claude_code(): | ||
| task_root = ROOT / "local/tasks" | ||
| missing = [] | ||
| for dockerfile in task_root.glob("*/environment/Dockerfile"): | ||
| text = dockerfile.read_text(encoding="utf-8").lower() | ||
| if "@anthropic-ai/claude-code" not in text: | ||
| missing.append(str(dockerfile.relative_to(ROOT))) | ||
|
|
||
| assert missing == [] |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Assert an actual installation command and reject an empty Dockerfile set.
This check only searches for a package-name string, so a comment or unrelated Dockerfile text can satisfy it without installing Claude Code. It also passes when glob() finds no files. Match a non-comment RUN npm install -g @anthropic-ai/claude-code... instruction and assert that at least one Dockerfile was discovered.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@eval/local/tests/tasks/test_validation_scripts.py` around lines 183 - 191,
Update test_claude_eval_task_images_install_claude_code to require at least one
discovered Dockerfile, then validate each file for a non-comment RUN instruction
that executes npm install -g `@anthropic-ai/claude-code` with any permitted
version or arguments. Do not accept package-name text appearing only in comments
or unrelated Dockerfile content, and preserve reporting of missing Dockerfile
paths.
0ba0c4f to
384aec0
Compare
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
eval/local/tests/scaffold/test_profiles.py (1)
902-924: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winCover the remaining structured-path variants.
This only exercises one literal
pathsentry. Add cases forpath, multiple required paths, and a wildcard so the newly supported branches cannot regress unnoticed.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@eval/local/tests/scaffold/test_profiles.py` around lines 902 - 924, Expand test_generic_rubric_scores_structured_expected_artifact_paths to cover structured artifact expectations using a singular path, multiple required paths, and a wildcard path. Create the corresponding fixture files and assert each variant passes through the artifact_presence rubric, while retaining the existing literal paths coverage.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@eval/local/tasks/authoring-skill-smoke/environment/Dockerfile`:
- Around line 9-11: Remove the registry.npmmirror.com override before global npm
installs and use the official npm registry or an approved authenticated internal
proxy with provenance controls. Apply this to
eval/local/tasks/authoring-skill-smoke/environment/Dockerfile lines 9-11,
eval/local/tasks/generic-skill-smoke/environment/Dockerfile lines 9-11,
eval/local/tasks/workflow-overlay-contract/environment/Dockerfile lines 11-13,
eval/local/tasks/workflow-route-conformance/environment/Dockerfile lines 9-11,
and the corresponding Dockerfiles under
eval/local/tasks/comet-observability-env-template/* lines 7,
comet-noise-distractor/* lines 7, comet-persistence-threading/* lines 7, and
comet-cross-file-refactor/* lines 7; retain the global installs without an
unapproved public mirror.
In `@eval/local/tests/tasks/test_validation_scripts.py`:
- Around line 193-194: Update the validation logic that populates root_images to
determine the effective USER directive in the final Dockerfile stage, rather
than accepting any earlier non-root USER. Reject images whose final runtime user
is root or UID 0, including Dockerfiles that switch back to root after setting a
non-root user.
In `@eval/scaffold/python/validation/generic_rubric.py`:
- Around line 68-71: Update _path_exists to reject absolute paths and paths
containing parent traversal, then resolve test_dir and ensure every literal or
glob match resolves beneath it. Filter out matches whose resolved paths escape
the resolved task directory, including symlink targets, while preserving the
existing boolean result for valid in-directory matches.
---
Nitpick comments:
In `@eval/local/tests/scaffold/test_profiles.py`:
- Around line 902-924: Expand
test_generic_rubric_scores_structured_expected_artifact_paths to cover
structured artifact expectations using a singular path, multiple required paths,
and a wildcard path. Create the corresponding fixture files and assert each
variant passes through the artifact_presence rubric, while retaining the
existing literal paths coverage.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 00649426-fae2-4490-aaf6-ae764517cf55
📒 Files selected for processing (10)
CHANGELOG.mdeval/local/tasks/authoring-skill-smoke/environment/Dockerfileeval/local/tasks/generic-skill-smoke/environment/Dockerfileeval/local/tasks/workflow-overlay-contract/environment/Dockerfileeval/local/tasks/workflow-route-conformance/environment/Dockerfileeval/local/tests/scaffold/test_profiles.pyeval/local/tests/scaffold/test_utils.pyeval/local/tests/tasks/test_validation_scripts.pyeval/scaffold/python/utils.pyeval/scaffold/python/validation/generic_rubric.py
🚧 Files skipped from review as they are similar to previous changes (1)
- CHANGELOG.md
There was a problem hiding this comment.
🧹 Nitpick comments (2)
eval/local/tests/tasks/test_tasks.py (1)
471-471: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd regression coverage for the changed failure gate.
The new test verifies treatment classification, but not that
test_task_treatmentactually tolerates failed validations forCONTROLwhile still failing dynamic treatments. Add a focused test around this branch to prevent accidental broadening or removal of the exception.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@eval/local/tests/tasks/test_tasks.py` at line 471, Add focused regression coverage for the failure gate in test_task_treatment: verify failed validation is tolerated for CONTROL treatments, while failed validation still causes dynamic treatments to fail. Reuse the existing treatment-classification fixtures and _is_observational_baseline_run behavior, keeping the test scoped to this branch.test/domains/factory/factory-package.test.ts (1)
739-757: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winStrengthen
sourceSummariesassertion to cover all mapped fields.The assertion only checks
name/description;query,platform,scope,origin, andreferencesfrom the new mapping aren't verified, so a regression there would slip through.♻️ Suggested strengthening
expect(resolvedSkills.sourceSummaries).toEqual([ expect.objectContaining({ name: 'research-skill', description: 'Collects focused research notes.', + query: 'research-skill', + platform: 'codex', + scope: 'project', + origin: 'project', + references: [], }), ]);🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/domains/factory/factory-package.test.ts` around lines 739 - 757, Strengthen the resolvedSkills.sourceSummaries assertion in the factory package test to validate every mapped field: name, description, query, platform, scope, origin, and references. Keep the existing expected entry and add the corresponding values from the fixture or generated output so regressions in any mapping field fail the test.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@eval/local/tests/tasks/test_tasks.py`:
- Line 471: Add focused regression coverage for the failure gate in
test_task_treatment: verify failed validation is tolerated for CONTROL
treatments, while failed validation still causes dynamic treatments to fail.
Reuse the existing treatment-classification fixtures and
_is_observational_baseline_run behavior, keeping the test scoped to this branch.
In `@test/domains/factory/factory-package.test.ts`:
- Around line 739-757: Strengthen the resolvedSkills.sourceSummaries assertion
in the factory package test to validate every mapped field: name, description,
query, platform, scope, origin, and references. Keep the existing expected entry
and add the corresponding values from the fixture or generated output so
regressions in any mapping field fail the test.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 080874e0-814b-481c-9da7-15e6e91609a2
📒 Files selected for processing (7)
domains/factory/package.tseval/local/tests/scaffold/test_native_aligned_eval.pyeval/local/tests/scaffold/test_profiles.pyeval/local/tests/tasks/test_tasks.pyeval/scaffold/python/native_eval.pyeval/scaffold/python/validation/authoring_rubric.pytest/domains/factory/factory-package.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- eval/local/tests/scaffold/test_profiles.py
|
请先修复以下必要问题:
|
|
已在 38ccf40 修正并推送:
验证: |
|
已在 70d9c06 推送 portable evidence 后续修复: • 项目内 Creator authoring 路径落盘为相对引用,读取时再 hydration 为项目内运行路径。 PR 描述已同步。 |
Summary
Fixes #229.
This PR fixes the full local workflow-evaluation path used by Creator-generated bundles. The original failure was that local task images did not contain the Claude runner required by the evaluator's execution-identity contract. Follow-up diagnosis found additional contract drift between Creator output, the local evaluator, and platform distribution readiness.
What changed
agentuser.scaffoldruntime reliably.sourceSummariesfrom resolved Skill sources;Passed: yes.review output;generic-skill-smokefrom workflow-kernel defaults; its genericresult.mdexpectation is unrelated to Issue → worktree → PR workflows.CONTROLtreatment failures in the report as observational baseline data without using them as eval gates../references and hydrate them only for runtime consumers; discard external eval-report paths rather than recording host-specific locations in repository evidence.Validation
Upstream focused checks
pnpm exec vitest run test/domains/factory/factory-package.test.ts test/domains/bundle/bundle-command.test.ts test/domains/bundle/bundle-eval.test.ts test/domains/bundle/bundle-review-summary.test.ts test/domains/bundle/bundle-publish.test.ts—84 passedpnpm exec vitest run test/domains/bundle/bundle-authoring.test.ts test/domains/bundle/bundle-eval.test.ts—28 passedcd eval && uv run pytest local/tests/scaffold/test_profiles.py local/tests/scaffold/test_native_aligned_eval.py local/tests/tasks/test_validation_scripts.py local/tests/scaffold/test_utils.py::test_copied_scaffold_is_importable_by_validator_script -q—56 passedpnpm buildpnpm lintpnpm format:checkEnd-to-end Docker eval
Using the generated
fix-from-issuesworkflow against the configured Claude provider:authoring-skill-smoke/DYNAMIC_SKILL:18/18checks passedworkflow-route-conformance/DYNAMIC_SKILL:17/17checks passedThe report still includes intentionally failing
CONTROLrows because they run without the Skill package. They are retained as baseline observations and are not gating failures.