Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

fix: make local workflow eval portable and Creator-compatible#230

Open
mouyase wants to merge 8 commits into
rpamis:masterrpamis/comet:masterfrom
mouyase:fix/eval-python-only-identitymouyase/comet:fix/eval-python-only-identityCopy head branch name to clipboard
Open

fix: make local workflow eval portable and Creator-compatible#230
mouyase wants to merge 8 commits into
rpamis:masterrpamis/comet:masterfrom
mouyase:fix/eval-python-only-identitymouyase/comet:fix/eval-python-only-identityCopy head branch name to clipboard

Conversation

@mouyase

@mouyase mouyase commented Jul 23, 2026

Copy link
Copy Markdown

Summary

Fixes #229.

This PR fixes the full local workflow-evaluation path used by Creator-generated bundles. The original failure was that local task images did not contain the Claude runner required by the evaluator's execution-identity contract. Follow-up diagnosis found additional contract drift between Creator output, the local evaluator, and platform distribution readiness.

What changed

  • Install Claude Code in the local task images used by the evaluator and run them as a non-root agent user.
  • Make copied validator tasks import the bundled scaffold runtime reliably.
  • Align Creator-generated workflow evidence with the authoring evaluator:
    • emit real sourceSummaries from resolved Skill sources;
    • use the canonical six authoring lanes and accept legacy lane encoding while reading;
    • recognize the Creator's Passed: yes. review output;
    • explicitly reference both workflow protocol and resolved Skill evidence from the entry Skill.
  • Remove generic-skill-smoke from workflow-kernel defaults; its generic result.md expectation is unrelated to Issue → worktree → PR workflows.
  • Keep no-Skill CONTROL treatment failures in the report as observational baseline data without using them as eval gates.
  • Keep Creator-time authoring subagent guidance portable. Generated Bundles no longer declare a required Claude custom agent, so Codex readiness is not blocked by an unused platform-specific runtime capability.
  • Persist project-internal Creator authoring paths as portable ./ references and hydrate them only for runtime consumers; discard external eval-report paths rather than recording host-specific locations in repository evidence.

Validation

Upstream focused checks

  • pnpm exec vitest run test/domains/factory/factory-package.test.ts test/domains/bundle/bundle-command.test.ts test/domains/bundle/bundle-eval.test.ts test/domains/bundle/bundle-review-summary.test.ts test/domains/bundle/bundle-publish.test.ts84 passed
  • pnpm exec vitest run test/domains/bundle/bundle-authoring.test.ts test/domains/bundle/bundle-eval.test.ts28 passed
  • cd eval && uv run pytest local/tests/scaffold/test_profiles.py local/tests/scaffold/test_native_aligned_eval.py local/tests/tasks/test_validation_scripts.py local/tests/scaffold/test_utils.py::test_copied_scaffold_is_importable_by_validator_script -q56 passed
  • pnpm build
  • pnpm lint
  • pnpm format:check

End-to-end Docker eval

Using the generated fix-from-issues workflow against the configured Claude provider:

  • authoring-skill-smoke / DYNAMIC_SKILL: 18/18 checks passed
  • workflow-route-conformance / DYNAMIC_SKILL: 17/17 checks passed
  • Comet recorded the quick eval evidence as passed for the current generated Bundle hash.

The report still includes intentionally failing CONTROL rows because they run without the Skill package. They are retained as baseline observations and are not gating failures.

@sourcery-ai

sourcery-ai Bot commented Jul 23, 2026

Copy link
Copy Markdown

Reviewer's Guide

Ensures all local eval task Docker images install the Claude Code CLI used by the local runner, and adds a regression test to enforce this, along with a version/changelog bump.

Flow diagram for enforcing Claude Code installation in local task Docker images

flowchart LR
    TaskDockerfile[Task Dockerfile]
    ClaudeInstallStep[RUN npm install -g @anthropic-ai/claude-code]
    RegressionTest[test_validation_scripts_py]
    TaskImage[Built task Docker image]
    LocalEvaluator[Local evaluator]
    ClaudeRunner[Claude runner inside image]
    ClaudeCodeCLI[Claude Code CLI]

    TaskDockerfile --> ClaudeInstallStep
    ClaudeInstallStep --> TaskImage
    RegressionTest --> TaskDockerfile
    RegressionTest -->|requires ClaudeInstallStep| TaskDockerfile
    TaskImage --> LocalEvaluator
    LocalEvaluator -->|starts| ClaudeRunner
    ClaudeRunner -->|uses| ClaudeCodeCLI
Loading

File-Level Changes

Change Details Files
Add regression test that validates every local task Dockerfile installs Claude Code.
  • Introduce a test that scans all local task Dockerfiles under local/tasks for the @anthropic-ai/claude-code package string.
  • Fail the test with a list of offending Dockerfiles if any image does not declare Claude Code installation.
eval/local/tests/tasks/test_validation_scripts.py
Install Claude Code in all Python-only local eval task Docker images that previously lacked it.
  • Extend Python-based task Dockerfiles to install Node.js and npm via apt-get with minimal recommended packages.
  • Configure npm to use the npmmirror registry during installation, install @anthropic-ai/claude-code globally at latest, and then remove the custom registry setting.
  • Align installation pattern across authoring-skill-smoke, generic-skill-smoke, workflow-route-conformance, and workflow-overlay-contract task images.
eval/local/tasks/authoring-skill-smoke/environment/Dockerfile
eval/local/tasks/generic-skill-smoke/environment/Dockerfile
eval/local/tasks/workflow-route-conformance/environment/Dockerfile
eval/local/tasks/workflow-overlay-contract/environment/Dockerfile
Document the fix and bump the package version.
  • Add a 0.4.0-beta.9 changelog entry describing the Claude Code installation fix for local eval task images.
  • Increment package.json version from 0.4.0-beta.8 to 0.4.0-beta.9.
CHANGELOG.md
package.json

Assessment against linked issues

Issue Objective Addressed Explanation
#229 Ensure Python-only smoke/contract task images (generic-skill-smoke, authoring-skill-smoke, workflow-route-conformance, etc.) are evaluable by resolving the mandatory Claude CLI execution-identity failure (e.g., by making Claude optional or installing it in the task images).
#229 Add regression coverage to prevent recurrence of this issue, ensuring that local eval task images participating in the Claude execution-identity contract are configured appropriately (e.g., have Claude Code installed or are explicitly handled).

Possibly linked issues


Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@coderabbitai

coderabbitai Bot commented Jul 23, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Local evaluation images install Claude Code and run as a non-root user. Validator scaffolding supports scripts launched from validation/, generic rubrics accept structured artifact paths, workflow package contracts and baseline handling are updated, regression tests cover these behaviors, and the release advances to v0.4.0-beta.9.

Changes

Local evaluation runtime and workflow updates

Layer / File(s) Summary
Install Claude Code in task images
eval/local/tasks/*/environment/Dockerfile
Task images install Node.js/npm and CA certificates, globally install Claude Code using a temporary npm mirror, and configure non-root agent execution.
Make copied validators importable
eval/scaffold/python/utils.py, eval/local/tests/scaffold/test_utils.py
The scaffold runtime is copied into both the test root and validation/, with coverage for executing a validator script from that directory.
Score structured expected artifacts
eval/scaffold/python/validation/generic_rubric.py, eval/local/tests/scaffold/test_profiles.py
Artifact scoring accepts string or dictionary declarations, multiple candidate paths, and wildcard paths, with coverage for structured workflow artifacts.
Generate and validate workflow evidence
domains/factory/package.ts, domains/bundle/*.ts, eval/scaffold/python/validation/authoring_rubric.py, test/domains/factory/factory-package.test.ts, test/domains/bundle/bundle-command.test.ts
Generated workflow packages include reference and resolved-source metadata, use updated authoring lanes and overlay tasks, remove Claude-specific agent resources, and validate the revised evidence contract.
Handle observational baseline treatments
eval/scaffold/python/native_eval.py, eval/local/tests/tasks/test_tasks.py, eval/local/tests/scaffold/test_native_aligned_eval.py
Treatment classification is centralized, and validation failures no longer hard-fail observational baseline runs.
Validate and publish the contract
eval/local/tests/tasks/test_validation_scripts.py, CHANGELOG.md, package.json
Validation checks Claude Code installation and non-root users across task Dockerfiles, while release metadata records v0.4.0-beta.9.

Estimated code review effort: 3 (Moderate) | ~30 minutes

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Out of Scope Changes check ⚠️ Warning Several changes touch bundle generation and authoring/rubric validation, which are unrelated to fixing Claude Code in local task images. Limit the PR to local task image Dockerfiles, the regression test, and the release metadata tied to #229.
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The PR installs Claude Code in the affected local task images and adds regression coverage for the installation.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title matches the PR’s main goal: making local eval task images portable by installing Claude Code and updating related validation.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 1 issue, and left some high level feedback:

  • The new test_claude_eval_task_images_install_claude_code relies on a simple text search; consider tightening it (e.g., matching the actual install command or a specific label) so a commented-out line or unrelated mention of @anthropic-ai/claude-code doesn’t cause false positives.
  • You duplicated the apt/node/npm + Claude Code install sequence across four Dockerfiles; factoring this into a common base image or shared include would make future changes to the install process easier to maintain.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- The new `test_claude_eval_task_images_install_claude_code` relies on a simple text search; consider tightening it (e.g., matching the actual install command or a specific label) so a commented-out line or unrelated mention of `@anthropic-ai/claude-code` doesn’t cause false positives.
- You duplicated the apt/node/npm + Claude Code install sequence across four Dockerfiles; factoring this into a common base image or shared include would make future changes to the install process easier to maintain.

## Individual Comments

### Comment 1
<location path="eval/local/tasks/authoring-skill-smoke/environment/Dockerfile" line_range="9-10" />
<code_context>
+    ca-certificates nodejs npm && \
+    rm -rf /var/lib/apt/lists/*
+
+RUN npm config set registry https://registry.npmmirror.com && \
+    npm install -g @anthropic-ai/claude-code@latest && \
+    npm config delete registry
</code_context>
<issue_to_address>
**suggestion (bug_risk):** Consider pinning the `@anthropic-ai/claude-code` version instead of using `@latest` for reproducible builds.

Using `@latest` allows rebuilds to silently pick up behavior or breaking changes from `@anthropic-ai/claude-code`. Pinning a specific version (or at least a major/minor range) keeps the image reproducible and safer to rebuild over time.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment on lines +9 to +10
RUN npm config set registry https://registry.npmmirror.com && \
npm install -g @anthropic-ai/claude-code@latest && \

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (bug_risk): Consider pinning the @anthropic-ai/claude-code version instead of using @latest for reproducible builds.

Using @latest allows rebuilds to silently pick up behavior or breaking changes from @anthropic-ai/claude-code. Pinning a specific version (or at least a major/minor range) keeps the image reproducible and safer to rebuild over time.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@eval/local/tasks/authoring-skill-smoke/environment/Dockerfile`:
- Around line 9-11: Replace the mutable `@latest` Claude Code install with one
exact reviewed version, preferably sourced from a shared environment or
build-time variable. Apply this change in
eval/local/tasks/authoring-skill-smoke/environment/Dockerfile lines 9-11,
eval/local/tasks/generic-skill-smoke/environment/Dockerfile lines 9-11,
eval/local/tasks/workflow-overlay-contract/environment/Dockerfile lines 11-13,
and eval/local/tasks/workflow-route-conformance/environment/Dockerfile lines
9-11, preserving the existing registry setup and cleanup.

In `@eval/local/tests/tasks/test_validation_scripts.py`:
- Around line 183-191: Update test_claude_eval_task_images_install_claude_code
to require at least one discovered Dockerfile, then validate each file for a
non-comment RUN instruction that executes npm install -g
`@anthropic-ai/claude-code` with any permitted version or arguments. Do not accept
package-name text appearing only in comments or unrelated Dockerfile content,
and preserve reporting of missing Dockerfile paths.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 224397d2-d8b6-4b41-abf6-556b9bc586cb

📥 Commits

Reviewing files that changed from the base of the PR and between 7580fc0 and 0ba0c4f.

📒 Files selected for processing (7)
  • CHANGELOG.md
  • eval/local/tasks/authoring-skill-smoke/environment/Dockerfile
  • eval/local/tasks/generic-skill-smoke/environment/Dockerfile
  • eval/local/tasks/workflow-overlay-contract/environment/Dockerfile
  • eval/local/tasks/workflow-route-conformance/environment/Dockerfile
  • eval/local/tests/tasks/test_validation_scripts.py
  • package.json

Comment on lines +183 to +191
def test_claude_eval_task_images_install_claude_code():
task_root = ROOT / "local/tasks"
missing = []
for dockerfile in task_root.glob("*/environment/Dockerfile"):
text = dockerfile.read_text(encoding="utf-8").lower()
if "@anthropic-ai/claude-code" not in text:
missing.append(str(dockerfile.relative_to(ROOT)))

assert missing == []

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Assert an actual installation command and reject an empty Dockerfile set.

This check only searches for a package-name string, so a comment or unrelated Dockerfile text can satisfy it without installing Claude Code. It also passes when glob() finds no files. Match a non-comment RUN npm install -g @anthropic-ai/claude-code... instruction and assert that at least one Dockerfile was discovered.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@eval/local/tests/tasks/test_validation_scripts.py` around lines 183 - 191,
Update test_claude_eval_task_images_install_claude_code to require at least one
discovered Dockerfile, then validate each file for a non-comment RUN instruction
that executes npm install -g `@anthropic-ai/claude-code` with any permitted
version or arguments. Do not accept package-name text appearing only in comments
or unrelated Dockerfile content, and preserve reporting of missing Dockerfile
paths.

@mouyase
mouyase force-pushed the fix/eval-python-only-identity branch from 0ba0c4f to 384aec0 Compare July 24, 2026 01:35

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (1)
eval/local/tests/scaffold/test_profiles.py (1)

902-924: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Cover the remaining structured-path variants.

This only exercises one literal paths entry. Add cases for path, multiple required paths, and a wildcard so the newly supported branches cannot regress unnoticed.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@eval/local/tests/scaffold/test_profiles.py` around lines 902 - 924, Expand
test_generic_rubric_scores_structured_expected_artifact_paths to cover
structured artifact expectations using a singular path, multiple required paths,
and a wildcard path. Create the corresponding fixture files and assert each
variant passes through the artifact_presence rubric, while retaining the
existing literal paths coverage.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@eval/local/tasks/authoring-skill-smoke/environment/Dockerfile`:
- Around line 9-11: Remove the registry.npmmirror.com override before global npm
installs and use the official npm registry or an approved authenticated internal
proxy with provenance controls. Apply this to
eval/local/tasks/authoring-skill-smoke/environment/Dockerfile lines 9-11,
eval/local/tasks/generic-skill-smoke/environment/Dockerfile lines 9-11,
eval/local/tasks/workflow-overlay-contract/environment/Dockerfile lines 11-13,
eval/local/tasks/workflow-route-conformance/environment/Dockerfile lines 9-11,
and the corresponding Dockerfiles under
eval/local/tasks/comet-observability-env-template/* lines 7,
comet-noise-distractor/* lines 7, comet-persistence-threading/* lines 7, and
comet-cross-file-refactor/* lines 7; retain the global installs without an
unapproved public mirror.

In `@eval/local/tests/tasks/test_validation_scripts.py`:
- Around line 193-194: Update the validation logic that populates root_images to
determine the effective USER directive in the final Dockerfile stage, rather
than accepting any earlier non-root USER. Reject images whose final runtime user
is root or UID 0, including Dockerfiles that switch back to root after setting a
non-root user.

In `@eval/scaffold/python/validation/generic_rubric.py`:
- Around line 68-71: Update _path_exists to reject absolute paths and paths
containing parent traversal, then resolve test_dir and ensure every literal or
glob match resolves beneath it. Filter out matches whose resolved paths escape
the resolved task directory, including symlink targets, while preserving the
existing boolean result for valid in-directory matches.

---

Nitpick comments:
In `@eval/local/tests/scaffold/test_profiles.py`:
- Around line 902-924: Expand
test_generic_rubric_scores_structured_expected_artifact_paths to cover
structured artifact expectations using a singular path, multiple required paths,
and a wildcard path. Create the corresponding fixture files and assert each
variant passes through the artifact_presence rubric, while retaining the
existing literal paths coverage.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 00649426-fae2-4490-aaf6-ae764517cf55

📥 Commits

Reviewing files that changed from the base of the PR and between 384aec0 and 7ce39c9.

📒 Files selected for processing (10)
  • CHANGELOG.md
  • eval/local/tasks/authoring-skill-smoke/environment/Dockerfile
  • eval/local/tasks/generic-skill-smoke/environment/Dockerfile
  • eval/local/tasks/workflow-overlay-contract/environment/Dockerfile
  • eval/local/tasks/workflow-route-conformance/environment/Dockerfile
  • eval/local/tests/scaffold/test_profiles.py
  • eval/local/tests/scaffold/test_utils.py
  • eval/local/tests/tasks/test_validation_scripts.py
  • eval/scaffold/python/utils.py
  • eval/scaffold/python/validation/generic_rubric.py
🚧 Files skipped from review as they are similar to previous changes (1)
  • CHANGELOG.md

Comment thread eval/local/tasks/authoring-skill-smoke/environment/Dockerfile Outdated
Comment thread eval/local/tests/tasks/test_validation_scripts.py Outdated
Comment thread eval/scaffold/python/validation/generic_rubric.py Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
eval/local/tests/tasks/test_tasks.py (1)

471-471: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add regression coverage for the changed failure gate.

The new test verifies treatment classification, but not that test_task_treatment actually tolerates failed validations for CONTROL while still failing dynamic treatments. Add a focused test around this branch to prevent accidental broadening or removal of the exception.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@eval/local/tests/tasks/test_tasks.py` at line 471, Add focused regression
coverage for the failure gate in test_task_treatment: verify failed validation
is tolerated for CONTROL treatments, while failed validation still causes
dynamic treatments to fail. Reuse the existing treatment-classification fixtures
and _is_observational_baseline_run behavior, keeping the test scoped to this
branch.
test/domains/factory/factory-package.test.ts (1)

739-757: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Strengthen sourceSummaries assertion to cover all mapped fields.

The assertion only checks name/description; query, platform, scope, origin, and references from the new mapping aren't verified, so a regression there would slip through.

♻️ Suggested strengthening
     expect(resolvedSkills.sourceSummaries).toEqual([
       expect.objectContaining({
         name: 'research-skill',
         description: 'Collects focused research notes.',
+        query: 'research-skill',
+        platform: 'codex',
+        scope: 'project',
+        origin: 'project',
+        references: [],
       }),
     ]);
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/domains/factory/factory-package.test.ts` around lines 739 - 757,
Strengthen the resolvedSkills.sourceSummaries assertion in the factory package
test to validate every mapped field: name, description, query, platform, scope,
origin, and references. Keep the existing expected entry and add the
corresponding values from the fixture or generated output so regressions in any
mapping field fail the test.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@eval/local/tests/tasks/test_tasks.py`:
- Line 471: Add focused regression coverage for the failure gate in
test_task_treatment: verify failed validation is tolerated for CONTROL
treatments, while failed validation still causes dynamic treatments to fail.
Reuse the existing treatment-classification fixtures and
_is_observational_baseline_run behavior, keeping the test scoped to this branch.

In `@test/domains/factory/factory-package.test.ts`:
- Around line 739-757: Strengthen the resolvedSkills.sourceSummaries assertion
in the factory package test to validate every mapped field: name, description,
query, platform, scope, origin, and references. Keep the existing expected entry
and add the corresponding values from the fixture or generated output so
regressions in any mapping field fail the test.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 080874e0-814b-481c-9da7-15e6e91609a2

📥 Commits

Reviewing files that changed from the base of the PR and between 7ce39c9 and 0710f20.

📒 Files selected for processing (7)
  • domains/factory/package.ts
  • eval/local/tests/scaffold/test_native_aligned_eval.py
  • eval/local/tests/scaffold/test_profiles.py
  • eval/local/tests/tasks/test_tasks.py
  • eval/scaffold/python/native_eval.py
  • eval/scaffold/python/validation/authoring_rubric.py
  • test/domains/factory/factory-package.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • eval/local/tests/scaffold/test_profiles.py

@mouyase mouyase changed the title fix(eval): install Claude Code in local task images fix(eval): make local workflow eval portable and Creator-compatible Jul 24, 2026
@benym benym changed the title fix(eval): make local workflow eval portable and Creator-compatible fix: make local workflow eval portable and Creator-compatible Jul 24, 2026
@benym

benym commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

请先修复以下必要问题:

  1. 同步 0.4.0-beta.9 版本信息
    package.json 已升级到 0.4.0-beta.9,但 package-lock.json、assets/manifest.json 和 test/app/cli-help.test.ts 仍是 0.4.0-beta.8,目前已导致 Quality and coverage、Node 22 compatibility 检查失败。

  2. 限制 structured artifact 的路径范围
    eval/scaffold/python/validation/generic_rubric.py 的 _path_exists() 当前允许绝对路径、../ 和指向 test_dir 外部的 symlink,可能让评估错误地把目录外已有文件计为预期产物。请确保 literal/glob 的最终解析结果只能位于 test_dir 内,并补充对应回归测试。

  3. 补全 Changelog
    当前 Changelog 没有记录本 PR 的一项重要用户可见变化:Creator 生成的 Bundle 不再要求 Claude 专属 runtime agent,而是使用可移植的 authoring brief,从而避免阻塞 Codex 分发 readiness。请在现有 0.4.0-beta.9 条目中补充这一最终行为

@mouyase

mouyase commented Jul 24, 2026

Copy link
Copy Markdown
Author

已在 38ccf40 修正并推送:

  1. package-lock.jsonassets/manifest.json 与 CLI 版本断言同步到 0.4.0-beta.9
  2. 为 generic rubric 增加绝对路径、../、literal symlink 与 glob symlink 越界回归测试;现有 _path_exists() 已在最终解析后限制于 test_dir 内;
  3. 0.4.0-beta.9 Changelog 中补充 Creator Bundle 使用可移植 authoring brief、避免 Claude-only capability 阻塞 Codex readiness 的行为说明。

验证:pnpm exec vitest run test/app/cli-help.test.ts、相关 pytest 回归均通过。

@mouyase

mouyase commented Jul 24, 2026

Copy link
Copy Markdown
Author

已在 70d9c06 推送 portable evidence 后续修复:

• 项目内 Creator authoring 路径落盘为相对引用,读取时再 hydration 为项目内运行路径。
• 外部 eval report 路径不再写入 repo evidence。
• 新增 state 与 eval 的回归测试;pnpm exec vitest run test/domains/bundle/bundle-authoring.test.ts test/domains/bundle/bundle-eval.test.ts 为 28/28,通过 pnpm format:check、pnpm lint 与 pnpm build。

PR 描述已同步。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

comet eval: local workflow eval fails on task-image and Creator contract drift

2 participants

Morty Proxy This is a proxified and sanitized view of the page, visit original site.