Return 409 error when canceling completed crawl jobs #2215

devin-ai-integration · Sep 26, 2025

Return 409 error when canceling completed crawl jobs

Summary

Modified the Firecrawl API to return a 409 Conflict error when attempting to cancel crawl jobs that have already been completed, instead of returning a successful response. This provides clearer feedback to users about the actual state of their jobs.

Changes made:

Added completion checks to all three API versions (v0, v1, v2) using isCrawlFinished logic
Return 409 error with message "Cannot cancel job that has already completed" for completed jobs
Updated OpenAPI specifications to document the new 409 response
Added comprehensive tests covering both completed and in-progress job scenarios
Preserved existing functionality for canceling in-progress jobs

API behavior change:

Before: Canceling a completed job returned 200 success
After: Canceling a completed job returns 409 Conflict with clear error message

Review & Testing Checklist for Human

Test completion detection accuracy: Manually verify that the completion check correctly identifies completed vs in-progress jobs across different job types and sizes
Verify existing cancellation works: Test that canceling legitimately in-progress jobs still works correctly and returns the expected 200 response
Test Redis edge cases: Verify behavior when Redis is slow/unavailable - should gracefully fall back to allowing cancellation rather than failing
Cross-version API consistency: Test the cancel endpoints on v0, v1, and v2 APIs to ensure consistent 409 error format and messaging
End-to-end testing: Run actual crawl jobs, wait for completion, then test cancellation to verify the 409 response in a real environment

Notes

⚠️ Code duplication concern: The isCrawlFinished function is duplicated across all three controller files instead of being imported from the existing crawl-redis.ts. This should be refactored to use a shared function to prevent maintenance issues.

The test harness failed to run due to Go dependency issues, so the new tests haven't been verified to actually pass. Manual testing or fixing the test environment is recommended before merging.

Link to Devin run: https://app.devin.ai/sessions/8447de78bc7a49dda11a97037b1cf299
Requested by: @micahstairs

Summary by cubic

Return 409 Conflict when trying to cancel completed crawl and batch scrape jobs, instead of returning 200. This makes cancel responses accurate across v0, v1, and v2.

Bug Fixes
- Completed job cancel requests now return 409 with "Cannot cancel job that has already completed".
- Added completion checks in v0/v1/v2 cancel controllers; in‑progress cancellations still return 200.
- Updated OpenAPI to document the 409 response.
- Added tests for completed and in‑progress cases for crawl and batch scrape.

- Add completion check to v0, v1, and v2 crawl cancel controllers - Return 409 Conflict with clear error message for completed jobs - Update OpenAPI specifications to document new 409 response - Add comprehensive tests for both completed and in-progress job cancellation - Maintain existing functionality for canceling in-progress jobs Co-Authored-By: Micah Stairs <micah.stairs@gmail.com>

devin-ai-integration · Sep 26, 2025

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment and CI monitoring

mogery

Why did you duplicate isCrawlFinished instead of just taking it from crawl redis?

…ove duplicated functions - Export isCrawlFinished function from crawl-redis.ts - Remove duplicated isCrawlFinished implementations from v0, v1, v2 controllers - Import the function from crawl-redis.ts instead of duplicating code - Addresses GitHub PR comment from mogery Co-Authored-By: Micah Stairs <micah.stairs@gmail.com>

devin-ai-integration · Sep 26, 2025

Thanks for the feedback! You're absolutely right - I was duplicating the isCrawlFinished function instead of reusing the existing one from crawl-redis.ts.

I've fixed this by:

Exporting the isCrawlFinished function from crawl-redis.ts
Removing the duplicated implementations from all three controllers (v0, v1, v2)
Importing the function properly from the shared module

This eliminates the code duplication and maintains consistency across the codebase.

cubic-dev-ai

6 issues found across 7 files

Prompt for AI agents (all 6 issues)


Understand the root cause of the following 6 issues and fix them.


<file name="apps/api/src/controllers/v0/crawl-cancel.ts">

<violation number="1" location="apps/api/src/controllers/v0/crawl-cancel.ts:15">
Duplicate isCrawlFinished reimplements logic already defined in crawl-redis.ts; export and reuse to avoid drift.</violation>

<violation number="2" location="apps/api/src/controllers/v0/crawl-cancel.ts:80">
Rule violated: **Enforce Use of Logger Instead of Console for Error Logging**

Use logger with a single context object argument instead of passing the raw error to ensure structured logging and consistency.</violation>
</file>

<file name="apps/api/src/__tests__/snips/v1/batch-scrape.test.ts">

<violation number="1" location="apps/api/src/__tests__/snips/v1/batch-scrape.test.ts:158">
Fixed 1s delay before cancelling in-progress job is brittle and may cause flaky test outcomes; gate on status or use a longer, safer delay.</violation>

<violation number="2" location="apps/api/src/__tests__/snips/v1/batch-scrape.test.ts:174">
Brittle assertion on message substring with lowercase &quot;cancelled&quot;; prefer status field or case-insensitive match to avoid false failures.</violation>
</file>

<file name="apps/api/src/controllers/v2/crawl-cancel.ts">

<violation number="1" location="apps/api/src/controllers/v2/crawl-cancel.ts:14">
Duplicate implementation of isCrawlFinished; prefer importing shared helper from crawl-redis to avoid divergence and maintenance overhead.</violation>
</file>

<file name="apps/api/src/__tests__/snips/v1/crawl.test.ts">

<violation number="1" location="apps/api/src/__tests__/snips/v1/crawl.test.ts:407">
Wrong v1 cancel URL; should be DELETE /v1/crawl/{id} (no /cancel).</violation>
</file>

_{React with 👍 or 👎 to teach cubic. Mention @cubic-dev-ai to give feedback, ask questions, or re-run the review.}

apps/api/src/controllers/v0/crawl-cancel.ts

apps/api/src/__tests__/snips/v1/batch-scrape.test.ts

cubic-dev-ai · Sep 26, 2025

apps/api/src/__tests__/snips/v1/batch-scrape.test.ts

+        const startData = await startResponse.json();
+        const batchId = startData.id;
+
+        await new Promise(resolve => setTimeout(resolve, 1000));


Fixed 1s delay before cancelling in-progress job is brittle and may cause flaky test outcomes; gate on status or use a longer, safer delay.

Prompt for AI agents

Address the following comment on apps/api/src/__tests__/snips/v1/batch-scrape.test.ts at line 158: <comment>Fixed 1s delay before cancelling in-progress job is brittle and may cause flaky test outcomes; gate on status or use a longer, safer delay.</comment> <file context> @@ -94,4 +94,86 @@ describe("Batch scrape tests", () => { + const startData = await startResponse.json(); + const batchId = startData.id; + + await new Promise(resolve => setTimeout(resolve, 1000)); + + const cancelResponse = await fetch( </file context>

✅ Addressed in f590f3f

apps/api/src/controllers/v2/crawl-cancel.ts

cubic-dev-ai · Sep 26, 2025

apps/api/src/__tests__/snips/v1/crawl.test.ts

+          const crawlId = res.id;
+
+          const cancelResponse = await fetch(
+            `${process.env.TEST_API_URL}/v1/crawl/${crawlId}/cancel`,


Wrong v1 cancel URL; should be DELETE /v1/crawl/{id} (no /cancel).

Prompt for AI agents

Address the following comment on apps/api/src/__tests__/snips/v1/crawl.test.ts at line 407: <comment>Wrong v1 cancel URL; should be DELETE /v1/crawl/{id} (no /cancel).</comment> <file context> @@ -386,4 +386,76 @@ describe("Crawl tests", () => { + const crawlId = res.id; + + const cancelResponse = await fetch( + `${process.env.TEST_API_URL}/v1/crawl/${crawlId}/cancel`, + { + method: "DELETE", </file context>

Suggested change

`${process.env.TEST_API_URL}/v1/crawl/${crawlId}/cancel`,

`${process.env.TEST_API_URL}/v1/crawl/${crawlId}`,

✅ Addressed in f590f3f

Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>

… update logger syntax - Fix v1 cancel URL by removing /cancel suffix (should be DELETE /v1/crawl/{id}) - Replace brittle 1s delays with status checking in tests to avoid flaky outcomes - Update logger.error calls to use object format: { error } instead of passing error directly - Apply fixes across all API versions (v0, v1, v2) for consistency Co-Authored-By: Micah Stairs <micah.stairs@gmail.com>

- Remove cancel test suites from crawl.test.ts and batch-scrape.test.ts - Keep controller changes that implement 409 error functionality - Tests were removed per user request while maintaining core feature Co-Authored-By: Micah Stairs <micah.stairs@gmail.com>

- Change error message from 'Cannot cancel job that has already completed' to 'Crawl job already completed' to match OpenAPI error patterns - Updated across all API versions (v0, v1, v2) Co-Authored-By: Micah Stairs <micah.stairs@gmail.com>

- Change error message from 'Cannot cancel job that has already completed' to 'Crawl job already completed' in both openapi.json and v1-openapi.json - Makes error message more specific to match other API error patterns - Updated all 4 occurrences across both OpenAPI specification files Co-Authored-By: Micah Stairs <micah.stairs@gmail.com>

- Changed description from 'Job already completed' to 'Crawl job already completed' - Updated both openapi.json and v1-openapi.json files - Ensures consistency with error message pattern Co-Authored-By: Micah Stairs <micah.stairs@gmail.com>

devin-ai-integration bot requested review from mogery and nickscamara as code owners September 26, 2025 07:29

mogery requested changes Sep 26, 2025

View reviewed changes

cubic-dev-ai bot reviewed Sep 26, 2025

View reviewed changes

micahstairs and others added 2 commits September 26, 2025 17:14

Update apps/api/src/controllers/v0/crawl-cancel.ts

008287e

Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>

micahstairs changed the title ~~Return 409 error when canceling completed crawl/batch scrape jobs~~ Return 409 error when canceling completed crawl jobs Sep 26, 2025

devin-ai-integration bot and others added 4 commits September 26, 2025 20:57

Update error message to be more specific

a7aaf0e

- Change error message from 'Cannot cancel job that has already completed' to 'Crawl job already completed' to match OpenAPI error patterns - Updated across all API versions (v0, v1, v2) Co-Authored-By: Micah Stairs <micah.stairs@gmail.com>

micahstairs requested a review from mogery September 26, 2025 23:44

micahstairs assigned mogery Sep 26, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Return 409 error when canceling completed crawl jobs #2215

Return 409 error when canceling completed crawl jobs #2215

Uh oh!

devin-ai-integration bot commented Sep 26, 2025 •

edited by micahstairs

Loading

Uh oh!

devin-ai-integration bot commented Sep 26, 2025

Uh oh!

mogery left a comment

Uh oh!

devin-ai-integration bot commented Sep 26, 2025

Uh oh!

cubic-dev-ai bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai bot Sep 26, 2025 •

edited

Loading

Uh oh!

Uh oh!

cubic-dev-ai bot Sep 26, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	`${process.env.TEST_API_URL}/v1/crawl/${crawlId}/cancel`,
	`${process.env.TEST_API_URL}/v1/crawl/${crawlId}`,

Search code, repositories, users, issues, pull requests...

Return 409 error when canceling completed crawl jobs #2215

Are you sure you want to change the base?

Return 409 error when canceling completed crawl jobs #2215

Uh oh!

Conversation

devin-ai-integration bot commented Sep 26, 2025 • edited by micahstairs Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Return 409 error when canceling completed crawl jobs

Summary

Review & Testing Checklist for Human

Notes

Summary by cubic

Uh oh!

devin-ai-integration bot commented Sep 26, 2025

🤖 Devin AI Engineer

Uh oh!

mogery left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration bot commented Sep 26, 2025

Uh oh!

cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai bot Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai bot Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

devin-ai-integration bot commented Sep 26, 2025 •

edited by micahstairs

Loading

cubic-dev-ai bot Sep 26, 2025 •

edited

Loading

cubic-dev-ai bot Sep 26, 2025 •

edited

Loading