…s ERROR

The reaper only DELETEd expired pg_barrier_state rows — a stranded execution
vanished silently (stuck in EXECUTING). This makes it RECOVER them — the hard
dependency from PR 2c review for enabling the PG transport (un-catchable strand
windows otherwise bottom out at the ~6h barrier expiry).

Per expired barrier the leader (recover_expired_barriers), best-effort + per-exec:
- Marks the execution ERROR via the internal API (the path the normal callback
  uses — business state never goes direct-DB; the API is functional: execution_time,
  error truncation, attempts, events/notifications, multi-tenant boundary). Message
  distinguishes remaining>0 (work incomplete) vs remaining==0 (callback never fired).
  Reads status first and SKIPS if already terminal (a remaining==0 row can be a
  COMPLETED exec whose row-delete failed; update_execution has no terminal guard) or
  if the row has no org.
- Reclaims pg_batch_dedup + pg_barrier_state directly in PG (same boundary as the
  rest of queue_backend). Recover-THEN-delete: a failed mark leaves the row for the
  next sweep to retry (single-leader → no double-claim).

- backend: organization_id column on PgBarrierState + migration 0007 (reaper needs
  it for the org-scoped status API).
- workers: PgBarrier.enqueue stamps organization_id into the UPSERT; PgReaper holds
  a lazily-built InternalAPIClient; sweep_expired_barriers → recover_expired_barriers.

Tests: reaper suite reworked — real-PG recovery w/ fake API client (mark-ERROR
remaining>0/==0, terminal-skip no-overwrite, org-missing skip, API-failure leaves
row, dedup reclaim, tick-via-real-conn). 109 reaper/barrier/dedup tests green.
Dev-tested vs real InternalAPIClient+backend: PENDING stranded → ERROR + cleaned;
COMPLETED → NOT overwritten + cleaned (terminal-guard verified end-to-end).

Deferred: callback re-fire for remaining==0 (needs callback_descriptor on the row).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

for more information, see https://pre-commit.ci

…in reaper tests

Replaced `api.update_calls[0]` (which Sonar can't prove non-empty → IndexError
risk) with single-element unpacking `(call,) = api.update_calls` — removes the
index and additionally asserts exactly one status mark was made. Behaviour
identical; 52 reaper tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…RROR (Critical) + hardening

- [Critical] _execution_status now RAISES when the status read fails
  (get_workflow_execution returns success=False, never raises) — a transient
  blip no longer reads as "non-terminal" and flips a COMPLETED execution to
  ERROR; the caller's except retains the row for retry. + test.
- [Medium] use ExecutionStatus.is_completed (single source of truth) instead of
  a local _TERMINAL_STATUSES frozenset that could drift; dropped the fake
  in (_TERMINAL_STATUSES) parens.
- [Medium] remaining is NOT-NULL int → typed int, three-way branch
  (>0 work-incomplete / ==0 callback-never-fired / <0 already-torn-down) with
  accurate messages.
- [Medium] re-guard the barrier DELETE on `expires_at < now()` + only clear
  dedup when the barrier row was actually reclaimed — a same-id re-enqueue
  (UPSERT resets expires_at) is no longer torn down mid-recovery.
- [Medium] org-missing now LEAVES the row (doesn't erase the only recovery
  handle) + logs ERROR; PgBarrier.enqueue logs ERROR at write time if a barrier
  is enqueued without an org (should never happen).
- [Low] recover_expired_barriers emits an aggregate summary; escalates to
  logger.error when a non-empty sweep recovers nothing (systemic: API down).
- [Low] refreshed the stale "(future) periodic sweep" comment in pg_barrier.

Tests: _FakeApiClient models the real success/failure contract + records
file_execution; +read-failure-retains-row, +file_execution=False, +multi-row
isolation (one fails, others recover), +org-stamping x3 (stamp/default/UPSERT
refresh). 115 reaper/barrier/dedup tests green; ruff + ruff-format clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

A new occurrence introduced by the prior review-round test
(test_status_read_passes_file_execution_false used api.get_calls[0][2]). Replaced
with single-element unpacking — [(_exec_id, _org, file_execution)] = api.get_calls
— index-free and additionally pins exactly one status read. 55 reaper tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ate skip-vs-fail

- [main] re-arm race: the ERROR mark now fires only after a re-check that the
  row is STILL expired (_still_expired) immediately before marking — so a same-id
  re-enqueue (expires_at reset to future) between the sweep SELECT and the mark
  can't get its freshly-running execution flagged ERROR. The DELETE stays guarded
  on expires_at < now() as the second line of defence. + test (re-arm via the
  status-read side-effect) asserting no mark + row left intact.
- status=None on a success=True read no longer falls through to mark ERROR —
  treated as indeterminate: skip + leave the row for the next sweep. + test.
- aggregate logging distinguishes genuine failures (exceptions) from benign skips
  (terminal / re-armed / no-status / no-org): the systemic "recovered NONE / API
  down" ERROR escalates only on real failures, not on all-skipped sweeps. + caplog
  test that an all-org-missing sweep doesn't escalate.

58 reaper + 51 barrier tests green; ruff + ruff-format clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The comment ended with a parenthesised tuple "(exec_id, org, file_execution)" —
a complete Python expression S125 reads as commented-out code. Reworded to prose
(and the stale `sweep_expired_barriers`→recover rename + a couple of inline
`foo()` prose mentions neutralised while here). Comment-only; tests unaffected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>