UN-3560 [FEAT] PG Queue 9 — run-worker.sh `-L celery` log alias by muhammad-ali-e · Pull Request #2066 · Zipstack/unstract

muhammad-ali-e · Jun 17, 2026

What & why

./run-worker.sh -L pg-queue already tails the PG-queue set's logs, but there was no symmetric way to tail only the Celery set. -L (no arg) tails everything — Celery and the PG-queue consumer/reaper — because list_core_worker_dirs includes the PG worker dirs. With both transports running side by side (the strangler-fig setup), the Celery logs get buried in PG-queue chatter.

This adds a -L celery log alias — the mirror of -L pg-queue — that tails every worker log except the PG-queue members (pg_queue_consumer, pg_queue_reaper).

Logs-only: the Celery set is still run via all (PG-queue is opt-in, never in all), so no run-path change is needed.

Change

workers/run-worker.sh only:

a CELERY_SET="celery" constant (mirrors PG_QUEUE_SET)
a branch in tail_logs() that iterates core + pluggable dirs, skipping the two PG types
usage text + examples

Dev-test

With stub PG log files present:

-L → 11 files (9 Celery + 2 PG)
-L celery → 9 files (PG excluded) ✅
-L pg-queue → 2 files (PG only) ✅

bash -n run-worker.sh clean.

Notes

Targets feat/UN-3445-pg-queue-integration (not main).
Independent of the 9e seam PR (UN-3559 / UN-3559 [FEAT] PG Queue 9e PR 1 — execution transport seam (inert) #2062).
Ticket: UN-3560.

🤖 Generated with Claude Code

`./run-worker.sh -L pg-queue` already tails the PG-queue set's logs, but there was no symmetric way to tail only the Celery set: `-L` (no arg) tails EVERYTHING (Celery + PG-queue consumer/reaper), since list_core_worker_dirs includes the PG worker dirs. Add a `-L celery` log alias (mirror of `-L pg-queue`): tails every worker log EXCEPT the PG-queue members (pg_queue_consumer, pg_queue_reaper). Logs-only — the Celery set is still run via 'all'. run-worker.sh only: CELERY_SET constant + a branch in tail_logs() + usage/examples. Dev-tested with stub PG logs: -L = 11 files, -L celery = 9 (PG excluded), -L pg-queue = 2 (PG only). bash -n clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

coderabbitai · Jun 17, 2026

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 1c5b65b6-611f-4db2-8b50-0087e4065a7f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch UN-3560-celery-log-alias

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

muhammad-ali-e

Automated PR Review — `-L celery` log alias

Ran the PR Review Toolkit (Code Reviewer, Silent-Failure Hunter, Type-Design Analyzer, Test Analyzer, Comment Analyzer, Code Simplifier) against the diff vs feat/UN-3445-pg-queue-integration.

Verdict: no blocking issues. Quoting, set -e handling, word-splitting, and the local f / f=$(...) split all follow this file's established conventions, and the empty-result case correctly falls through to the shared guard rather than exec tail with no args.

The inline comments below are quality/consistency improvements, ordered by value:

P1 — worth fixing here

Comment accuracy: the "Mirror" wording (constant block + inline) misdescribes the control flow — celery and pg-queue are complementary sets in different branches, not mirror implementations.
Duplication: the new celery branch is a near-verbatim copy of the all branch; they can collapse into one loop with an exclusion guard.

P2 — hardening / consistency
3. Modeling: the {consumer, reaper} membership test is now the 4th hand-rolled copy in this file — a single PG_QUEUE_MEMBERS source of truth would keep the celery = all − pg-queue invariant true by construction.
4. UX (inherited from siblings): -L celery with no logs exits 0 with a generic message that misattributes the cause for this set.

Tests: none added; none strictly required given this is logs-only tooling with no existing shell-test harness. If a regression guard is wanted cheaply, gate the final exec tail -F on a dry-run env var and add a ~30-line bash test asserting the celery set excludes both PG members and the empty set exits cleanly.

…G members + dedup Address PR #2066 toolkit review: - Comment accuracy (P1, x2): reword 'mirrors PG_QUEUE_SET' / 'mirror of the pg-queue alias' — the celery set is the COMPLEMENT (all minus the two PG members), in a different branch, not a mirror. Drop the directional 'below'. - Modeling (P2): add a single 'PG_QUEUE_MEMBERS' source of truth (readonly assoc array) so 'celery = all - pg-queue' stays correct by construction; the celery branch tests membership via it instead of hand-rolling consumer/reaper. - Simplification (P1): collapse the near-duplicate 'all' and 'celery' tail_logs branches into one loop with a membership-guarded skip. Deferred (P2, inherited): set-aware empty-result message for the shared zero-log guard — spans all three set aliases, best done as one follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

sonarqubecloud · Jun 17, 2026

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

muhammad-ali-e · Jun 17, 2026

Review addressed — commit `4b675a14b`

Thanks for the pass. Per-thread replies above; summary:

Fixed:

Comment accuracy (P1 ×2): the celery set is now described as the complement of the pg-queue tail alias (all workers minus the two PG members), in a separate code path — not a "mirror". Dropped the directional "below".
Single source of truth (P2): added readonly -A PG_QUEUE_MEMBERS; the celery exclusion tests membership via it, so celery = all − pg-queue stays correct by construction as membership grows.
Simplification (P1): collapsed the duplicate all/celery tail_logs branches into one loop with a membership-guarded skip.

Deferred (P2, inherited): set-aware empty-result message + non-zero exit for the shared zero-log guard — spans all three set aliases (all/celery/pg-queue), so best as one follow-up rather than celery-only here.

Re-verified: -L = 11, -L celery = 9 (PG excluded), -L pg-queue = 2; bash -n clean.

greptile-apps · Jun 17, 2026

Greptile Summary

This PR adds a -L celery log-tail alias to run-worker.sh, mirroring the existing -L pg-queue, so operators can tail only the Celery transport's logs during a dual-transport (strangler-fig) setup where PG-queue chatter would otherwise bury Celery output.

Introduces PG_QUEUE_MEMBERS (a readonly associative array keyed on the two PG worker type strings) as an O(1) membership set, and CELERY_SET=\"celery\" as the log-only alias constant.
Updates tail_logs() to branch on CELERY_SET, reusing the existing dir-iteration loop but skipping any dir whose key appears in PG_QUEUE_MEMBERS; usage text and examples are updated accordingly.

Confidence Score: 5/5

Safe to merge — the change is confined to a single shell script, touches only the log-tail path, and carries no risk to worker startup, killing, or restart logic.

The new code is a straightforward extension of an already-proven pattern (-L pg-queue). The PG_QUEUE_MEMBERS lookup is guarded with :- to avoid any unbound-variable failure, the CELERY_SET constant is properly scoped to the log-tail branch, and no run-path code was altered. The one conceptual note — that run_pg_queue_set still enumerates the two members explicitly rather than iterating PG_QUEUE_MEMBERS — is pre-existing and does not affect correctness of this change.

No files require special attention.

Important Files Changed

Filename	Overview
workers/run-worker.sh	Adds `PG_QUEUE_MEMBERS` associative array, `CELERY_SET` constant, a filter branch in `tail_logs()`, and updated usage/examples; logic is correct and well-contained.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["-L [ARG]"] --> B{"requested == '' or 'all' or CELERY_SET?"}
    B -- "yes (iterate all dirs)" --> C["for d in core + pluggable dirs"]
    C --> D{"requested == 'celery' AND d in PG_QUEUE_MEMBERS?"}
    D -- yes --> E["skip (continue)"]
    D -- no --> F["resolve_log_file(d)"]
    F --> G["append to log_files[]"]
    B -- "no" --> H{"resolve via WORKERS or PLUGGABLE_WORKERS"}
    H -- "unknown" --> I["Error: Unknown worker type"]
    H -- "canonical == PG_QUEUE_SET" --> J["append consumer + reaper logs"]
    H -- "single worker" --> K["resolve_log_file(canonical)"]
    K --> L{"file found?"}
    L -- "no" --> M["warn: start with -d first"]
    L -- "yes" --> N["append to log_files[]"]
    G --> O{"log_files empty?"}
    J --> O
    N --> O
    O -- "yes" --> P["warn: no log files"]
    O -- "no" --> Q["exec tail -F log_files"]

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A["-L [ARG]"] --> B{"requested == '' or 'all' or CELERY_SET?"}
    B -- "yes (iterate all dirs)" --> C["for d in core + pluggable dirs"]
    C --> D{"requested == 'celery' AND d in PG_QUEUE_MEMBERS?"}
    D -- yes --> E["skip (continue)"]
    D -- no --> F["resolve_log_file(d)"]
    F --> G["append to log_files[]"]
    B -- "no" --> H{"resolve via WORKERS or PLUGGABLE_WORKERS"}
    H -- "unknown" --> I["Error: Unknown worker type"]
    H -- "canonical == PG_QUEUE_SET" --> J["append consumer + reaper logs"]
    H -- "single worker" --> K["resolve_log_file(canonical)"]
    K --> L{"file found?"}
    L -- "no" --> M["warn: start with -d first"]
    L -- "yes" --> N["append to log_files[]"]
    G --> O{"log_files empty?"}
    J --> O
    N --> O
    O -- "yes" --> P["warn: no log files"]
    O -- "no" --> Q["exec tail -F log_files"]

_{Reviews (1): Last reviewed commit: "UN-3560 [FIX] -L celery review — complem..." | Re-trigger Greptile}

muhammad-ali-e commented Jun 17, 2026

View reviewed changes

Comment thread workers/run-worker.sh Outdated

Comment thread workers/run-worker.sh Outdated

Comment thread workers/run-worker.sh Outdated

Comment thread workers/run-worker.sh Outdated

Comment thread workers/run-worker.sh

muhammad-ali-e marked this pull request as ready for review June 17, 2026 05:42

muhammad-ali-e merged commit 473603a into feat/UN-3445-pg-queue-integration Jun 17, 2026
5 checks passed

muhammad-ali-e deleted the UN-3560-celery-log-alias branch June 17, 2026 05:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

UN-3560 [FEAT] PG Queue 9 — run-worker.sh `-L celery` log alias#2066

UN-3560 [FEAT] PG Queue 9 — run-worker.sh `-L celery` log alias#2066
muhammad-ali-e merged 2 commits into
feat/UN-3445-pg-queue-integrationZipstack/unstract:feat/UN-3445-pg-queue-integrationfrom
UN-3560-celery-log-aliasZipstack/unstract:UN-3560-celery-log-aliasCopy head branch name to clipboard

muhammad-ali-e commented Jun 17, 2026

Uh oh!

coderabbitai Bot commented Jun 17, 2026 •

edited

Loading

Review skipped

Uh oh!

muhammad-ali-e left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud Bot commented Jun 17, 2026

Uh oh!

muhammad-ali-e commented Jun 17, 2026

Uh oh!

Uh oh!

greptile-apps Bot commented Jun 17, 2026

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Flowchart

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Search code, repositories, users, issues, pull requests...

Uh oh!

Conversation

muhammad-ali-e commented Jun 17, 2026

What & why

Change

Dev-test

Notes

Uh oh!

coderabbitai Bot commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

muhammad-ali-e left a comment

Choose a reason for hiding this comment

Automated PR Review — -L celery log alias

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud Bot commented Jun 17, 2026

Quality Gate passed

Uh oh!

muhammad-ali-e commented Jun 17, 2026

Review addressed — commit 4b675a14b

Uh oh!

Uh oh!

greptile-apps Bot commented Jun 17, 2026

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Flowchart

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

coderabbitai Bot commented Jun 17, 2026 •

edited

Loading

Automated PR Review — `-L celery` log alias

Review addressed — commit `4b675a14b`