fix: capture panics in pipeline's base resolver, and return as errors. by senojj · Pull Request #2994 · openfga/openfga

senojj · Mar 20, 2026

Description

What problem is being solved?

Currently, if a panic occurs during the processing of a message within the pipeline, there are locations where the panic could be returned as errors, instead of terminating the process.

How is it being solved?

Recover from panics that could instead be returned as errors.

What changes are made to solve it?

References

Review Checklist

I have clicked on "allow edits by maintainers".
I have added documentation for new/changed functionality in this PR or in a PR to openfga.dev [Provide a link to any relevant PRs in the references section above]
The correct base branch is being used, if not main
I have added tests to validate that the change in functionality is working as expected

Summary by CodeRabbit

Bug Fixes
- Recoverable panics during pipeline/list processing no longer terminate the process; they are now captured and reported as errors, improving stability.
Chores
- Added panic-recovery for concurrent operations.
- Refactored error handling to consistently route and report errors from background tasks and processors.

coderabbitai · Mar 20, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: ddda56c8-1417-456e-aa65-fb8353fc20c3

📥 Commits

Reviewing files that changed from the base of the PR and between f0fa979 and 7e64f48.

📒 Files selected for processing (1)

pkg/server/commands/reverseexpand/pipeline/resolver_base.go

🚧 Files skipped from review as they are similar to previous changes (1)

pkg/server/commands/reverseexpand/pipeline/resolver_base.go

📝 Walkthrough

Walkthrough

Adds a panic-recovery helper and updates the reverse-expand pipeline to use pointer-based error propagation and deferred handlers so panics are captured and routed through resolver error flow rather than crashing goroutines.

Changes

Cohort / File(s)	Summary
Panic Recovery `internal/concurrency/panic.go`	New `RecoverFromPanic(err error)` that calls `recover()`, captures a fixed-size goroutine stack via `runtime.Stack`, and assigns a formatted error to `err` when provided.
Resolver core & error surface `pkg/server/commands/reverseexpand/pipeline/resolver_core.go`	Changed `resolverCore.error` signature to accept `*error` and only send when the pointer and its value are non-nil.
Resolver goroutines & base processing `pkg/server/commands/reverseexpand/pipeline/resolver_base.go`	Goroutine handlers now declare a named `err`, `defer r.error(&err)`, `defer concurrency.RecoverFromPanic(&err)`, and move `sentCount.Add(...)` into a deferred call to ensure execution on exit. Also `msg.Done()` moved to `defer`.
Processor error handling `pkg/server/commands/reverseexpand/pipeline/processor_operator.go`	Callsites updated to pass `&err` to processor error handling when `obj.Object()` fails (pass error pointer instead of value).
Changelog `CHANGELOG.md`	Added Unreleased note describing that recoverable panics in list objects are converted to returned errors (`#2994`).

Sequence Diagram(s)

sequenceDiagram
  participant G as Goroutine
  participant P as Processor
  participant R as ResolverCore
  participant E as ErrorsChannel

  G->>P: process items
  alt item.Object() panics
    Note right of G: deferred RecoverFromPanic(&err)
    G->>R: RecoverFromPanic triggers recover(), captures stack, sets *err
  end
  alt err != nil
    Note right of G: deferred r.error(&err)
    G->>E: send dereferenced err
  else success
    G->>P: append item to results
  end
  G->>G: deferred sentCount.Add(...) and msg.Done() run on exit

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested reviewers

justincoh
Vic-Dev

Poem

🐰 I nibble bugs and hug the stack,
I catch the falls and bring them back.
With deferred paws and tiny hop,
No panic eats the whole shop.
Hooray—errors hop into place! 🥕

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: introducing panic recovery in the pipeline's base resolver to convert panics into errors instead of terminating the process.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch lo-panic-handling

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Copilot

Pull request overview

This PR aims to prevent goroutine panics in the reverse-expand pipeline’s base resolver from terminating the process by recovering panics and reporting them back through the pipeline’s existing error stream.

Changes:

Added a RecoverFromPanic(*error) helper under internal/concurrency to convert panics into errors with a stack trace.
Wrapped baseResolver worker goroutines with panic recovery and deferred error reporting.
Changed pipeline error reporting helper to accept *error and updated call sites accordingly.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

File	Description
pkg/server/commands/reverseexpand/pipeline/resolver_core.go	Changes error reporting helper signature (currently introduces a nil-error enqueue bug).
pkg/server/commands/reverseexpand/pipeline/resolver_base.go	Adds panic recovery and deferred error/sent-count reporting around worker goroutines.
pkg/server/commands/reverseexpand/pipeline/processor_operator.go	Updates processor error reporting to pass `*error`.
internal/concurrency/panic.go	Introduces `RecoverFromPanic` utility for converting panics into errors.

coderabbitai

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@pkg/server/commands/reverseexpand/pipeline/resolver_base.go`:
- Around line 104-109: The worker goroutine's panic recovery can leak an
in-flight message because r.drain calls processor.process and if that panics
before msg.Done() is invoked the in-flight counter is not released; update
baseProcessor.process to call defer msg.Done() at the start of the function (and
remove any explicit msg.Done() at the end) so message cleanup runs even if
processing panics, and apply the same change to the other processor.process
implementations referenced (the occurrences noted around the 125-130 region) so
all paths release the tracker before goroutine-level RecoverFromPanic in
r.drain/wgRecursive.
- Line 107: The defer currently captures processor.SentCount at defer
registration time, yielding zero; change the defer to call an anonymous function
so the metric is read at function exit (e.g., replace defer
sentCount.Add(processor.SentCount) with defer func() {
sentCount.Add(processor.SentCount) }()); do the same for the other similar defer
at the second occurrence (ensure you update the matching deferred Add call that
reads a processor field after r.drain()).

In `@pkg/server/commands/reverseexpand/pipeline/resolver_core.go`:
- Around line 26-29: The resolverCore.error helper currently sends the
dereferenced pointer unconditionally when the pointer is non-nil, which can push
a nil error value into r.errors; update resolverCore.error to only send when the
pointer is non-nil and its pointee is non-nil (i.e., check err != nil && *err !=
nil) so you never send a nil error into r.errors (or alternatively change the
caller to pass an error value instead of a *error, but the minimal fix is to add
the pointee nil check in resolverCore.error to avoid false error signals); keep
the function name resolverCore.error and the channel r.errors unchanged.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: afb2023b-c700-4a68-b9c8-1882a5d7830f

📥 Commits

Reviewing files that changed from the base of the PR and between 4600682 and 514e1da.

📒 Files selected for processing (4)

internal/concurrency/panic.go
pkg/server/commands/reverseexpand/pipeline/processor_operator.go
pkg/server/commands/reverseexpand/pipeline/resolver_base.go
pkg/server/commands/reverseexpand/pipeline/resolver_core.go

coderabbitai

Actionable comments posted: 1

♻️ Duplicate comments (2)

pkg/server/commands/reverseexpand/pipeline/resolver_base.go (2)
24-64: ⚠️ Potential issue | 🔴 Critical

Make msg.Done() panic-safe in baseProcessor.process.

With the new worker-boundary recovery, a panic anywhere before Line 63 exits the goroutine without releasing the current message. That leaves the in-flight counter stuck, so WaitForDrain() at Line 141 can hang instead of returning the recovered error.
Suggested fix
 func (p *baseProcessor) process(ctx context.Context, edge *Edge, msg *message) {
+	defer msg.Done()
+
 	var unseen []string
@@
 	p.SentCount += int64(p.broadcast(ctx, output, p.listeners))
-
-	msg.Done()
 }
Based on learnings, `baseResolver.Resolve` intentionally calls `WaitForDrain()` before closing listeners, so leaked in-flight messages block shutdown.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pkg/server/commands/reverseexpand/pipeline/resolver_base.go` around lines 24
- 64, Add an early defer in baseProcessor.process to ensure msg.Done() is always
invoked even if a panic occurs: at the top of process, immediately do defer
func() { if msg != nil { msg.Done() } }(), so the in-flight counter is released
on panic (do not recover the panic here), preserving existing shutdown semantics
that rely on WaitForDrain() in baseResolver.Resolve.
108-108: ⚠️ Potential issue | 🟡 Minor

Read processor.SentCount when the defer runs, not when it is registered.

defer sentCount.Add(processor.SentCount) captures processor.SentCount before r.drain() executes, so the items.count span attribute at Line 152 stays wrong even when workers emit results.
Suggested fix
-					defer sentCount.Add(processor.SentCount)
+					defer func() {
+						sentCount.Add(processor.SentCount)
+					}()
@@
-				defer sentCount.Add(processor.SentCount)
+				defer func() {
+					sentCount.Add(processor.SentCount)
+				}()
Also applies to: 129-129
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pkg/server/commands/reverseexpand/pipeline/resolver_base.go` at line 108, The
defer currently evaluates processor.SentCount immediately because its value is
passed as an argument; change deferred calls like "defer
sentCount.Add(processor.SentCount)" to a deferred closure that reads
processor.SentCount when the function returns, e.g. use "defer func() {
sentCount.Add(processor.SentCount) }()" (apply the same fix for the other
occurrence around line 129); this ensures the value is captured at
defer-execution time after r.drain() and worker results are applied.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@internal/concurrency/panic.go`:
- Around line 20-24: The panic recovery currently sets *err =
fmt.Errorf("recovered from panic %v. Call stack:\n%s", r, stacktrace) which
stringifies error panics and breaks error chains; update the recovery to check
whether the recovered value r is an error and, if so, wrap it using
fmt.Errorf("recovered from panic: %w\nCall stack:\n%s", r, stacktrace)
(preserving the error chain), otherwise keep the existing %v formatting for
non-error panics—modify the assignment to *err accordingly (referencing the
recovered value variable r, the error pointer err, and stacktrace).

---

Duplicate comments:
In `@pkg/server/commands/reverseexpand/pipeline/resolver_base.go`:
- Around line 24-64: Add an early defer in baseProcessor.process to ensure
msg.Done() is always invoked even if a panic occurs: at the top of process,
immediately do defer func() { if msg != nil { msg.Done() } }(), so the in-flight
counter is released on panic (do not recover the panic here), preserving
existing shutdown semantics that rely on WaitForDrain() in baseResolver.Resolve.
- Line 108: The defer currently evaluates processor.SentCount immediately
because its value is passed as an argument; change deferred calls like "defer
sentCount.Add(processor.SentCount)" to a deferred closure that reads
processor.SentCount when the function returns, e.g. use "defer func() {
sentCount.Add(processor.SentCount) }()" (apply the same fix for the other
occurrence around line 129); this ensures the value is captured at
defer-execution time after r.drain() and worker results are applied.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 605b6893-03d8-4a48-832c-1403bdb8409a

📥 Commits

Reviewing files that changed from the base of the PR and between 514e1da and f0fa979.

📒 Files selected for processing (5)

CHANGELOG.md
internal/concurrency/panic.go
pkg/server/commands/reverseexpand/pipeline/processor_operator.go
pkg/server/commands/reverseexpand/pipeline/resolver_base.go
pkg/server/commands/reverseexpand/pipeline/resolver_core.go

✅ Files skipped from review due to trivial changes (1)

CHANGELOG.md

🚧 Files skipped from review as they are similar to previous changes (1)

pkg/server/commands/reverseexpand/pipeline/processor_operator.go

codecov · Mar 20, 2026

Codecov Report

❌ Patch coverage is 61.29032% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.53%. Comparing base (ff2040a) to head (7e64f48).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/concurrency/panic.go	7.70%	10 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2994      +/-   ##
==========================================
+ Coverage   90.53%   90.53%   +0.01%     
==========================================
  Files         187      188       +1     
  Lines       20227    20249      +22     
==========================================
+ Hits        18311    18331      +20     
- Misses       1259     1263       +4     
+ Partials      657      655       -2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Add support for exporting logs via OTLP, allowing integration with any OpenTelemetry-compatible backend (Grafana Loki, Datadog, GCP Cloud Logging, etc.). When log.otlp.enabled is set, logs are exported via OTLP in addition to stdout. The otelzap bridge attaches span context (trace ID, span ID) to each log record, enabling log-trace correlation in backends that support it. Configuration mirrors the trace configuration shape: the standard OTel env vars (OTEL_EXPORTER_OTLP_LOGS_ENDPOINT, OTEL_EXPORTER_OTLP_ENDPOINT) only answer where logs are sent, while the explicit OpenFGA flag (log.otlp.enabled / --log-otlp-enabled / OPENFGA_LOG_OTLP_ENABLED) answers whether to export at all — so a cluster that injects the generic endpoint variable for tracing does not silently start exporting logs on upgrade. zap's production sampling is applied outside the tee, so the keep/drop decision is made once, before fan-out: stdout and OTLP receive the same deterministically sampled stream and collector egress stays bounded during log storms. Changes: - Add otelzap bridge core backed by an OTLP log provider (internal/telemetry/logging.go) - Add WithOTELCore logger option that tees log entries to the bridge. The bridge core is capped to the configured log level, the stdout core strips the bridge-only context field via contextFilterCore, and the sampler wraps the tee (pkg/logger/logger.go) - Attach the context field in the *WithContext logging methods only when an OTEL core is configured, and use those methods in the gRPC logging interceptor so the bridge can extract span context (pkg/middleware/logging/logging.go) - Add config: log.otlp.enabled, log.otlp.endpoint, log.otlp.tls.enabled with OPENFGA_LOG_OTLP_* env vars and OTEL_EXPORTER_OTLP_* endpoint fallbacks (pkg/server/config/config.go, cmd/run/, .config-schema.json) feat: add potential v2 breaking change logs for Expand and ListUsers (openfga#3182) release: update changelog for release 1.18.1 (openfga#3188) release: Update changelog to prep for 1.18.1 release (openfga#3186) test: fix flaky TestV2CheckWithIteratorCache_HigherConsistencyBypassesCache (openfga#3061) Co-authored-by: Joshua Jones <joshua.jones.software@gmail.com> Merge commit from fork * test reproducing ListUsers report * implement fix * additional test with 3 operands fix: match IPv4-mapped IPv6 addresses in the in_cidr condition (openfga#3181) Signed-off-by: kanywst <niwatakuma@icloud.com> fix: use deterministic proto marshaling for stored authorization models (openfga#3171) chore: create draft release and publish after provenance succeeds for immutable tags/releases (openfga#3178) docs: fix changelog entry (openfga#3177) Signed-off-by: Saad Hussain <saad.hussain@okta.com> feat: add v2Check logs for resolution breaking change (openfga#3149) feat: BatchCheck uses v2Check when ExperimentalWeightedGraphCheck is enabled (openfga#3154) Signed-off-by: Saad Hussain <saad.hussain@okta.com> chore: update changelog to add CVE identifiers for recent fixes (openfga#3176) chore: update todo comment string in migration guide (openfga#3175) release: update changelog for release `v1.18.0` (openfga#3174) Co-authored-by: adriantam <adrian.tam@okta.com> Merge commit from fork * fix(authn): require issuer and audience when OIDC authn is enabled Enforce configuration authn.oidc.audience and authn.oidc.issuer when `authn.method` is set to `oidc`. * fixed based on code review feedback * fix: adding comments + test case on empty space for audience * update based on code review feedback * Update CHANGELOG.md Co-authored-by: Joshua Jones <joshua.jones.software@gmail.com> * update changelog * Update CHANGELOG.md --------- Co-authored-by: Joshua Jones <joshua.jones.software@gmail.com> Merge commit from fork * fix(mysql): collate identifier columns as utf8mb4_bin * fix: add operator note to changelog * fix: move tests to shared storage * fix: separate migrations for each table * fix: add runbook for migrations * fix: set lock_wait_timeout * fix: add docker instructions * fix: add docker instructions * fix: combine the migrations back into one, fix documentation for this * fix: include details about table copy and * Update CHANGELOG.md Co-authored-by: Adrian Tam <adrian.tam@okta.com> --------- Co-authored-by: Adrian Tam <adrian.tam@okta.com> fix: use constant-time comparison for preshared key authentication (openfga#3168) chore(deps): bump the dependencies group with 2 updates (openfga#3166) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Justin Cohen <justincoh@gmail.com> chore(deps): bump the dependencies group with 2 updates (openfga#3167) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> release: update changelog for release `v1.17.1` (openfga#3165) Signed-off-by: Saad Hussain <saad.hussain@okta.com> chore: bump grpc-health-probe to v0.4.52 (openfga#3164) Co-authored-by: Saad Hussain <saad.hussain@okta.com> docs: update caching docs (openfga#3163) Signed-off-by: Saad Hussain <saad.hussain@okta.com> fix: continuation token deserializer - handle `|` in type names (openfga#3152) chore(deps): bump the dependencies group across 1 directory with 13 updates (openfga#3162) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Justin Cohen <justincoh@gmail.com> chore(deps): bump the dependencies group across 1 directory with 10 updates (openfga#3156) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Justin Cohen <justincoh@gmail.com> fix: use query start time as iterator cache entry LastModified to prevent stale-read survival (openfga#3155) chore(deps): bump grpc-ecosystem/grpc-health-probe from v0.4.50 to v0.4.51 in the dependencies group (openfga#3157) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Saad Hussain <saad.hussain@okta.com> Co-authored-by: Justin Cohen <justincoh@gmail.com> chore: Bump go toolchain to 1.26.4 (openfga#3159) fix: prevent v2Check from falling back for throttling and validation (openfga#3150) ci: make pr benchmark comparisons less fragile (openfga#3153) Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> release: update changelog for release `v1.17.0` (openfga#3151) Co-authored-by: Vic-Dev <vmichellej@gmail.com> refactor: redesign cache key generation, making it more secure and consistent (openfga#3148) feat: add configurable trace sampler with ParentBased support (openfga#3072) release: update changelog for release `v1.16.1` (openfga#3147) chore: update grpc-health-probe to latest to addres std lib CVEs (openfga#3146) fix: skip v2Check weight2 pruning if iterator is unordered (openfga#3145) Signed-off-by: Saad Hussain <saad.hussain@okta.com> chore: add concurrency-group to PR-related CI steps (openfga#3140) docs: move OIDC JWKS refresh entry from v1.15.1 to v1.16.0 (openfga#3142) fix: when v2Check is primary algorithm, fix fallback condition and emit metrics (openfga#3141) release: update changelog for release `v1.16.0` (openfga#3139) Merge commit from fork * fix: Standardize cache key generation everywhere * Revert "fix: Standardize cache key generation everywhere" This reverts commit ce60c6f9ae48a310a4dea2b824b5993520c8ba1c. * length-encoded approach * Revert "length-encoded approach" This reverts commit b94637c187c57a2d3f3904247267bbc8ad21a41a. * implement Hexer() and use in some cache keys * appendConditionsHash -> generateConditionsHash * mv Hexer -> BuildKey * BuildKey -> BuildCacheKey refactor to BuildCacheKey on each individual prefix component to remove collision risk * make v2 cacheKey functions the standard * make cache key prefixes consistent * generateConditionsHash -> generateConditionsString * delimit condition name string * remove extra empty check * relocate and reuse existing userTypeRestrictions function * replace hashing of conditions for brevity * fix tests * cleanup, add test file * remove unused slice capacity * make prefix treatments consistent * add nil-byte separator in object ids * hash user type restrictions for brevity * update comment string, use nil-byte separator for future-proofing * remove call to xxhash.new * adjustments for performance, consistency, and interface ergonomics. * make condition hash generation more efficient. * add clarifying comments for size caluculations * fix builder growth calculations to match original intentions * update outdated comment * don't hash potentially zero values * Revert "don't hash potentially zero values" This reverts commit 2ad6c5b796bc20a82c1414c5146c3c708330eefa. * add condition key separator unconditionally * patch test expectations for new key structure * export V2IteratorCachePrefix, use in tests --------- Co-authored-by: justin <justin.cohen@okta.com> Co-authored-by: Joshua Jones <joshua.jones@okta.com> fix: unintentional zeroing of slice values by setting slice to nil (openfga#3135) increase the check v2 trace information fidelity (openfga#3134) feat: add datastore ping and ping retry configurations (openfga#3113) fix: prevent v2Check strategies returning spurious false on context cancellation (openfga#3128) Signed-off-by: Saad Hussain <saad.hussain@okta.com> fix(authn/oidc): refresh JWKS on unknown kid to handle key rotation (openfga#3101) fix: v2Check falls back to v1 on errors (openfga#3126) fix: don't cache false results from cancelled-context goroutines in v2Check (openfga#3125) Signed-off-by: Saad Hussain <saad.hussain@okta.com> make union cache key unique by including the node input's label (openfga#3117) Signed-off-by: Saad Hussain <saad.hussain@okta.com> Co-authored-by: Saad Hussain <saad.hussain@okta.com> fix: shadow v2check and check use the same trace (openfga#3118) fix: bump go toolchain version to 1.26.3 (openfga#3115) chore: add more spans/attributes to v1 and v2 Check (openfga#3116) Signed-off-by: Saad Hussain <saad.hussain@okta.com> fix: add matches attribute to shadowv2Check and Check spans (openfga#3114) fix: use the same `request_id` for shadow traces in v2Check (openfga#3110) release: update changelog for release `v1.15.1` (openfga#3112) feat: reuse MySQL container across tests (openfga#3042) fix: close all channels opened thus far, before return on error (openfga#3111) chore: Add more spans/attributes to v2Check (openfga#3102) Signed-off-by: Saad Hussain <saad.hussain@okta.com> fix: ensure acquired limiter token is released on throttle context cancelation (openfga#3106) fix: cancel context before waiting on worker pool in ResolveUnionEdges (openfga#3105) fix: replace golang.org/x/exp/maps with stdlib maps to resolve govet inline errors (openfga#3104) fix: v2Check EdgeCacheKey collisions (openfga#3097) Signed-off-by: Saad Hussain <saad.hussain@okta.com> fix: expose context errors when they can be the potential cause of an underlying datastore error (openfga#3096) chore(deps): bump the dependencies group across 1 directory with 2 updates (openfga#3093) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> fix: move subproblem cache lookup from ResolveUnionEdges into ResolveUnion (openfga#3095) chore(deps): bump the dependencies group with 4 updates (openfga#3092) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> fix: error handler panic (openfga#3091) release: update changelog for release `v1.15.0` (openfga#3090) chore(deps): bump the dependencies group across 1 directory with 4 updates (openfga#3087) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> fix: v2Check correctly uses query cache even when cache controller is disabled (openfga#3086) Signed-off-by: Saad Hussain <saad.hussain@okta.com> chore(deps): bump the dependencies group across 1 directory with 7 updates (openfga#3081) Signed-off-by: dependabot[bot] <support@github.com> chore(deps): bump github.com/jackc/pgx/v5 from 5.9.1 to 5.9.2 (openfga#3085) Signed-off-by: dependabot[bot] <support@github.com> chore(deps): bump grpc-ecosystem/grpc-health-probe from v0.4.47 to v0.4.48 in the dependencies group across 1 directory (openfga#3065) Signed-off-by: dependabot[bot] <support@github.com> feat: try to use UDS internally between HTTP server and gRPC server (openfga#2937) feat: add jitter to internal cache TTLs to prevent thundering herd effects (openfga#3033) Signed-off-by: Asish Kumar <officialasishkumar@gmail.com> list objects pipeline edge pruning (openfga#3075) update toolchain go to 1.26.2 to address stdlib CVEs (openfga#3084) chore: add store ID and datastore query/item count to shadowV2Check log (openfga#3073) Signed-off-by: Saad Hussain <saad.hussain@okta.com> CI speed up (openfga#3062) Co-authored-by: Joshua Jones <joshua.jones.software@gmail.com> Co-authored-by: Joshua Jones <joshua.jones@okta.com> Add tracing to v2 Check planner strategy selection (openfga#3077) fix: v2Check honours `check-query-cache-enabled` flag (openfga#3070) Signed-off-by: Saad Hussain <saad.hussain@okta.com> feat: reuse PostgreSQL container across tests (openfga#3018) chore: fix reporter links in changelog (openfga#3069) release: update changelog for release `v1.14.2` (openfga#3068) fix: add null byte delimiter in contextual tuple cache keys and validation in v2Check (openfga#3064) release: update changelog for release `v1.14.1` (openfga#3060) chore(deps): bump the dependencies group across 1 directory with 7 updates (openfga#3056) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Adrian Tam <adrian.tam@okta.com> fix: use a baseURL for AuthZEN configuration endpoint (openfga#3057) chore: replace docker with moby (openfga#3047) Iterator Cache V2: Storage Wrapper Pattern with Lock-Free Design (openfga#3016) Signed-off-by: Saad Hussain <saad.hussain@okta.com> Co-authored-by: Saad Hussain <saad.hussain@okta.com> Co-authored-by: Joshua Jones <joshua.jones.software@gmail.com> test: fix flaky condition test (openfga#3058) Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> chore(deps): bump the dependencies group across 1 directory with 6 updates (openfga#3045) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> chore(deps): bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp from 1.41.0 to 1.43.0 (openfga#3054) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Adrian Tam <adrian.tam@okta.com> feat: add graceful shutdown timeout configuration (openfga#2976) chore: update changelog to reflect playground off-by-default behavior (openfga#3053) chore: decouple Error() and Unwrap() (openfga#3051) perf: remove `fmt` from cache key generation (openfga#3006) perf: enhancements for list objects (openfga#3043) chore: remove stale notice file (openfga#3050) docs: update 1.14.0 for CVE fix (openfga#3046) feat: Add datastore throttling & concurrency limiting to v2Check (openfga#3035) Signed-off-by: Saad Hussain <saad.hussain@okta.com> release: update changelog for release `v1.14.0` (openfga#3040) batch check cache (openfga#3025) Merge commit from fork Also prevent enabling playground when the server requires authentication feat: add stats on tuple iterator query (openfga#3030) fix: remove unnecessary non-deterministic test (openfga#3038) remove unnecessary import (openfga#3032) perf: improve the intersection algorithm, reducing latency and memory use (openfga#3031) fix: ListObjects pipeline algorithm enhancements and fix for potential deadlock (openfga#3028) chore: Also update openfga/helm-charts in release script (openfga#3010) chore: update CICD to enforce GRPC healthprobe changes (openfga#2990) fix: SQL `TupleOperation` serialization and `pgx.ErrNoRows` error handling (openfga#3014) docs: update changelog for CVE-2026-33729 (openfga#3017) chore: output the diff after running keep-a-changelog (openfga#3015) chore(deps): bump the dependencies group across 1 directory with 11 updates (openfga#2998) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> chore(deps): bump the dependencies group across 1 directory with 10 updates (openfga#2999) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> release: update changelog for release `v1.13.1` (openfga#3002) Merge commit from fork * strip unicode control characters, encode context key length in cache key * add cache_key tests to ensure we escape control characters and encode context key length * run linter * docs: update changelog * Apply suggestion from @adriantam Co-authored-by: Adrian Tam <adrian.tam@okta.com> * Apply suggestion from @adriantam Co-authored-by: Adrian Tam <adrian.tam@okta.com> --------- Co-authored-by: Saad Hussain <saad.hussain@okta.com> Co-authored-by: Saad Hussain <saad.h@outlook.com> Co-authored-by: Adrian Tam <adrian.tam@okta.com> release: update changelog for release `v1.13.0` (openfga#2997) refactor: Separate caches for v1 and v2 Check (openfga#2968) Co-authored-by: Justin Cohen <justincoh@gmail.com> docs: fix typos in comments (openfga#2972) Signed-off-by: Artem Muterko <artem@sopho.tech> Co-authored-by: Adrian Tam <adrian.tam@okta.com> observability: aggregate message statistics for each list-objects sender into a single span (openfga#2993) Co-authored-by: Justin Cohen <justincoh@gmail.com> fix: capture panics in pipeline's base resolver, and return as errors. (openfga#2994) docs: fix typos in RELEASES.md and Makefile (openfga#2980) Co-authored-by: Adrian Tam <adrian.tam@okta.com> AuthZen v1.0 Implementation (openfga#2875) Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Rokibul Hasan <mdrokibulhasan@appscode.com> Signed-off-by: Vihang Mehta <vihang@gimletlabs.ai> Co-authored-by: Karl Persson <kalle.persson92@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Maria Ines Parnisari <maria.inesparnisari@okta.com> Co-authored-by: Rokibul Hasan <mdrokibulhasan18@gmail.com> Co-authored-by: José Padilla <jose.padilla@okta.com> Co-authored-by: Adrian Tam <adrian.tam@okta.com> Co-authored-by: Vihang Mehta <vihang@gimletlabs.ai> Co-authored-by: Joshua Jones <joshua.jones.software@gmail.com> Co-authored-by: Justin Cohen <justincoh@gmail.com> release: update changelog for release `v1.12.1` (openfga#2992) chore(deps): bump google.golang.org/grpc from 1.79.1 to 1.79.3 (openfga#2988) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Adrian Tam <adrian.tam@okta.com> chore: enforce minor version upgrade rule (openfga#2978) Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> perf: tuple validation performance tweaks (openfga#2984) fix: support URI schemes in OTLP endpoint configuration (openfga#2981) refactor: remove custom pipes and replace with channels (openfga#2977) chore(docs): added caching docs (openfga#2664) Co-authored-by: Saad Hussain <saad.hussain@okta.com> release: update changelog for release `v1.12.0` (openfga#2974) chore: update toolchain go to 1.26.1 (openfga#2975) fix: update toolchain go to 1.25.8 to address stdlib CVEs (openfga#2971) fix: correct swapped format args in DecodeParameterType error message (openfga#2961) Signed-off-by: Artem Muterko <artem@sopho.tech> Co-authored-by: Adrian Tam <adrian.tam@okta.com> perf: small tweaks to tuple validation functions (openfga#2963) Co-authored-by: Vic-Dev <vmichellej@gmail.com> test: add tests for condition parameter type any with complex context structures (openfga#2959) Signed-off-by: Artem Muterko <artem@sopho.tech> test: add functional test for ReadAssertions API endpoint (openfga#2960) Signed-off-by: Artem Muterko <artem@sopho.tech> Co-authored-by: Adrian Tam <adrian.tam@okta.com> make configurable the maximum received grpc message size (openfga#2952) Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> fix: set ExperimentalPipelineListObjects in experimentals by default (openfga#2957) fix: `cache_item_count` metric overcounting (openfga#2950) Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> chore(deps): bump the dependencies group across 1 directory with 7 updates (openfga#2953) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Adrian Tam <adrian.tam@okta.com> refactor: Make shadowV2Check final log clearer (openfga#2946) Move telemetry package to internal/telemetry (openfga#2938) Signed-off-by: Oleksandr Shestopal <ar.shestopal-oshegithub@gmail.com> Co-authored-by: Adrian Tam <adrian.tam@okta.com> chore(deps): bump the dependencies group across 1 directory with 3 updates (openfga#2956) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> fix: Resolve race condition in check reducers (openfga#2947) Co-authored-by: Adrian Tam <adrian.tam@okta.com> fix: gateway grpc client tls cert rotation (openfga#2951) Signed-off-by: Shashank Goel <goelshashank13@gmail.com> fix: disable LO pipeline if ff has it set for a store (openfga#2945) chore(deps): bump the dependencies group across 1 directory with 4 updates (openfga#2949) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> fix: various random fixes (openfga#2942) refactor: check using weighted graph (openfga#2816) Co-authored-by: Yissell Garma <yissell.garma@okta.com> release: update changelog for release `v1.11.6` (openfga#2939) feat: enable pipeline algorithm by default (openfga#2921) add tests to exercise reported bug (openfga#2934) Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Adrian Tam <adrian.tam@okta.com> refine AGENTS.md to make it more concise with higher value. (openfga#2936) Co-authored-by: Justin Cohen <justincoh@gmail.com> migrate grpc dial context to NewClient (openfga#2714) Co-authored-by: Adrian Tam <adrian.tam@okta.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> chore(deps): bump filippo.io/edwards25519 from 1.1.0 to 1.1.1 (openfga#2935) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

senojj requested a review from a team as a code owner March 20, 2026 16:08

senojj added the bug Something isn't working label Mar 20, 2026

Copilot AI review requested due to automatic review settings March 20, 2026 16:08

Copilot started reviewing on behalf of senojj March 20, 2026 16:10 View session

senojj added 2 commits March 20, 2026 12:10

capture panics in pipeline's base resolver, and return as errors.

117a6a0

update changelog

bb2634e

senojj force-pushed the lo-panic-handling branch from 2a9f4ed to bb2634e Compare March 20, 2026 16:11

Copilot AI reviewed Mar 20, 2026

View reviewed changes

senojj added 3 commits March 20, 2026 12:15

ensure error is not nil before sending

6a01e59

update panic recovery

eacef88

use %v to render panic messages

effd4fc

coderabbitai Bot reviewed Mar 20, 2026

View reviewed changes

Comment thread pkg/server/commands/reverseexpand/pipeline/resolver_base.go Outdated

Comment thread pkg/server/commands/reverseexpand/pipeline/resolver_base.go Outdated

Comment thread pkg/server/commands/reverseexpand/pipeline/resolver_core.go

senojj added 3 commits March 20, 2026 12:25

properly handle empty pointer in panic recovery

f0fa979

defer message.Done to ensure counters act properly after a panic

24fbec6

use a closure to capture the correct sent count value

7e64f48

coderabbitai Bot reviewed Mar 20, 2026

View reviewed changes

Comment thread internal/concurrency/panic.go

justincoh reviewed Mar 20, 2026

View reviewed changes

Comment thread internal/concurrency/panic.go

senojj requested a review from justincoh March 20, 2026 16:41

justincoh approved these changes Mar 20, 2026

View reviewed changes

senojj merged commit e538a88 into main Mar 20, 2026
20 of 22 checks passed

senojj deleted the lo-panic-handling branch March 20, 2026 16:48

Search code, repositories, users, issues, pull requests...

Uh oh!

Conversation

senojj commented Mar 20, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

What problem is being solved?

How is it being solved?

What changes are made to solve it?

References

Review Checklist

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Suggested reviewers

Poem

❌ Failed checks (1 warning)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

senojj commented Mar 20, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Mar 20, 2026 •

edited

Loading

codecov Bot commented Mar 20, 2026 •

edited

Loading