[Access] fix event cache corruption#8448
[Access] fix event cache corruption#8448peterargue merged 15 commits intomasteronflow/flow-go:masterfrom peter/fix-event-cache-copyonflow/flow-go:peter/fix-event-cache-copyCopy head branch name to clipboard
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds deterministic execution-order sorting and deep-copying of events on storage retrieval; index layer stops re-sorting and returns storage-provided ordering. Separately, propagates per-transaction ComputationUsed through local provider, RPC conversion, REST models, and tests; adds/updates tests covering both areas. Changes
Sequence Diagram(s)mermaid Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Tip Issue Planner is now in beta. Read the docs and try it out! Share your feedback on Discord. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
There was a problem hiding this comment.
🧹 Nitpick comments (2)
storage/store/events.go (1)
212-218:ServiceEvents.ByBlockIDhas the same mutation vulnerability that was just fixed forEvents.
valis returned directly, so any caller that sorts or overwritesPayloadbytes on the returned slice will corrupt the cached entry for every concurrent/future reader — identical to theEventsbug described in the PR.♻️ Suggested fix (mirrors the `Events.ByBlockID` change)
func (e *ServiceEvents) ByBlockID(blockID flow.Identifier) ([]flow.Event, error) { val, err := e.cache.Get(e.db.Reader(), blockID) if err != nil { return nil, err } - return val, nil + result := make([]flow.Event, len(val)) + copy(result, val) + for i, event := range result { + payload := make([]byte, len(event.Payload)) + copy(payload, event.Payload) + result[i].Payload = payload + } + return result, nil }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@storage/store/events.go` around lines 212 - 218, ServiceEvents.ByBlockID currently returns the cached slice `val` directly from `e.cache.Get`, which lets callers mutate event entries (e.g., sort or modify Event.Payload) and corrupt the cache; update `ServiceEvents.ByBlockID` to create and return a deep copy of the events: allocate a new slice for the events and for each `flow.Event` copy the struct and clone the `Payload` byte slice before returning, mirroring the same defensive copy logic used in `Events.ByBlockID` so cached data from `e.cache.Get` remains immutable for other callers.storage/store/events_test.go (1)
259-262: Silent error drop in goroutine could hide test-setup failures.
if err != nil { return }silently swallows anyByBlockIDerror, so a broken store setup would make the race test pass vacuously. A common pattern that is safe from goroutines is to send errors to a buffered channel and drain it afterwg.Wait().♻️ Suggested improvement
+ errs := make(chan error, 10) for range 10 { wg.Add(1) go func() { defer wg.Done() got, err := eventsStore.ByBlockID(blockID) if err != nil { - return + errs <- err + return } sort.Slice(got, func(i, j int) bool { if got[i].TransactionIndex == got[j].TransactionIndex { return got[i].EventIndex < got[j].EventIndex } return got[i].TransactionIndex < got[j].TransactionIndex }) }() } wg.Wait() + close(errs) + for err := range errs { + require.NoError(t, err) + }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@storage/store/events_test.go` around lines 259 - 262, The goroutine currently swallows ByBlockID errors with "if err != nil { return }", which can hide setup failures; change the goroutine to send any error from eventsStore.ByBlockID (and any other internal failures) into a buffered channel (e.g., errCh := make(chan error, N)) instead of returning, and after wg.Wait() drain the channel and fail the test on the first non-nil error (t.Fatalf or require.NoError) so test failures in ByBlockID(blockID) are surfaced; reference the eventsStore.ByBlockID call, the blockID variable, the goroutine that invokes it, and vg/wg.Wait() when making the change.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@storage/store/events_test.go`:
- Around line 259-262: The goroutine currently swallows ByBlockID errors with
"if err != nil { return }", which can hide setup failures; change the goroutine
to send any error from eventsStore.ByBlockID (and any other internal failures)
into a buffered channel (e.g., errCh := make(chan error, N)) instead of
returning, and after wg.Wait() drain the channel and fail the test on the first
non-nil error (t.Fatalf or require.NoError) so test failures in
ByBlockID(blockID) are surfaced; reference the eventsStore.ByBlockID call, the
blockID variable, the goroutine that invokes it, and vg/wg.Wait() when making
the change.
In `@storage/store/events.go`:
- Around line 212-218: ServiceEvents.ByBlockID currently returns the cached
slice `val` directly from `e.cache.Get`, which lets callers mutate event entries
(e.g., sort or modify Event.Payload) and corrupt the cache; update
`ServiceEvents.ByBlockID` to create and return a deep copy of the events:
allocate a new slice for the events and for each `flow.Event` copy the struct
and clone the `Payload` byte slice before returning, mirroring the same
defensive copy logic used in `Events.ByBlockID` so cached data from
`e.cache.Get` remains immutable for other callers.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
| return events[i].EventIndex < events[j].EventIndex | ||
| } | ||
| return events[i].TransactionIndex < events[j].TransactionIndex | ||
| }) |
There was a problem hiding this comment.
all callers realistically will want events in execution order. currently they are sorted by Transaction ID
| // No error returns are expected during normal operation. | ||
| func (e *Events) ByBlockIDTransactionID(blockID flow.Identifier, txID flow.Identifier) ([]flow.Event, error) { | ||
| events, err := e.ByBlockID(blockID) | ||
| events, err := e.cache.Get(e.db.Reader(), blockID) |
There was a problem hiding this comment.
stopped using ByBlockID so we only copy the events that are actually returned
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
storage/store/events.go (1)
227-234:⚠️ Potential issue | 🟠 Major
ServiceEvents.ByBlockIDreturns the cached slice directly — same vulnerability class as the one fixed here
Events.ByBlockIDwas patched because returning the cached slice allowed callers to corrupt the cache via in-place sorts or payload overwrites.ServiceEvents.ByBlockIDhas the identical pattern (return val, nil) andServiceEvents.BatchStorecaches the caller-providedeventsslice directly at line 221. Any caller that mutates service-event payloads or sorts the returned slice in place will silently corrupt the cache.🛡️ Proposed fix — apply the same deep-copy idiom
func (e *ServiceEvents) ByBlockID(blockID flow.Identifier) ([]flow.Event, error) { val, err := e.cache.Get(e.db.Reader(), blockID) if err != nil { return nil, err } - return val, nil + result := make([]flow.Event, len(val)) + for i, event := range val { + result[i] = copyEvent(event) + } + return result, nil }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@storage/store/events.go` around lines 227 - 234, ServiceEvents.ByBlockID currently returns the cached slice directly which allows callers to mutate the cache; change ByBlockID to return a deep copy of the cached []flow.Event (same deep-copy idiom used in Events.ByBlockID) so callers can sort or modify the slice without touching the cache. Locate ServiceEvents.ByBlockID and ServiceEvents.BatchStore in the diff (BatchStore caches the caller-provided events) and implement copying by allocating a new slice of the same length and copying each flow.Event element (not just the slice header) into it before returning.
🧹 Nitpick comments (1)
storage/store/events.go (1)
63-81:BatchStorecaches events without sorting, diverging from theretrievepath
retrievesorts loaded events by(TransactionIndex, EventIndex)before they enter the cache, butBatchStoreinsertscombinedEventsdirectly without sorting. If a caller ever passesblockEventsin non-execution order, the cache hit path returns a different ordering than a subsequent DB-reload (cache miss) path. The invariant is implicit and undocumented.Applying the same sort before
cache.Insertmakes both paths consistent regardless of input order.♻️ Proposed defensive fix
+ sort.Slice(combinedEvents, func(i, j int) bool { + if combinedEvents[i].TransactionIndex == combinedEvents[j].TransactionIndex { + return combinedEvents[i].EventIndex < combinedEvents[j].EventIndex + } + return combinedEvents[i].TransactionIndex < combinedEvents[j].TransactionIndex + }) + storage.OnCommitSucceed(batch, func() { e.cache.Insert(blockID, combinedEvents) })🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@storage/store/events.go` around lines 63 - 81, The BatchStore path builds combinedEvents from blockEvents and inserts them into the cache without sorting, causing a cache/DB ordering mismatch with the retrieve path; before calling e.cache.Insert(blockID, combinedEvents) sort combinedEvents the same way retrieve does (by Event.TransactionIndex then Event.EventIndex) — e.g., after building combinedEvents and before storage.OnCommitSucceed's callback, perform a stable sort on combinedEvents comparing TransactionIndex then EventIndex so cache.Insert and retrieve produce the same ordering regardless of blockEvents input order.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@storage/store/events_test.go`:
- Around line 259-262: The goroutine in the test currently swallows errors from
eventsStore.ByBlockID by doing a bare return; change it to report the failure to
the test harness by calling t.Errorf (or t.Fatalf) with the error details
instead of returning so the test fails visibly; update the anonymous goroutine
that calls eventsStore.ByBlockID to use t.Errorf("ByBlockID error: %v", err)
when err != nil so failures are signalled to the test runner.
In `@storage/store/events.go`:
- Line 28: Fix the typo in the comment that reads "events are stored orderd by
[blockID, txID, txIndex, eventIndex]" by changing "orderd" to "ordered" so the
comment correctly reads "events are stored ordered by [blockID, txID, txIndex,
eventIndex]"; locate this exact comment string in events.go and update it.
---
Outside diff comments:
In `@storage/store/events.go`:
- Around line 227-234: ServiceEvents.ByBlockID currently returns the cached
slice directly which allows callers to mutate the cache; change ByBlockID to
return a deep copy of the cached []flow.Event (same deep-copy idiom used in
Events.ByBlockID) so callers can sort or modify the slice without touching the
cache. Locate ServiceEvents.ByBlockID and ServiceEvents.BatchStore in the diff
(BatchStore caches the caller-provided events) and implement copying by
allocating a new slice of the same length and copying each flow.Event element
(not just the slice header) into it before returning.
---
Nitpick comments:
In `@storage/store/events.go`:
- Around line 63-81: The BatchStore path builds combinedEvents from blockEvents
and inserts them into the cache without sorting, causing a cache/DB ordering
mismatch with the retrieve path; before calling e.cache.Insert(blockID,
combinedEvents) sort combinedEvents the same way retrieve does (by
Event.TransactionIndex then Event.EventIndex) — e.g., after building
combinedEvents and before storage.OnCommitSucceed's callback, perform a stable
sort on combinedEvents comparing TransactionIndex then EventIndex so
cache.Insert and retrieve produce the same ordering regardless of blockEvents
input order.
| got, err := eventsStore.ByBlockID(blockID) | ||
| if err != nil { | ||
| return | ||
| } |
There was a problem hiding this comment.
Silently swallowing errors in goroutine hides unexpected failures
If ByBlockID returns an error, the goroutine returns without signalling the test, so the race-detection run can pass silently even when something is wrong. t.Errorf is goroutine-safe and should be used here instead of a bare return.
🐛 Proposed fix
got, err := eventsStore.ByBlockID(blockID)
if err != nil {
- return
+ t.Errorf("ByBlockID returned unexpected error: %v", err)
+ return
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| got, err := eventsStore.ByBlockID(blockID) | |
| if err != nil { | |
| return | |
| } | |
| got, err := eventsStore.ByBlockID(blockID) | |
| if err != nil { | |
| t.Errorf("ByBlockID returned unexpected error: %v", err) | |
| return | |
| } |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@storage/store/events_test.go` around lines 259 - 262, The goroutine in the
test currently swallows errors from eventsStore.ByBlockID by doing a bare
return; change it to report the failure to the test harness by calling t.Errorf
(or t.Fatalf) with the error details instead of returning so the test fails
visibly; update the anonymous goroutine that calls eventsStore.ByBlockID to use
t.Errorf("ByBlockID error: %v", err) when err != nil so failures are signalled
to the test runner.
| return events[i].EventIndex < events[j].EventIndex | ||
| } | ||
| return events[i].TransactionIndex < events[j].TransactionIndex | ||
| }) |
There was a problem hiding this comment.
moved this into the storage implementation
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@storage/store/events_test.go`:
- Around line 257-270: The test spawns a goroutine that calls require.NoError
(which calls FailNow) inside the goroutine; move the assertion out of the
spawned goroutine by having the goroutine send back results and errors (e.g.,
via a channel or slice protected by mutex) from eventsStore.ByBlockID and the
sorted slice, then in the main test goroutine receive/collect the error and call
require.NoError (or t.Fatalf) there before continuing; keep the in-goroutine
sort of got but return got and err from that goroutine (reference
eventsStore.ByBlockID and the in-place sort block) so all test-failing
assertions run on the main test goroutine.
---
Duplicate comments:
In `@storage/store/events_test.go`:
- Around line 675-690: In the goroutine inside the test (the anonymous func that
calls svcEventsStore.ByBlockID), don't silently return on error; instead report
the failure with t.Errorf so errors from ByBlockID are surfaced during the
concurrent test run. Replace the bare "if err != nil { return }" inside the
goroutine with a call to t.Errorf (including err and context such as blockID) so
the test records the failure safely from the goroutine while keeping the rest of
the concurrent logic (sort.Slice, wg.Done) intact.
There was a problem hiding this comment.
🧹 Nitpick comments (1)
engine/access/state_stream/backend/backend_events_test.go (1)
105-118: Ordering assumption betweensetupLocalStorageandsubscribe'sexpectedEventsis implicitly coupledThe sort comparator change is correct — (TransactionIndex, EventIndex) now mirrors the storage layer's documented execution order.
One fragility worth noting:
subscribe(line 378) buildsexpectedEventsby iteratings.blockEvents[b.ID()]in its original (unsorted) order, andrequireEventsResponsecompares viaassert.Equal(order-sensitive). WhensetupLocalStorage()is in effect, the mock returns execution-sorted events. The test only passes if test fixtures happen to already produce events in (TransactionIndex, EventIndex) order — a silent, implicit assumption.Consider making this explicit in the
setupLocalStoragepath:♻️ Suggested fix to make ordering explicit in the expected events construction
- var expectedEvents flow.EventsList - for _, event := range s.blockEvents[b.ID()] { - if test.filter.Match(event) { - expectedEvents = append(expectedEvents, event) - } - } + // When local storage is used, events are returned in execution order + // (TransactionIndex, then EventIndex). Build expectedEvents in the same order. + srcEvents := s.blockEvents[b.ID()] + sorted := make([]flow.Event, len(srcEvents)) + copy(sorted, srcEvents) + sort.Slice(sorted, func(i, j int) bool { + if sorted[i].TransactionIndex == sorted[j].TransactionIndex { + return sorted[i].EventIndex < sorted[j].EventIndex + } + return sorted[i].TransactionIndex < sorted[j].TransactionIndex + }) + var expectedEvents flow.EventsList + for _, event := range sorted { + if test.filter.Match(event) { + expectedEvents = append(expectedEvents, event) + } + }This would need to be conditional on whether local storage is active, or you could factor it into a per-test ordering helper. Alternatively, ensure
BackendExecutionDataSuite.SetupTestalways generates events in execution order and add an assertion on the fixture data.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@engine/access/state_stream/backend/backend_events_test.go` around lines 105 - 118, The test assumes event order implicitly; make expectedEvents match storage's execution order when setupLocalStorage() is active by sorting s.blockEvents before building expectedEvents (or by using the same comparator used earlier: TransactionIndex then EventIndex); update the subscribe test path that builds expectedEvents (referencing subscribe, expectedEvents, requireEventsResponse) to either conditionally sort events when setupLocalStorage() is enabled or call a shared helper that returns execution-sorted events, and/or add an assertion in BackendExecutionDataSuite.SetupTest that fixture s.blockEvents are already execution-ordered to avoid the fragile implicit coupling.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@engine/access/state_stream/backend/backend_events_test.go`:
- Around line 105-118: The test assumes event order implicitly; make
expectedEvents match storage's execution order when setupLocalStorage() is
active by sorting s.blockEvents before building expectedEvents (or by using the
same comparator used earlier: TransactionIndex then EventIndex); update the
subscribe test path that builds expectedEvents (referencing subscribe,
expectedEvents, requireEventsResponse) to either conditionally sort events when
setupLocalStorage() is enabled or call a shared helper that returns
execution-sorted events, and/or add an assertion in
BackendExecutionDataSuite.SetupTest that fixture s.blockEvents are already
execution-ordered to avoid the fragile implicit coupling.
|
|
||
| // events are stored ordered by [blockID, txID, txIndex, eventIndex] | ||
| // sort them into execution order | ||
| sort.Slice(events, func(i, j int) bool { |
There was a problem hiding this comment.
Maybe extract into a function to be reused, as it's a canonical order. It's repeated below. If execution node also have this sorting, maybe that can also be reused.
| // events are stored ordered by [blockID, txID, txIndex, eventIndex] | ||
| // sort them into execution order |
There was a problem hiding this comment.
| // events are stored ordered by [blockID, txID, txIndex, eventIndex] | |
| // sort them into execution order | |
| // We want events sorted by [txIndex, eventIndex] (execution order). | |
| // Events are keyed by [blockID, txID, txIndex, eventIndex], so reading by txID | |
| // returns them in the correct order. However, reading by blockID returns events | |
| // sorted by txID (a hash) first, which is unpredictable. We must re-sort here. |
|
|
||
| result := make([]flow.Event, len(events)) | ||
| for i, event := range events { | ||
| result[i] = copyEvent(event) |
There was a problem hiding this comment.
I think it’s the caller’s responsibility not to mutate the returned data. If we start copying here, we’d need to do the same throughout the storage layer for consistency.
If CcfEventsToJsonEvents is the caller that mutates the event in place, can we add the copying there instead?
There was a problem hiding this comment.
I'm ok to remove the copying if that's consistent with the rest of the storage layer. I just worry that subtle bugs like this will be easy to introduce.
If CcfEventsToJsonEvents is the caller that mutates the event in place, can we add the copying there instead?
we're careful about this now, I'm just worried about future bugs
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (1)
storage/store/events_test.go (1)
262-264: Same redundanttxIDLow[0] = 0x00no-op as inTestByBlockID_SortsEventsByExecutionOrder.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@storage/store/events_test.go` around lines 262 - 264, The assignment txIDLow[0] = 0x00 is redundant because a zeroed flow.Identifier is already initialized to 0; remove that no-op and keep only the meaningful initialization for txIDHigh (txIDHigh[0] = 0xFF) or explicitly set both if you prefer clarity—update the test variables txIDLow and txIDHigh in events_test.go (the declarations of txIDLow and txIDHigh) to remove the redundant txIDLow[0] = 0x00 assignment.
🧹 Nitpick comments (4)
storage/store/events_test.go (2)
143-201: Consider extracting the near-identical sort-order test body into a shared table-driven helper.
TestByBlockID_SortsEventsByExecutionOrderandTestServiceEventByBlockID_SortsEventsByExecutionOrdershare identical fixture construction,freshStorebypass logic, and assertions — only the store constructor (NewEventsvs.NewServiceEvents) andBatchStoreargument type differ. A small local helper would eliminate the duplication and make future changes (e.g., adding a third event) single-site.Also applies to: 247-304
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@storage/store/events_test.go` around lines 143 - 201, Extract the duplicated test logic from TestByBlockID_SortsEventsByExecutionOrder and TestServiceEventByBlockID_SortsEventsByExecutionOrder into a small table-driven helper that builds the same fixtures and assertions once and is parameterized by the store constructor and the BatchStore call signature; e.g., create a local helper testSortEventsByExecOrder(t, createStore func(m metrics.Collector, db storage.DB) interface{}, batchStoreFunc func(store interface{}, lctx lockctx.Context, blockID flow.Identifier, lists []interface{}, rw storage.ReaderBatchWriter) error) that constructs txIDLow/txIDHigh, evtTx0/evtTx1A/evtTx1B, invokes BatchStore through the provided batchStoreFunc, creates a fresh store via createStore to force a DB load, calls ByBlockID (via type assertion inside the helper) and asserts the three events are in execution order; then replace both tests to call this helper with NewEvents/appropriate []flow.EventsList and NewServiceEvents/appropriate []flow.EventsList types.
159-161:txIDLow[0] = 0x00is a no-op — consider removing it or using a non-zero value.
var txIDLow flow.Identifierzero-initializes the[32]bytearray, so assigning0x00to index 0 leaves the value unchanged. The comment "first in DB key order" is still accurate (the zero ID will sort beforetxIDHigh), but the explicit assignment is misleading — it implies intentional mutation while doing nothing.Either drop the line entirely, or to make the test's intent maximally clear, assign a distinct non-zero value:
♻️ Proposed fix
var txIDLow, txIDHigh flow.Identifier -txIDLow[0] = 0x00 // first in DB key order +txIDLow[0] = 0x01 // first in DB key order txIDHigh[0] = 0xFF // second in DB key order🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@storage/store/events_test.go` around lines 159 - 161, The statement txIDLow[0] = 0x00 is a no-op because var txIDLow flow.Identifier zero-initializes the [32]byte; either remove that line to avoid misleading intent, or set a distinct non-zero value (e.g., txIDLow[0] = 0x01) so the test clearly uses two different IDs; update the code around txIDLow and txIDHigh to reflect the chosen approach (txIDHigh[0] = 0xFF can remain unchanged).utils/slices/slices.go (1)
55-73: Consider removing the mutation side-effect instead of just documenting it.The CAUTION comment is a good stopgap, but given the PR's explicit goal of eliminating in-place-mutation bugs, it's worth also fixing
AreStringSlicesEqualto not mutate its inputs. The"slices"stdlib package (already imported) provides everything needed — this would also let you drop the"sort"import and the CAUTION warning entirely.The project targets Go 1.25.1, so
slices.Clone,slices.Sort, andslices.Equal(all available since Go 1.21.0) are fully supported.♻️ Proposed non-mutating refactor
-// AreStringSlicesEqual returns true if the two string slices are equal. -// -// CAUTION: this function performs inplace sorts of the input slices. +// AreStringSlicesEqual returns true if the two string slices contain +// the same elements (order-independent). The input slices are not modified. func AreStringSlicesEqual(a, b []string) bool { if len(a) != len(b) { return false } - - sort.Strings(a) - sort.Strings(b) - - for i := range a { - if a[i] != b[i] { - return false - } - } - - return true + aCopy := slices.Clone(a) + bCopy := slices.Clone(b) + slices.Sort(aCopy) + slices.Sort(bCopy) + return slices.Equal(aCopy, bCopy) }And remove the now-unused
"sort"import:import ( "slices" - "sort" "golang.org/x/exp/constraints" )🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@utils/slices/slices.go` around lines 55 - 73, AreStringSlicesEqual mutates its inputs by calling sort.Strings; change it to clone both inputs using slices.Clone, sort the clones with slices.Sort, then compare with slices.Equal so the original slices aren't modified; update the function body accordingly and remove the "sort" import and the CAUTION comment since mutation no longer occurs.engine/access/rest/http/routes/transactions_test.go (1)
1808-1828: Consider setting a non-zeroComputationUsedintransactionResultFixture.
transactionResultFixtureremains at the Go zero value forComputationUsed, so every test that uses it (e.g. "Get by ID with results" at line 170, "get by block ID with expanded results" at line 676) still asserts"computation_used": "0". This only validates the zero path for the expanded-results endpoints at the REST layer — the same behavior as before this PR.Setting a non-zero value here would trivially cover those code paths without duplicating any logic.
♻️ Suggested update
func transactionResultFixture(tx flow.TransactionBody) *accessmodel.TransactionResult { txID := tx.ID() cid := unittest.IdentifierFixture() return &accessmodel.TransactionResult{ Status: flow.TransactionStatusSealed, StatusCode: 1, Events: []flow.Event{ unittest.EventFixture( unittest.Event.WithEventType(flow.EventAccountCreated), unittest.Event.WithTransactionIndex(0), unittest.Event.WithEventIndex(0), unittest.Event.WithTransactionID(txID), unittest.Event.WithPayload([]byte{}), ), }, - ErrorMessage: "", - BlockID: tx.ReferenceBlockID, - CollectionID: cid, - TransactionID: txID, + ErrorMessage: "", + BlockID: tx.ReferenceBlockID, + CollectionID: cid, + TransactionID: txID, + ComputationUsed: 1234, } }The corresponding expected JSON strings in tests using this fixture would then need
"computation_used": "1234".🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@engine/access/rest/http/routes/transactions_test.go` around lines 1808 - 1828, transactionResultFixture currently leaves accessmodel.TransactionResult.ComputationUsed at the Go zero value, causing tests to only validate the zero path; update transactionResultFixture (the function named transactionResultFixture) to set ComputationUsed to a non-zero value (e.g., 1234) in the returned accessmodel.TransactionResult, and then update the expected JSON in tests that use this fixture (e.g., "Get by ID with results" and "get by block ID with expanded results") to assert "computation_used": "1234".
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@docs/plans/2026-02-22-computation-used-in-transaction-result-design.md`:
- Around line 18-23: Update the doc note to clarify that all three
methods—TransactionResult, TransactionResultByIndex, and
TransactionResultsByBlockID—now include ComputationUsed in their returned
results; mention that TransactionResult was updated to set ComputationUsed from
the existing LightTransactionResult (txResult.ComputationUsed) so readers
understand every method now carries ComputationUsed and there’s no partial fix
implied.
In `@storage/store/events_test.go`:
- Around line 143-304: Events.ByBlockID currently returns cached slices/objects
directly (and ByBlockIDTransactionID / ByBlockIDTransactionIndex comments warn
about mutation), so callers can mutate the cache; implement deep-copy semantics
inside Events.ByBlockID to return a fresh slice and deep-cloned flow.Event
objects (including copying nested payloads/byte slices) so the cache cannot be
mutated; update or remove the CAUTION comments accordingly; then add tests
TestByBlockIDReturnsCopy (mutate returned slice and event fields/payload and
assert cache value unchanged) and TestByBlockIDConcurrentMutationIsRaceFree
(concurrent goroutines mutating returned values do not affect cached store or
race detector failures) to validate the fix.
---
Duplicate comments:
In `@storage/store/events_test.go`:
- Around line 262-264: The assignment txIDLow[0] = 0x00 is redundant because a
zeroed flow.Identifier is already initialized to 0; remove that no-op and keep
only the meaningful initialization for txIDHigh (txIDHigh[0] = 0xFF) or
explicitly set both if you prefer clarity—update the test variables txIDLow and
txIDHigh in events_test.go (the declarations of txIDLow and txIDHigh) to remove
the redundant txIDLow[0] = 0x00 assignment.
---
Nitpick comments:
In `@engine/access/rest/http/routes/transactions_test.go`:
- Around line 1808-1828: transactionResultFixture currently leaves
accessmodel.TransactionResult.ComputationUsed at the Go zero value, causing
tests to only validate the zero path; update transactionResultFixture (the
function named transactionResultFixture) to set ComputationUsed to a non-zero
value (e.g., 1234) in the returned accessmodel.TransactionResult, and then
update the expected JSON in tests that use this fixture (e.g., "Get by ID with
results" and "get by block ID with expanded results") to assert
"computation_used": "1234".
In `@storage/store/events_test.go`:
- Around line 143-201: Extract the duplicated test logic from
TestByBlockID_SortsEventsByExecutionOrder and
TestServiceEventByBlockID_SortsEventsByExecutionOrder into a small table-driven
helper that builds the same fixtures and assertions once and is parameterized by
the store constructor and the BatchStore call signature; e.g., create a local
helper testSortEventsByExecOrder(t, createStore func(m metrics.Collector, db
storage.DB) interface{}, batchStoreFunc func(store interface{}, lctx
lockctx.Context, blockID flow.Identifier, lists []interface{}, rw
storage.ReaderBatchWriter) error) that constructs txIDLow/txIDHigh,
evtTx0/evtTx1A/evtTx1B, invokes BatchStore through the provided batchStoreFunc,
creates a fresh store via createStore to force a DB load, calls ByBlockID (via
type assertion inside the helper) and asserts the three events are in execution
order; then replace both tests to call this helper with NewEvents/appropriate
[]flow.EventsList and NewServiceEvents/appropriate []flow.EventsList types.
- Around line 159-161: The statement txIDLow[0] = 0x00 is a no-op because var
txIDLow flow.Identifier zero-initializes the [32]byte; either remove that line
to avoid misleading intent, or set a distinct non-zero value (e.g., txIDLow[0] =
0x01) so the test clearly uses two different IDs; update the code around txIDLow
and txIDHigh to reflect the chosen approach (txIDHigh[0] = 0xFF can remain
unchanged).
In `@utils/slices/slices.go`:
- Around line 55-73: AreStringSlicesEqual mutates its inputs by calling
sort.Strings; change it to clone both inputs using slices.Clone, sort the clones
with slices.Sort, then compare with slices.Equal so the original slices aren't
modified; update the function body accordingly and remove the "sort" import and
the CAUTION comment since mutation no longer occurs.
ℹ️ Review info
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (10)
docs/plans/2026-02-22-computation-used-in-transaction-result-design.mdengine/access/rest/common/models/transaction.goengine/access/rest/http/routes/transactions_test.goengine/access/rpc/backend/transactions/provider/local.goengine/access/rpc/backend/transactions/transactions_functional_test.goengine/common/rpc/convert/transaction_result.goengine/common/rpc/convert/transaction_result_test.gostorage/store/events.gostorage/store/events_test.goutils/slices/slices.go
🚧 Files skipped from review as they are similar to previous changes (1)
- storage/store/events.go
| The `TransactionResult` method (lookup by block ID + tx ID) does not set `ComputationUsed` in | ||
| its return value. The `txResult` variable is already a `LightTransactionResult` with | ||
| `ComputationUsed` available — just add `ComputationUsed: txResult.ComputationUsed`. | ||
|
|
||
| The other two methods (`TransactionResultByIndex`, `TransactionResultsByBlockID`) are already | ||
| fixed on this branch. |
There was a problem hiding this comment.
Minor doc nit: clarify that all three methods are fixed.
Lines 22–23 say TransactionResultByIndex and TransactionResultsByBlockID "are already fixed on this branch", implying only TransactionResult is new. In the final committed state all three methods carry ComputationUsed, so a reader encountering this document may be confused about what the PR actually changed. A short update would prevent that ambiguity.
📝 Suggested update
-The other two methods (`TransactionResultByIndex`, `TransactionResultsByBlockID`) are already
-fixed on this branch.
+All three methods (`TransactionResult`, `TransactionResultByIndex`, `TransactionResultsByBlockID`)
+are updated on this branch.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| The `TransactionResult` method (lookup by block ID + tx ID) does not set `ComputationUsed` in | |
| its return value. The `txResult` variable is already a `LightTransactionResult` with | |
| `ComputationUsed` available — just add `ComputationUsed: txResult.ComputationUsed`. | |
| The other two methods (`TransactionResultByIndex`, `TransactionResultsByBlockID`) are already | |
| fixed on this branch. | |
| The `TransactionResult` method (lookup by block ID + tx ID) does not set `ComputationUsed` in | |
| its return value. The `txResult` variable is already a `LightTransactionResult` with | |
| `ComputationUsed` available — just add `ComputationUsed: txResult.ComputationUsed`. | |
| All three methods (`TransactionResult`, `TransactionResultByIndex`, `TransactionResultsByBlockID`) | |
| are updated on this branch. |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@docs/plans/2026-02-22-computation-used-in-transaction-result-design.md`
around lines 18 - 23, Update the doc note to clarify that all three
methods—TransactionResult, TransactionResultByIndex, and
TransactionResultsByBlockID—now include ComputationUsed in their returned
results; mention that TransactionResult was updated to set ComputationUsed from
the existing LightTransactionResult (txResult.ComputationUsed) so readers
understand every method now carries ComputationUsed and there’s no partial fix
implied.
| // TestByBlockID_SortsEventsByExecutionOrder verifies that events loaded from the | ||
| // database (cache miss path) are returned sorted by (TransactionIndex, | ||
| // EventIndex) regardless of the order imposed by the storage key, which is | ||
| // [blockID, txID, txIndex, eventIndex]. When two transactions have txIDs whose | ||
| // byte order differs from their execution order the DB iteration would return | ||
| // them in txID order; the retrieve function must re-sort them. | ||
| func TestByBlockID_SortsEventsByExecutionOrder(t *testing.T) { | ||
| lockManager := storage.NewTestingLockManager() | ||
| dbtest.RunWithDB(t, func(t *testing.T, db storage.DB) { | ||
| m := metrics.NewNoopCollector() | ||
| eventsStore := store.NewEvents(m, db) | ||
|
|
||
| blockID := unittest.IdentifierFixture() | ||
|
|
||
| // txIDLow sorts before txIDHigh in byte (DB key) order, but its | ||
| // TransactionIndex is higher, so in execution order it comes second. | ||
| var txIDLow, txIDHigh flow.Identifier | ||
| txIDLow[0] = 0x00 // first in DB key order | ||
| txIDHigh[0] = 0xFF // second in DB key order | ||
|
|
||
| // execution tx 0 uses txIDHigh; execution tx 1 uses txIDLow | ||
| evtTx0 := unittest.EventFixture( | ||
| unittest.Event.WithTransactionID(txIDHigh), | ||
| unittest.Event.WithTransactionIndex(0), | ||
| unittest.Event.WithEventIndex(0), | ||
| ) | ||
| evtTx1A := unittest.EventFixture( | ||
| unittest.Event.WithTransactionID(txIDLow), | ||
| unittest.Event.WithTransactionIndex(1), | ||
| unittest.Event.WithEventIndex(0), | ||
| ) | ||
| evtTx1B := unittest.EventFixture( | ||
| unittest.Event.WithTransactionID(txIDLow), | ||
| unittest.Event.WithTransactionIndex(1), | ||
| unittest.Event.WithEventIndex(1), | ||
| ) | ||
|
|
||
| err := unittest.WithLock(t, lockManager, storage.LockInsertEvent, func(lctx lockctx.Context) error { | ||
| return db.WithReaderBatchWriter(func(rw storage.ReaderBatchWriter) error { | ||
| return eventsStore.BatchStore(lctx, blockID, []flow.EventsList{{evtTx0}, {evtTx1A, evtTx1B}}, rw) | ||
| }) | ||
| }) | ||
| require.NoError(t, err) | ||
|
|
||
| // Use a fresh store to force a DB load (bypasses the cache populated by BatchStore). | ||
| freshStore := store.NewEvents(m, db) | ||
| got, err := freshStore.ByBlockID(blockID) | ||
| require.NoError(t, err) | ||
| require.Len(t, got, 3) | ||
|
|
||
| // Events must come back in execution order: txIndex 0, then 1 (eventIndex 0, 1). | ||
| require.Equal(t, uint32(0), got[0].TransactionIndex) | ||
| require.Equal(t, uint32(0), got[0].EventIndex) | ||
| require.Equal(t, uint32(1), got[1].TransactionIndex) | ||
| require.Equal(t, uint32(0), got[1].EventIndex) | ||
| require.Equal(t, uint32(1), got[2].TransactionIndex) | ||
| require.Equal(t, uint32(1), got[2].EventIndex) | ||
| }) | ||
| } | ||
|
|
||
| // TestServiceEventStoreRetrieve verifies basic store and retrieval for ServiceEvents. | ||
| func TestServiceEventStoreRetrieve(t *testing.T) { | ||
| lockManager := storage.NewTestingLockManager() | ||
| dbtest.RunWithDB(t, func(t *testing.T, db storage.DB) { | ||
| m := metrics.NewNoopCollector() | ||
| svcEventsStore := store.NewServiceEvents(m, db) | ||
|
|
||
| blockID := unittest.IdentifierFixture() | ||
| txID := unittest.IdentifierFixture() | ||
|
|
||
| evt1 := unittest.EventFixture( | ||
| unittest.Event.WithTransactionID(txID), | ||
| unittest.Event.WithTransactionIndex(0), | ||
| unittest.Event.WithEventIndex(0), | ||
| ) | ||
| evt2 := unittest.EventFixture( | ||
| unittest.Event.WithTransactionID(txID), | ||
| unittest.Event.WithTransactionIndex(1), | ||
| unittest.Event.WithEventIndex(0), | ||
| ) | ||
|
|
||
| err := unittest.WithLock(t, lockManager, storage.LockInsertServiceEvent, func(lctx lockctx.Context) error { | ||
| return db.WithReaderBatchWriter(func(rw storage.ReaderBatchWriter) error { | ||
| return svcEventsStore.BatchStore(lctx, blockID, []flow.Event{evt1, evt2}, rw) | ||
| }) | ||
| }) | ||
| require.NoError(t, err) | ||
|
|
||
| got, err := svcEventsStore.ByBlockID(blockID) | ||
| require.NoError(t, err) | ||
| require.Len(t, got, 2) | ||
| require.Contains(t, got, evt1) | ||
| require.Contains(t, got, evt2) | ||
|
|
||
| // Test loading from database (cache miss path). | ||
| freshStore := store.NewServiceEvents(m, db) | ||
| got, err = freshStore.ByBlockID(blockID) | ||
| require.NoError(t, err) | ||
| require.Len(t, got, 2) | ||
| require.Contains(t, got, evt1) | ||
| require.Contains(t, got, evt2) | ||
| }) | ||
| } | ||
|
|
||
| // TestServiceEventByBlockID_SortsEventsByExecutionOrder verifies that service events | ||
| // loaded from the database (cache miss path) are returned sorted by (TransactionIndex, | ||
| // EventIndex) regardless of the DB key order, which is keyed by txID byte order. | ||
| // Before the fix, the sort was missing for ServiceEvents (unlike Events which already | ||
| // had it), causing events to be returned in DB key order on a cache miss. | ||
| func TestServiceEventByBlockID_SortsEventsByExecutionOrder(t *testing.T) { | ||
| lockManager := storage.NewTestingLockManager() | ||
| dbtest.RunWithDB(t, func(t *testing.T, db storage.DB) { | ||
| m := metrics.NewNoopCollector() | ||
| svcEventsStore := store.NewServiceEvents(m, db) | ||
|
|
||
| blockID := unittest.IdentifierFixture() | ||
|
|
||
| // txIDLow sorts before txIDHigh in byte (DB key) order, but its | ||
| // TransactionIndex is higher, so in execution order it comes second. | ||
| var txIDLow, txIDHigh flow.Identifier | ||
| txIDLow[0] = 0x00 // first in DB key order | ||
| txIDHigh[0] = 0xFF // second in DB key order | ||
|
|
||
| // execution tx 0 uses txIDHigh; execution tx 1 uses txIDLow | ||
| evtTx0 := unittest.EventFixture( | ||
| unittest.Event.WithTransactionID(txIDHigh), | ||
| unittest.Event.WithTransactionIndex(0), | ||
| unittest.Event.WithEventIndex(0), | ||
| ) | ||
| evtTx1A := unittest.EventFixture( | ||
| unittest.Event.WithTransactionID(txIDLow), | ||
| unittest.Event.WithTransactionIndex(1), | ||
| unittest.Event.WithEventIndex(0), | ||
| ) | ||
| evtTx1B := unittest.EventFixture( | ||
| unittest.Event.WithTransactionID(txIDLow), | ||
| unittest.Event.WithTransactionIndex(1), | ||
| unittest.Event.WithEventIndex(1), | ||
| ) | ||
|
|
||
| err := unittest.WithLock(t, lockManager, storage.LockInsertServiceEvent, func(lctx lockctx.Context) error { | ||
| return db.WithReaderBatchWriter(func(rw storage.ReaderBatchWriter) error { | ||
| return svcEventsStore.BatchStore(lctx, blockID, []flow.Event{evtTx0, evtTx1A, evtTx1B}, rw) | ||
| }) | ||
| }) | ||
| require.NoError(t, err) | ||
|
|
||
| // Use a fresh store to force a DB load (bypasses the cache populated by BatchStore). | ||
| freshStore := store.NewServiceEvents(m, db) | ||
| got, err := freshStore.ByBlockID(blockID) | ||
| require.NoError(t, err) | ||
| require.Len(t, got, 3) | ||
|
|
||
| // Events must come back in execution order: txIndex 0, then 1 (eventIndex 0, 1). | ||
| require.Equal(t, uint32(0), got[0].TransactionIndex) | ||
| require.Equal(t, uint32(0), got[0].EventIndex) | ||
| require.Equal(t, uint32(1), got[1].TransactionIndex) | ||
| require.Equal(t, uint32(0), got[1].EventIndex) | ||
| require.Equal(t, uint32(1), got[2].TransactionIndex) | ||
| require.Equal(t, uint32(1), got[2].EventIndex) | ||
| }) | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check if the copy/race-free tests exist anywhere in the repo
rg -rn "TestByBlockIDReturnsCopy\|TestByBlockIDConcurrentMutationIsRaceFree" --type=goRepository: onflow/flow-go
Length of output: 40
🏁 Script executed:
#!/bin/bash
# Find the Events.ByBlockID implementation
rg -A 20 "func.*ByBlockID.*\(blockID" storage/store/events.go --type=go | head -50Repository: onflow/flow-go
Length of output: 1603
The primary fix is not implemented and lacks test coverage.
The stated PR objective — returning a full deep copy from Events.ByBlockID — is not present in the implementation. The method returns the cached value directly without copying. The expected validation tests TestByBlockIDReturnsCopy and TestByBlockIDConcurrentMutationIsRaceFree do not exist in the codebase. The CAUTION comments in ByBlockIDTransactionID and ByBlockIDTransactionIndex ("The returned events cannot be safely modified. Make a copy first.") indicate the cache remains vulnerable to mutation. Implement the deep-copy logic and add tests validating that mutations to returned slices and event payloads do not corrupt the cache.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@storage/store/events_test.go` around lines 143 - 304, Events.ByBlockID
currently returns cached slices/objects directly (and ByBlockIDTransactionID /
ByBlockIDTransactionIndex comments warn about mutation), so callers can mutate
the cache; implement deep-copy semantics inside Events.ByBlockID to return a
fresh slice and deep-cloned flow.Event objects (including copying nested
payloads/byte slices) so the cache cannot be mutated; update or remove the
CAUTION comments accordingly; then add tests TestByBlockIDReturnsCopy (mutate
returned slice and event fields/payload and assert cache value unchanged) and
TestByBlockIDConcurrentMutationIsRaceFree (concurrent goroutines mutating
returned values do not affect cached store or race detector failures) to
validate the fix.
| var events []flow.Event | ||
| err := operation.LookupEventsByBlockID(r, blockID, &events) | ||
|
|
||
| // We want events sorted by [txIndex, eventIndex] (execution order). |
There was a problem hiding this comment.
I wonder whether we want to verify the execution order somewhere?
I mean what guarantees the input events is sorted in [txIndex, eventIndex], it should be, because it's techinically done by event counter. But I couldn't find any sanity check to ensure that.
Does it make sense to add a sanity check in events.BatchStore?
We are iterating the events anyway, maybe we can add a sanity check to ensure the cached order is also in [txIndex, eventIndex].
| err := operation.LookupEventsByBlockID(r, blockID, &events) | ||
|
|
||
| // We want events sorted by [txIndex, eventIndex] (execution order). | ||
| // Events are keyed by [blockID, txID, txIndex, eventIndex], so reading by txID |
There was a problem hiding this comment.
The reason we added txID in the key is to be able to retrieve events by txID, and we do have a RetrieveEvents method does that.
However, the RetrieveEvents is unused. And eventStore.ByBlockIDTransactionID API is actually implemented by reading all events of the block and filter by txID, I guess this is to make use of the block level cache, but in that case, we could have removed the txID from the index key, and then we can also get rid of this sorting.
Just thoughts, not blocking this PR.
There was a problem hiding this comment.
I think it's OK. at this point all of the ANs/ENs already have existing data so we can't really change it until the next spork. We can add it as an improvement before the next spork, but I also see it as fairly low priority since event lists are typically less than 100 events per block
zhangchiqing
left a comment
There was a problem hiding this comment.
Looks good. Just had one more comment to add a sanity check in events.BatchStore
fxamacker
left a comment
There was a problem hiding this comment.
Besides the things Leo already pointed out it looks good.
zhangchiqing
left a comment
There was a problem hiding this comment.
The sanity check can be added by another PR.
Problem
Events.ByBlockIDreturned the cached[]flow.Eventslice directly. Since Go slices share the underlying array when assigned by value, any caller that mutated the returned slice was silently corrupting the cache for all future and concurrent readers.When getting events for a block, the events were sorted by transaction id instead of transaction index. This meant that they were not in execution order. The
EventIndexmodule used by the Access API sorted the returned slice into execution order. Since the underlying array was stored in the cache, this resulted in unprotected concurrent modification of the slice.When making concurrent calls to
GetTransactionResultsByBlockIDandGetTransactionsByBlockIDfor the same block, the events included withinGetTransactionResultsByBlockIDwere occasionally out of order, duplicated or missing. This was easy to detect by inspecting the event indexes.Fix
Update the events storage to sort block events by tx index and event index before adding them into the cache, and removing the sort done by the Access API.
I also added warnings to the godocs about making a copy of the slices if modification are needed.