re2: New release concurrency system#1804

Merged

ericallam merged 38 commits into

mainfrom

re2-reserve-concurrency

Mar 19, 2025

Commits on Mar 19, 2025

wip reserve concurrency system
ericallam
committed
8b3551c
Copy full SHA for 8b3551c
dequeue message
ericallam
committed
16546f1
Copy full SHA for 16546f1
ack
ericallam
committed
a5b2b39
Copy full SHA for a5b2b39
improve the dead letter queue stuff
ericallam
committed
880538f
Copy full SHA for 880538f
fixed some key producer tests
ericallam
committed
555d991
Copy full SHA for 555d991
the run engine now works with the new reserve concurrency system
ericallam
committed
1a233ef
Copy full SHA for 1a233ef
Update delays to use a redis worker and work with the new reserve concurrency system
ericallam
committed
e9fa4ce
Copy full SHA for e9fa4ce
remove unused method
ericallam
committed
2b6ce16
Copy full SHA for 2b6ce16
Add batchId to the delayed enqueueRun call
ericallam
committed
1362d82
Copy full SHA for 1362d82
WIP release concurrency queue
ericallam
committed
d11d491
Copy full SHA for d11d491
Get all the release concurrency queue tests passing
ericallam
committed
c8b9972
Copy full SHA for c8b9972
improve the consumer of the concurrency queue
ericallam
committed
bf41703
Copy full SHA for bf41703
Correctly use the new release concurrency queue in the run engine
ericallam
committed
b7ddf20
Copy full SHA for b7ddf20
If max tokens is 0, then don't do releasings
ericallam
committed
cce402d
Copy full SHA for cce402d
Add configuration for the release concurrency queue
ericallam
committed
de9e296
Copy full SHA for de9e296
remove project and task current concurrency tracking
ericallam
committed
cf3b238
Copy full SHA for cf3b238
WIP new reacquire concurrency system
ericallam
committed
fd9b0bf
Copy full SHA for fd9b0bf
Implement reserve concurrency clearing when the child run is acked
ericallam
committed
61c0834
Copy full SHA for 61c0834
go to QUEUE_EXECUTING state if reacquiring concurrency doesn't work
ericallam
committed
2048b72
Copy full SHA for 2048b72
Upgrade vitest in the run-engine package
ericallam
committed
063651c
Copy full SHA for 063651c
Remove reserve concurrency system from run queue
ericallam
committed
8c66ec3
Copy full SHA for 8c66ec3
Remove reserve concurrency system from run engine
ericallam
committed
7d11e82
Copy full SHA for 7d11e82
WIP run engine systems
ericallam
committed
1b61b95
Copy full SHA for 1b61b95
move startRunAttempt to RunAttemptSystem
ericallam
committed
717cec8
Copy full SHA for 717cec8
more system work
ericallam
committed
63b43ff
Copy full SHA for 63b43ff
Delayed run system
ericallam
committed
363f066
Copy full SHA for 363f066
ttl system
ericallam
committed
a4581f1
Copy full SHA for a4581f1
waiting for worker system
ericallam
committed
7866e95
Copy full SHA for 7866e95
More tests passing, fixed the heartbeat issue
ericallam
committed
67d74f0
Copy full SHA for 67d74f0
Fix more tests
ericallam
committed
42fb5d0
Copy full SHA for 42fb5d0
Implement checkpoint tests, handle dequeuing QUEUED_EXECUTING runs
ericallam
committed
29371e9
Copy full SHA for 29371e9
implement the QUEUED_EXECUTING dequeuing, and creating a checkpoint while the run is in QUEUED_EXECUTING state by saving the EXECUTING_WITH_WAITPOINTS snapshotId as the previousSnapshotId on the QU…
Show description for e7c8f94
ericallam
committed
e7c8f94
Copy full SHA for e7c8f94
fixed the create checkpoint valid snapshot logic
ericallam
committed
b170a62
Copy full SHA for b170a62
Added updated execution states chart and updated readme
ericallam
committed
f9c7e95
Copy full SHA for f9c7e95
Use releaserId in case we don't end up using run IDs
ericallam
committed
7eaf81a
Copy full SHA for 7eaf81a
Implement release concurrency system
ericallam
committed
28b3ed0
Copy full SHA for 28b3ed0
move the release concurrency queue into the release concurrency system, make it disabled by default, configure the run engine in the webapp with env vars
ericallam
committed
e5ea9cb
Copy full SHA for e5ea9cb
Fixed failing redis worker tests
ericallam
committed
38e1887
Copy full SHA for 38e1887