Internal packages (testcontainers, redis-worker and zod-worker)#1392

Merged

matt-aitken merged 118 commits into

mainfrom

internal-packages

Oct 8, 2024

Commits on Sep 25, 2024

Change some tsconfig
matt-aitken
committed
7f50b36
Copy full SHA for 7f50b36
Moved some things around
matt-aitken
committed
a3961da
Copy full SHA for a3961da
Added some packages
matt-aitken
committed
a236149
Copy full SHA for a236149
WIP on the RunQueue
matt-aitken
committed
416fd88
Copy full SHA for 416fd88

Commits on Sep 26, 2024

Fix for some imports
matt-aitken
committed
c3de286
Copy full SHA for c3de286
Key producer with some tests
matt-aitken
committed
08adc14
Copy full SHA for 08adc14
Removed the nv type from the keys… it’s not useful to do global queries
matt-aitken
committed
8d13edc
Copy full SHA for 8d13edc
Passing unit tests for all the public key producer functions
matt-aitken
committed
0fa6bbf
Copy full SHA for 0fa6bbf
Some basic tests passing for the RunQueue
matt-aitken
committed
07e76d7
Copy full SHA for 07e76d7
Simple enqueue test working
matt-aitken
committed
8f30533
Copy full SHA for 8f30533
Enqueue and dequeue for dev is working
matt-aitken
committed
ba8822f
Copy full SHA for ba8822f
Don’t log everything during the tests
matt-aitken
committed
63639d9
Copy full SHA for 63639d9
Enqueuing/dequeuing from the shared queue is working
matt-aitken
committed
c06731a
Copy full SHA for c06731a
Tests for getting a shared queue
matt-aitken
committed
2b8ec70
Copy full SHA for 2b8ec70

Commits on Sep 27, 2024

The key producer sharedQueue can now be named, to allow multiple separate queues
matt-aitken
committed
87364f1
Copy full SHA for 87364f1
The key producer uses the name of the queue as the input
matt-aitken
committed
4265e6d
Copy full SHA for 4265e6d
Extra info in the Prisma schema
matt-aitken
committed
e69c127
Copy full SHA for e69c127
Dequeuing a message gets the payload and sets the task concurrency all in one Lua script
matt-aitken
committed
5de384f
Copy full SHA for 5de384f
Adding more keys so we can read the concurrency from the queue
matt-aitken
committed
dc2d89e
Copy full SHA for dc2d89e
Setting the concurrency with dequeue and enquque is working
matt-aitken
committed
8af3003
Copy full SHA for 8af3003
Improved the tests and fixed some bugs
matt-aitken
committed
d2ed8ce
Copy full SHA for d2ed8ce
Acking is resetting the concurrencies
matt-aitken
committed
a6175a7
Copy full SHA for a6175a7

Commits on Sep 28, 2024

Check the key has been removed after acking
matt-aitken
committed
58daf85
Copy full SHA for 58daf85

Commits on Sep 29, 2024

Nacking is working
matt-aitken
committed
69b5c55
Copy full SHA for 69b5c55
Merge remote-tracking branch 'origin/main' into run-engine
matt-aitken
committed
e4993ec
Copy full SHA for e4993ec
Changed the package to CommonJS + Node10 so it works with Redlock
matt-aitken
committed
4309819
Copy full SHA for 4309819
Moved the database, otel and emails packages to be in internal-packages
matt-aitken
committed
dbbc81d
Copy full SHA for dbbc81d
Moved some Prisma code to the database package
matt-aitken
committed
82a0867
Copy full SHA for 82a0867
Started using the RunEngine for triggering
matt-aitken
committed
a20e180
Copy full SHA for a20e180

Commits on Sep 30, 2024

Progress on run engine triggering, first waitpoint code
matt-aitken
committed
f7d3c41
Copy full SHA for f7d3c41
Create a delay waitpoint
matt-aitken
committed
c196ca6
Copy full SHA for c196ca6
Moved ZodWorker to an internal package so it can be used in the run engine as well as the webapp
matt-aitken
committed
cf90b12
Copy full SHA for cf90b12
Web app now uses the zod worker package
matt-aitken
committed
99c3a96
Copy full SHA for 99c3a96
Added parseNaturalLanguageDuration to core/apps
matt-aitken
committed
c902ce4
Copy full SHA for c902ce4
internal-packages/zod-worker in the lockfile
matt-aitken
committed
cd36564
Copy full SHA for cd36564
Pass in the master queue, remove old rebalance workers code
matt-aitken
committed
79733ca
Copy full SHA for 79733ca
Add masterQueue to TaskRun
matt-aitken
committed
bb214d2
Copy full SHA for bb214d2
Fixed the tests
matt-aitken
committed
534b060
Copy full SHA for 534b060

Commits on Oct 1, 2024

Moved waitpoint code into the run engine, also the zod worker
matt-aitken
committed
3295597
Copy full SHA for 3295597
Completing waitpoints
matt-aitken
committed
f1232e5
Copy full SHA for f1232e5

Commits on Oct 2, 2024

An experiment to create a new test container with environment
matt-aitken
committed
d958679
Copy full SHA for d958679
More changes to triggering
matt-aitken
committed
cb29349
Copy full SHA for cb29349
Started testing triggering
matt-aitken
committed
f36b4b9
Copy full SHA for f36b4b9
Test for a run getting triggered and being enqueued
matt-aitken
committed
74d9523
Copy full SHA for 74d9523
Removed dequeueMessageInEnv
matt-aitken
committed
d632d48
Copy full SHA for d632d48
Update dev queue tests to use the shared queue function
matt-aitken
committed
ad864d0
Copy full SHA for ad864d0
Schema changes for TaskRunExecutionSnapshot
matt-aitken
committed
e6cb27d
Copy full SHA for e6cb27d
First execution snapshot when the run is created. Dequeue run function added to the engine
matt-aitken
committed
8a78b14
Copy full SHA for 8a78b14

Commits on Oct 3, 2024

Merge remote-tracking branch 'origin/main' into run-engine
Show description for 0bd237a
matt-aitken
committed
0bd237a
Copy full SHA for 0bd237a
Separate internal package for testcontainers so they can be used elsewhere
matt-aitken
committed
9927cdc
Copy full SHA for 9927cdc
Remove the simple queue and testcontainers from the run-engine. They’re going to be separate
matt-aitken
committed
ae1c23d
Copy full SHA for ae1c23d
Fix for the wrong path to the Prisma schem,a
matt-aitken
committed
9f96eb3
Copy full SHA for 9f96eb3
Added the testcontainers package to the run-engine
matt-aitken
committed
28a922c
Copy full SHA for 28a922c
redis-worker package, just a copy of the simple queue for now
matt-aitken
committed
c779991
Copy full SHA for c779991
The queue now uses Lua to enqueue dequeue
matt-aitken
committed
be98dbf
Copy full SHA for be98dbf

Commits on Oct 4, 2024

The queue now has a catalog and an invisible period after dequeuing
matt-aitken
committed
3d0bbac
Copy full SHA for 3d0bbac
Added a visibility timeout and acking, with tests
matt-aitken
committed
a7e114c
Copy full SHA for a7e114c
Added more Redis connection logging, deleted todos
matt-aitken
committed
75c2ad3
Copy full SHA for 75c2ad3
Visibility timeouts are now defined on the catalog and can be overridden when enqueuing
matt-aitken
committed
d806516
Copy full SHA for d806516
Dequeue multiple items at once
matt-aitken
committed
4a1fd0d
Copy full SHA for 4a1fd0d
Test for dequeuing multiple items
matt-aitken
committed
c8ce878
Copy full SHA for c8ce878
Export some types to be used elsewhere
matt-aitken
committed
cd2cfb9
Copy full SHA for cd2cfb9
Partial refactor of the processor
matt-aitken
committed
7d8db07
Copy full SHA for 7d8db07
First stab at a worker with concurrency and NodeWorkers
matt-aitken
committed
1c31aa0
Copy full SHA for 1c31aa0
Don’t have a default visibility timeout in the queue
matt-aitken
committed
fff3a35
Copy full SHA for fff3a35

Commits on Oct 5, 2024

Worker setup and processing items in a simple test
matt-aitken
committed
9e2b9e0
Copy full SHA for 9e2b9e0
Process jobs in parallel with retrying
matt-aitken
committed
b8d7380
Copy full SHA for b8d7380

Commits on Oct 6, 2024

Get the attempt when dequeuing
matt-aitken
committed
461ec42
Copy full SHA for 461ec42
Workers do exponential backoff
matt-aitken
committed
1a713b0
Copy full SHA for 1a713b0
Moved todos
matt-aitken
committed
eee5649
Copy full SHA for eee5649

Commits on Oct 8, 2024

DLQ functionality
matt-aitken
committed
0c25754
Copy full SHA for 0c25754
DLQ tests
matt-aitken
committed
aa84a69
Copy full SHA for aa84a69
Same cluster for all keys in the same queue
matt-aitken
committed
62c8b2c
Copy full SHA for 62c8b2c
Added DLQ tests
matt-aitken
committed
da02da9
Copy full SHA for da02da9
Whitespace
matt-aitken
committed
decf826
Copy full SHA for decf826
Redis pubsub to redrive from the worker
matt-aitken
committed
d5fdf7f
Copy full SHA for d5fdf7f
Fixed database paths
matt-aitken
committed
844cb77
Copy full SHA for 844cb77
Fix for path to zod-worker
matt-aitken
committed
25ee43b
Copy full SHA for 25ee43b
Fixes for typecheck errors, mostly with TS versions and module resolution
matt-aitken
committed
932dd5e
Copy full SHA for 932dd5e
Redlock required a patch
matt-aitken
committed
c0ad065
Copy full SHA for c0ad065
Merge remote-tracking branch 'origin/main' into run-engine
matt-aitken
committed
83beca2
Copy full SHA for 83beca2
Moved the new DB migrations to the new database package folder
matt-aitken
committed
2664c1d
Copy full SHA for 2664c1d
Remove the run-engine package
matt-aitken
committed
386ee00
Copy full SHA for 386ee00
Remove the RunEngine prisma schema changes
matt-aitken
committed
e1bba09
Copy full SHA for e1bba09
Delete triggerTaskV2
matt-aitken
committed
7d05c97
Copy full SHA for 7d05c97
Remove zodworker test script (no tests)
matt-aitken
committed
60e7b5f
Copy full SHA for 60e7b5f
Update test-containers readme
matt-aitken
committed
1254649
Copy full SHA for 1254649
Generate the client first
matt-aitken
committed
d328d71
Copy full SHA for d328d71
Use a specific version of the prisma package
matt-aitken
committed
1f6c86d
Copy full SHA for 1f6c86d
Generate the prisma client before running the unit tests
matt-aitken
committed
08cde55
Copy full SHA for 08cde55

Search code, repositories, users, issues, pull requests...

Uh oh!

Commits

Commits on Sep 19, 2024

Commits on Sep 20, 2024

Commits on Sep 22, 2024

Commits on Sep 23, 2024

Commits on Sep 24, 2024

Commits on Sep 25, 2024

Commits on Sep 26, 2024

Commits on Sep 27, 2024

Commits on Sep 28, 2024

Commits on Sep 29, 2024

Commits on Sep 30, 2024

Commits on Oct 1, 2024

Commits on Oct 2, 2024

Commits on Oct 3, 2024

Commits on Oct 4, 2024

Commits on Oct 5, 2024

Commits on Oct 6, 2024

Commits on Oct 8, 2024