Skip to content

Navigation Menu

Sign in
Appearance settings
Sign up
Appearance settings

Add MessageDb feed source - #181

#181
Merged
bartelink merged 44 commits into
jet:masterjet/propulsion:masterfrom
nordfjord:message-dbnordfjord/propulsion:message-dbCopy head branch name to clipboard
Nov 18, 2022
Merged

Add MessageDb feed source#181
bartelink merged 44 commits into
jet:masterjet/propulsion:masterfrom
nordfjord:message-dbnordfjord/propulsion:message-dbCopy head branch name to clipboard

Conversation

@nordfjord

@nordfjord nordfjord commented Nov 12, 2022

Copy link
Copy Markdown
Contributor

No description provided.

@nordfjord
nordfjord marked this pull request as ready for review November 12, 2022 14:27
Comment thread src/Propulsion.MessageDb/ReaderCheckpoint.fs Outdated
Comment thread src/Propulsion.MessageDb/ReaderCheckpoint.fs Outdated
Comment thread tests/Propulsion.MessageDb.Integration/Tests.fs
Comment thread src/Propulsion/Internal.fs
Comment thread tests/Propulsion.MessageDb.Integration/Tests.fs
Comment thread CHANGELOG.md Outdated
Comment thread README.md Outdated
Comment thread src/Propulsion.MessageDb/MessageDbSource.fs Outdated
Comment thread src/Propulsion.MessageDb/MessageDbSource.fs Outdated
Comment thread src/Propulsion.MessageDb/MessageDbSource.fs Outdated
Comment thread src/Propulsion.MessageDb/MessageDbSource.fs Outdated
Comment thread src/Propulsion.MessageDb/MessageDbSource.fs Outdated
Comment thread src/Propulsion.MessageDb/MessageDbSource.fs Outdated
Comment thread src/Propulsion.MessageDb/ReaderCheckpoint.fs Outdated
Comment thread src/Propulsion.MessageDb/ReaderCheckpoint.fs Outdated
Comment thread src/Propulsion.MessageDb/ReaderCheckpoint.fs Outdated
Comment thread src/Propulsion.MessageDb/ReaderCheckpoint.fs Outdated
Comment thread src/Propulsion.MessageDb/ReaderCheckpoint.fs Outdated
Comment thread src/Propulsion/Internal.fs
Comment thread tests/Propulsion.MessageDb.Integration/Program.fs Outdated
Comment thread tests/Propulsion.MessageDb.Integration/Propulsion.MessageDb.Integration.fsproj Outdated
Comment thread tests/Propulsion.MessageDb.Integration/Propulsion.MessageDb.Integration.fsproj Outdated
Comment thread tests/Propulsion.MessageDb.Integration/Propulsion.MessageDb.Integration.fsproj Outdated
@bartelink

Copy link
Copy Markdown
Collaborator

Might as well also wire this into propulsion checkpoint (see tools/propulsion.tool/program.fs)

Comment thread src/Propulsion.MessageDb/MessageDbSource.fs Outdated
Comment thread tests/Propulsion.MessageDb.Integration/Tests.fs Outdated
Comment thread tests/Propulsion.MessageDb.Integration/Tests.fs Outdated
@bartelink

Copy link
Copy Markdown
Collaborator

FYI I implemented an "all caught up" check in

member x.RunUntilCaughtUp(timeout : TimeSpan, statsInterval : IntervalTimer) = task {

which is used in Lambda reactors e.g. https://github.com/jet/dotnet-templates/blob/e063a661de8154336d29eba090e62157ed6dbe6d/equinox-shipping/Watchdog.Lambda/Function.fs#L97

This could probably be generalized somewhat (it's messy to do for Cosmos, but that impl would work for esdb, sss and also mdb)

For mdb, you might be able to do a slightly more optimized thing (though the monitor here has all the context to know the instant all feed positions have caught the tail, regardless of where those checkpoints live and/or whether they've actually been committed yet)

Probably the best resolution is to do a PR to move it somewhere more general if you ever end up needing it

@nordfjord

nordfjord commented Nov 15, 2022

Copy link
Copy Markdown
Contributor Author

FYI I implemented an "all caught up" check in

Interesting! I would love to learn more about this. I hadn't considered an "all caught up" check to be possible with an ever growing sequence of events 🤔.

Looking at the code it seems to use the isTail value of the batch which I naively set to false as a constant. I've fixed that now so we assume it's the tail when we reach an empty page

What's the context you're using this in?

Based on the comments I'm gathering that you have a processor running in lambda, and you run this lambda every minute via cron. I'm guessing this is a cost optimization strategy for you since after catching up you'd be billed for the lambda essentially doing nothing? So I guess it's in a context where

  1. you can afford up to a minute of delay in reactions
  2. idling costs are significant

Comment thread src/Propulsion.MessageDb/MessageDbSource.fs Outdated
Comment thread src/Propulsion.MessageDb/Readme.md Outdated
Comment thread tests/Propulsion.MessageDb.Integration/Propulsion.MessageDb.Integration.fsproj Outdated
Comment thread tools/Propulsion.Tool/Args.fs Outdated
@bartelink
bartelink force-pushed the master branch 3 times, most recently from 6f39a0e to 7f579b5 Compare November 18, 2022 15:51
Comment thread tools/Propulsion.Tool/Args.fs Outdated
Comment thread tools/Propulsion.Tool/Args.fs Outdated
Comment thread tools/Propulsion.Tool/Args.fs Outdated
@bartelink

Copy link
Copy Markdown
Collaborator

Lovely work, thanks!

@bartelink
bartelink merged commit a689ef3 into jet:master Nov 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Morty Proxy This is a proxified and sanitized view of the page, visit original site.