Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Conversation

itsmepetrov
Copy link

@itsmepetrov itsmepetrov commented Nov 4, 2024

Hi!

This PR fixes two issues:

  1. we noticed that sometimes we could see some log records repeated, like so:
    Processed: 500/407210 (took 0.84s)
    Processed: 26000/407210 (took 0.56s)
    Processed: 53500/407210 (took 0.51s)
    Processed: 53500/407210 (took 0.51s) <- double
    Processed: 98500/407210 (took 0.67s)
    Processed: 123500/407210 (took 0.53s)
    Processed: 123500/407210 (took 0.53s)
    Processed: 123500/407210 (took 0.53s) <- triple
    Processed: 152000/407210 (took 0.47s)
    
    it was caused by the pending key not being set correctly, so there was some sort of race condition;
  2. because of the race condition we could get ThrottlingException, get logs calls have a hard limit, and they don't qualify for a limit increase in AWS, so having two, three... tasks running simultaneously could cause the error quickly, the easiest way just to ignore it as it do not affect anything.

I've also added configuration for durationBetweenPolls and timeoutBeforeFirstLogs

@itsmepetrov
Copy link
Author

@ohookins could you please review it

@ohookins
Copy link
Contributor

@itsmepetrov thanks for the contribution! It's been a bit busy recently but I will review it soon :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Morty Proxy This is a proxified and sanitized view of the page, visit original site.