Hi!
Thanks for the reply. To clarify some things first :)

I'm using bkt for multiple things, but this in particular is for downloading multiple blocklists in different cronjobs.
Some of these blocklists are hosted on servers that only allow 1-2 request per 24/hours.

In this particular case - the memo script is only caching the result of an invocation (the exit code), not the file or stdout (even if one could use do that as well ofc)

This line just creates an invocation to a temporary file in a temporary directory
https://github.com/borestad/ci-utils/blob/f046c3b4fd62959656b2aef51312cfd6ee332930/memo#L69

https://github.com/borestad/ci-utils/blob/f046c3b4fd62959656b2aef51312cfd6ee332930/memo#L72
This line just checks if the file was ever created. If it was, then the TTL was expired.

Another thing is that I use often use git as a database, and touch/stat doesn't get carried over with git. In a ci-pipeline (I'm just using @action/cache to cache the bkt cache folder & restore it. Since it's only storing the status of invocations, it's usually just a couple hundred bytes to restore => instant.

wget --timestamping has proven to be extremely unreliable for me in real-world-scenarios. Proxys/CDN's not updating it properly, or some servers not support it at all (same with with etags or content-disposition). Too unreliable to use in production. I've had cases where the file on disk was set to the wrong year in the future, due to misconfigured servers, and in that case, the file was never update and had to be flushed manually.

The reason for live output was to be able to more easily debug really slow requests and see their progress.

All in all, this is experimental, but I wanted to show you how I'm personally are using bkt 🙂

Also, normally - the scripts looks something like this (with a download helper/curl wrapper)

memo --ttl=1800s -- timeout 120s dl http://someserver.com/file.txt

in pseudocode this might have looked like this

bkt --ttl=1800s --discard-failures --exit-direct-if-failed --delete-cache-if-fail --timeout 120s --expose-stats --dont-cache-stdout --stream-to-stdout -- dl http://someserver.com/file.txt

All the best
Johan

And I'm glad you're finding bkt useful and appreciate the input, to be clear! :)

That said I do think some of these usages are better served by other approaches - at least as-implemented there's no real need to use in your memo script. I'm not trying to say you can't use bkt, just that I think there may be simpler options for that task.

Of course the goal "hit a web server no more than once every twelve hours" is very well suited for bkt, and I'd love to better understand why you need the wrapper script at all for that task. I'd think bkt --ttl=12h -- wget ... and done.

Going through the flags in your pseduocode:

• --ttl=1800s - works today (though you could say 30m if you wanted, IMO that's more readable)
• --discard-failures - works today, though I'd be concerned about passing this when hitting a server that requires traffic throttling since it could lead to an increase in hits if there's a failure for some other reason than the server being unreachable
• --exit-direct-if-failed - what does this mean?
• --delete-cache-if-fail - same here, what would this be for? Hopefully Mechanism for invalidating (parts of) the cache #5 can address this, though I'd probably offload the exact invalidate-if-fail behavior to the caller
• --timeout 120s - I don't think this should be implemented by bkt, consider timeout (and cache that call with bkt)
• --expose-stats - I assume this is Record cache statistics #19 - unless you're asking to have them logged to stdout?
• --dont-cache-stdout - what's the motivation for this? I don't think we'd add this to bkt because you can discard stdout via a shell, e.g. bkt ... -- bash -c 'foo >/dev/null'
• --stream-to-stdout - This is of course Feature request: stream the subprocess' stdout and stderr while it's running #31 and will be the only behavior once implemented, it won't be an option

Haha, I should not write this much when tired and feverish (tonsillitis is horrible) because when I read the post again, it's clear that my thoughts wasn't clearly explained or even any good.

The pseudocode above was clearly not a good example, because as you said - i.e --timeout (as I'm using it as well), should not be a part of bkt, 100% agree! As is the ability to just pipe to /dev/null. Sorry for the confusion. Also the use of --discard-failures was just to show how I'm using it today (and it works great). Same with --delete-cache-if-fail and --exit-direct-if-failed - they were just there for emulation. I wish I just had never written that pseudocode, because it just confused things. Sorry about that.

So....now. To clearify again (hopefully) 😂

1. bkt is extremely useful for me
2. I wanted to show some of my use cases
3. The already existing issues Mechanism for invalidating (parts of) the cache #5 , Feature request: stream the subprocess' stdout and stderr while it's running #31 and Record cache statistics #19 would solve a lot for me without the need of a wrapper (which in the end is what I'm trying to avoid btw 🥰 )
4. For ci/scripting purposes, I'm lacking a way to be able to retrieve the cached result, without risking to execute the command again if the ttl has expired. (dry-run?)

For ci/scripting purposes, I'm lacking a way to be able to retrieve the cached result, without risking to execute the command again if the ttl has expired. (dry-run?)

Ok that sounds like #24 - the CI use case is an interesting one, it helps me understand why behavior like that might be useful.