Hey @adamnovak ! Thank you for this and the linked PR. I worry that this will cause existing stored images to be ignored. Any idea on how we can diverge less from the previously naming scheme?

One problem that's being solved here is name collisions: it's possible with the previous naming scheme to have two distinct images (mostly pairs like a/b_c and a_b/c) share a cache key.

The obvious way to re-use existing cached images would be to look and see if the files already exist under the old scheme. Maybe that's OK even if there were possible collisions, since we won't create any new collisions and if people have images cached they must have been working.

The obvious way to re-use existing cached images would be to look and see if the files already exist under the old scheme.

yes, please and thank you

OK, the problem with checking the old schemes' locations (and in general with the new scheme I guess) is that names generated under the old schemes can collide with names generated under the new scheme.

For example, somename_/_someimage:s_latest under the cwl-utils 0.42 scheme is somename___someimage_s_latest.sif. But that's the same value you get from somename_someimage/latest under the new unified scheme.

I don't see a great way out of this; the old schemes can generate almost any path without a / in it.

I think I need to start versioning the cache directory, so we can check the cwl-utils 0.42 path and the cwltool 3.2.20260720092025 path, without worrying about whether those might also be paths reachable by the new scheme, and then we can check the new scheme path under a v2 subdirectory, where we can't possibly mistake the new files for something leftover from the old scheme.

Actually, just dumping everything into a v2 subdirectory of the cache directory won't work well, because the cache directory is often just the current working directory, and we don't want to go creating a bunch of directories in there.

We also have those cwltool test workflows that want to pull the saved SIF files by name: https://github.com/common-workflow-language/cwltool/pull/2284/changes#diff-fe637a76528e6390b81f3303265bed4573355ca63517a31491701f2097b30574

They would start looking even weirder if we were prefixing these with something like v2.

I think the right answer might be using a directory name that doesn't look extremely weird when dropped in the current directory, as the disambiguating prefix.

The problem with that is it creates a tripping hazard: people will see the cwl_image_cache_v2 path or whatever and then try and pass that as CWL_SINGULARITY_CACHE it won't work.

I think what I really want to do is make both re-using images cached under the old scheme best effort. I can generate the paths under the old schemes, and then check if they would be possible to generate under the new scheme, or if they contain substrings (like _ not part of ___ or _s_) that the new scheme can't generate. Most images should. Then I can check the old paths if they aren't at risk of colliding with new paths, and ignore them otherwise. So almost all the existing cached images will be re-used.

This doesn't help the problem where existing images might be at paths the new scheme wants to use. But they'd have to be excitingly named (with user names or repo names starting or ending with things like _ or s_), and if someone runs into the problem it won't recur after they clear their cache.