c8d/push: don't automatically pick platform if multiple candidates #48737

laurazard · Oct 23, 2024

Related: #48731

- What I did

Previously, when pushing an image that contains more than a single platform, if no platform was specified and there are more than one but not all platforms available, we either

"automatically" pick the host/engine's platform, if available, or
error out if there are multiple candidates but none are the host/engine's platform.

This commit drops the special handling for the case where the engine's platform image is available, instead always returning an error in the case that not all platforms are available and there are multiple candidates for pushing.

- How to verify it

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

Previously, when pushing an image that contains more than a single platform, if no platform was specified and there are more than one but not all platforms available, we either - "automatically" pick the host/engine's platform, if available, or - error out if there are multiple candidates but none are the host/engine's platform. This commit drops the special handling for the case where the engine's platform image is available, instead always returning an error in the case that not all platforms are available and there are multiple candidates for pushing. Signed-off-by: Laura Brehm <laurabrehm@hey.com>

thaJeztah · Oct 24, 2024

Tried some variations with this PR; notes below 😅

👉 not all of these scenarios are related to this PR, but it was purely to test these variations ❤️

1. Pushing a multi-platform image with dangling references and a single variant present that matches the host

In this case, we should try to push, otherwise fall back to the "pre-containerd" behavior (which pulled multi-platform images as single platform)

Try if we can push (registry may accept dangling) 👈 do we do this?
Try if we can cross-mount 👈 do we do this?
There's only a single platform present, we'll push that image
Inform the user that we pushed, but discarded information (such as attestations) and that the digest will be different (as it's only the single platform)

✅ this looks to be working as expected
❓ do we do steps 1. and 2. ?

$ docker rmi alpine:latest localhost:5001/myimage:latest
Untagged: alpine:latest
Untagged: localhost:5001/myimage:latest
Deleted: sha256:beefdbd8a1da6d2915566fde36db9db0b524eb737fc57cd1367effd16dc0d06d

$ docker pull alpine
$ docker tag alpine localhost:5001/myimage:latest
$ docker image ls --tree localhost:5001/myimage:latest
IMAGE                           ID             DISK USAGE   CONTENT SIZE   USED
localhost:5001/myimage:latest   beefdbd8a1da       13.6MB         4.09MB
├─ linux/amd64                  33735bd63cf8           0B             0B
├─ linux/arm/v6                 50f635c8b04d           0B             0B
├─ linux/arm/v7                 f2f82d424957           0B             0B
├─ linux/arm64/v8               9cee2b382fe2       13.6MB         4.09MB
├─ linux/386                    b3e87f642f5c           0B             0B
├─ linux/ppc64le                c7a6800e3dc5           0B             0B
├─ linux/riscv64                80cde017a105           0B             0B
└─ linux/s390x                  2b5b26e09ca2           0B             0B

$ docker push localhost:5001/myimage:latest
The push refers to repository [localhost:5001/myimage]
cf04c63912e1: Layer already exists
latest: digest: sha256:9cee2b382fe2412cd77d5d437d15a93da8de373813621f2e4d406e3df0cf0e7c size: 528

i Info → Not all multiplatform-content is present and only the available single-platform image was pushed
         sha256:beefdbd8a1da6d2915566fde36db9db0b524eb737fc57cd1367effd16dc0d06d -> sha256:9cee2b382fe2412cd77d5d437d15a93da8de373813621f2e4d406e3df0cf0e7c

$ docker buildx imagetools inspect localhost:5001/myimage:latest
Name:      localhost:5001/myimage:latest
MediaType: application/vnd.docker.distribution.manifest.v2+json
Digest:    sha256:9cee2b382fe2412cd77d5d437d15a93da8de373813621f2e4d406e3df0cf0e7c

2. Pushing a multi-platform image with dangling references and a single variant present that's not the native platform

ℹ️ This case is similar to the above, but the available variant does not match the host.

In this case, we should try to push, otherwise fall back to the "pre-containerd" behavior (which pulled multi-platform images as single platform)

Try if we can push (registry may accept dangling) 👈 do we do this?
Try if we can cross-mount 👈 do we do this?
There's only a single platform present, regardless of what platform that is, we'll push that image
Inform the user that we pushed, but discarded information (such as attestations) and that the digest will be different (as it's only the single platform)

✅ this looks to be working as expected
❓ do we do steps 1. and 2. ?

$ docker rmi alpine:latest localhost:5001/myimage:latest
Untagged: alpine:latest
Untagged: localhost:5001/myimage:latest
Deleted: sha256:beefdbd8a1da6d2915566fde36db9db0b524eb737fc57cd1367effd16dc0d06d

$ docker pull --platform=linux/s390x alpine
$ docker tag alpine localhost:5001/myimage:latest
$ docker image ls --tree localhost:5001/myimage:latest
IMAGE                           ID             DISK USAGE   CONTENT SIZE   USED
localhost:5001/myimage:latest   beefdbd8a1da       11.9MB         3.46MB
├─ linux/amd64                  33735bd63cf8           0B             0B
├─ linux/arm/v6                 50f635c8b04d           0B             0B
├─ linux/arm/v7                 f2f82d424957           0B             0B
├─ linux/arm64/v8               9cee2b382fe2           0B             0B
├─ linux/386                    b3e87f642f5c           0B             0B
├─ linux/ppc64le                c7a6800e3dc5           0B             0B
├─ linux/riscv64                80cde017a105           0B             0B
└─ linux/s390x                  2b5b26e09ca2       11.9MB         3.46MB

$ docker push localhost:5001/myimage:latest
The push refers to repository [localhost:5001/myimage]
df110db6acd6: Pushed
latest: digest: sha256:2b5b26e09ca2856f50ac88312348d26c1ac4b8af1df9f580e5cf465fd76e3d4d size: 528

i Info → Not all multiplatform-content is present and only the available single-platform image was pushed
         sha256:beefdbd8a1da6d2915566fde36db9db0b524eb737fc57cd1367effd16dc0d06d -> sha256:2b5b26e09ca2856f50ac88312348d26c1ac4b8af1df9f580e5cf465fd76e3d4d

$ docker buildx imagetools inspect localhost:5001/myimage:latest
Name:      localhost:5001/myimage:latest
MediaType: application/vnd.docker.distribution.manifest.v2+json
Digest:    sha256:2b5b26e09ca2856f50ac88312348d26c1ac4b8af1df9f580e5cf465fd76e3d4d

3. Pushing a multi-platform image with dangling references and multiple variants present

ℹ️ this is the issue this PR is solving; #48731

In this case, we have a multi-platform that's incomplete (dangling references);

Try if we can push (registry may accept dangling) 👈 do we do this?
Try if we can cross-mount 👈 do we do this?
If the above failed, we must ask the user what to do, because the intent is ambiguous:
- does the user want a complete image? In that case they must pull the remaining variants
- does the user want a single-platform image? In that case they must use the --platform flag to push one of the variants as a single-platform image (discarding attestations). ℹ️ This is equivalent to the pre-containerd situation.
- 💡 in future we may offer a docker image convert (name TBD) command to extract one or more variants and create a new multi-platform image from that; Add image convert (client-side) docker/cli#5249

✅ does produce an error now (🎉)
⚠️ but doesn't seem to trigger the hint on the CLI side; do we know why? ❓ https://github.com/docker/cli/blob/61baf2a3d984a1e59fcadb38e28d77b571434b88/cli/command/image/push.go#L173-L183
❓ do we do steps 1. and 2. ?

$ docker pull --platform=linux/s390x alpine
Using default tag: latest
latest: Pulling from library/alpine
df110db6acd6: Download complete
Digest: sha256:beefdbd8a1da6d2915566fde36db9db0b524eb737fc57cd1367effd16dc0d06d
Status: Image is up to date for alpine:latest
docker.io/library/alpine:latest

$ docker image ls --tree localhost:5001/myimage:latest
IMAGE                           ID             DISK USAGE   CONTENT SIZE   USED
localhost:5001/myimage:latest   beefdbd8a1da       25.5MB         7.55MB
├─ linux/amd64                  33735bd63cf8           0B             0B
├─ linux/arm/v6                 50f635c8b04d           0B             0B
├─ linux/arm/v7                 f2f82d424957           0B             0B
├─ linux/arm64/v8               9cee2b382fe2       13.6MB         4.09MB
├─ linux/386                    b3e87f642f5c           0B             0B
├─ linux/ppc64le                c7a6800e3dc5           0B             0B
├─ linux/riscv64                80cde017a105           0B             0B
└─ linux/s390x                  2b5b26e09ca2       11.9MB         3.46MB

$ docker push localhost:5001/myimage:latest
The push refers to repository [localhost:5001/myimage]
multiple matching manifests found but no specific platform requested

$ docker buildx imagetools inspect localhost:5001/myimage:latest
ERROR: localhost:5001/myimage:latest: not found

4. Pushing a multi-platform image with a single platform (no dangling references)

In this case, we have a multi-platform that only has a single platform. The image is complete because all variants (which is only one) are present.

✅ this looks to be working as expected; the image is pushed as a multi-arch image, and preserves attestations

$ echo -e 'FROM alpine\nRUN echo foo > bar.txt' | docker build --platform=linux/s390x -t localhost:5001/builtimage:latest -

$ docker image ls --tree localhost:5001/builtimage:latest
IMAGE                              ID             DISK USAGE   CONTENT SIZE   USED
localhost:5001/builtimage:latest   f498676c34ef       3.46MB         3.46MB
└─ linux/s390x                     c428486d9358       3.46MB         3.46MB

$ docker push localhost:5001/builtimage:latest
The push refers to repository [localhost:5001/builtimage]
0d5da46e1ac3: Pushed
0a3341740f0b: Pushed
df110db6acd6: Pushed
latest: digest: sha256:f498676c34ef2d98438a93a9552609c623d728f03d20d91b21ace5ea9b26092a size: 855

$ docker buildx imagetools inspect localhost:5001/builtimage:latest
Name:      localhost:5001/builtimage:latest
MediaType: application/vnd.oci.image.index.v1+json
Digest:    sha256:f498676c34ef2d98438a93a9552609c623d728f03d20d91b21ace5ea9b26092a

Manifests:
  Name:        localhost:5001/builtimage:latest@sha256:c428486d9358f3a8c515a818715f586e45a0b0ea06ada398dcaf623f8628144a
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/s390x

  Name:        localhost:5001/builtimage:latest@sha256:63edd1c26bd77894e0710af6c9169f7731b91e166f7cdb54a9f8f0631e94f6fc
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations:
    vnd.docker.reference.digest: sha256:c428486d9358f3a8c515a818715f586e45a0b0ea06ada398dcaf623f8628144a
    vnd.docker.reference.type:   attestation-manifest

5. Pushing a multi-platform image with a single platform, but specifying `--platform`

ℹ️ the image is the same as the image from the previous step ☝️

ℹ️ this is the issue reported in docker/cli#5571

In this case, we have a multi-platform that only has a single platform. The image is complete because all variants (which is only one) are present.

However, in this case;

the user specifies --platform when pushing.
the specified platform matches the platform of the image 💡

✅ / ⚠️ the image is pushed as a single-platform image, discarding attestations
❓ do we want --platform to force "dismantling" the image?
❓ should we have a --only-platform flag to "force"? and --platform to "select" (and a no-op in this case) would that make it clearer?

$ docker tag localhost:5001/builtimage:latest localhost:5001/builtimage2:latest

$ docker push --platform=linux/s390x localhost:5001/builtimage2:latest
i Info → Using --platform pushes only the specified platform manifest of a multi-platform image index.
         Other components, like attestations, will not be included.
         To push the complete multi-platform image, remove the --platform flag.

The push refers to repository [localhost:5001/builtimage2]
0a3341740f0b: Mounted from builtimage
df110db6acd6: Mounted from builtimage
latest: digest: sha256:c428486d9358f3a8c515a818715f586e45a0b0ea06ada398dcaf623f8628144a size: 668

$ docker buildx imagetools inspect localhost:5001/builtimage2:latest
Name:      localhost:5001/builtimage2:latest
MediaType: application/vnd.oci.image.manifest.v1+json
Digest:    sha256:c428486d9358f3a8c515a818715f586e45a0b0ea06ada398dcaf623f8628144a

vvoland

#48731 (comment)

vvoland · Oct 25, 2024

Try if we can push (registry may accept dangling) 👈 do we do this?

Try if we can cross-mount 👈 do we do this?

Yes, when the --platform is not specified, we always do the "full" push first and only if it fails, we fallback to the reduced push:

moby/daemon/containerd/image_push.go

Lines 171 to 175 in b04246c

    
           err = remotes.PushContent(ctx, pusher, target, store, limiter, platforms.All, handlerWrapper) 
        
           if err != nil { 
        
           	// If push failed because of a missing content, no specific platform was requested 
        
           	// and the target is an index, select a platform-specific manifest to push instead. 
        
           	if cerrdefs.IsNotFound(err) && containerdimages.IsIndexType(target.MediaType) && platform == nil {

laurazard force-pushed the c8d-push-no-platform branch from 857418b to 1d5fa6f Compare October 23, 2024 16:50

laurazard force-pushed the c8d-push-no-platform branch from 1d5fa6f to 9d35d87 Compare October 23, 2024 17:18

thaJeztah added area/distribution area/images containerd-integration Issues and PRs related to containerd integration process/cherry-pick/27.x status/2-code-review labels Oct 23, 2024

thaJeztah added this to the 28.0.0 milestone Oct 23, 2024

thaJeztah assigned laurazard Oct 23, 2024

thaJeztah added the area/ux label Oct 24, 2024

vvoland reviewed Oct 24, 2024

View reviewed changes

thompson-shaun modified the milestones: 28.0.0, 29.0.0 Jan 17, 2025

thompson-shaun requested a review from thaJeztah January 17, 2025 19:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

c8d/push: don't automatically pick platform if multiple candidates #48737

c8d/push: don't automatically pick platform if multiple candidates #48737

laurazard commented Oct 23, 2024

thaJeztah commented Oct 24, 2024 •

edited

Loading

vvoland left a comment

vvoland commented Oct 25, 2024 •

edited

Loading

Search code, repositories, users, issues, pull requests...

c8d/push: don't automatically pick platform if multiple candidates #48737

Are you sure you want to change the base?

c8d/push: don't automatically pick platform if multiple candidates #48737

Conversation

laurazard commented Oct 23, 2024

thaJeztah commented Oct 24, 2024 • edited Loading

1. Pushing a multi-platform image with dangling references and a single variant present that matches the host

2. Pushing a multi-platform image with dangling references and a single variant present that's not the native platform

3. Pushing a multi-platform image with dangling references and multiple variants present

4. Pushing a multi-platform image with a single platform (no dangling references)

5. Pushing a multi-platform image with a single platform, but specifying --platform

vvoland left a comment

Choose a reason for hiding this comment

vvoland commented Oct 25, 2024 • edited Loading

thaJeztah commented Oct 24, 2024 •

edited

Loading

5. Pushing a multi-platform image with a single platform, but specifying `--platform`

vvoland commented Oct 25, 2024 •

edited

Loading