Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

BuildKit builder with inline cache causes daemon crash #44918

Copy link
Copy link
Closed
#44920
Closed
BuildKit builder with inline cache causes daemon crash#44918
#44920
Copy link
Assignees
tonistiigi
Labels
area/builder/buildkitBuildBuildkind/bugBugs are bugs. The cause may or may not be known at triage time so debugging may be needed.Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed.priority/P1Important: P1 issues are a top priority and a must-have for the next release.Important: P1 issues are a top priority and a must-have for the next release.status/0-triageversion/23.0
Milestone
23.0.1

Description

@felipecrs
felipecrs
opened on Feb 3, 2023
Issue body actions

Description

After upgrading to Docker v23.0.0, most of my devcontainers and CI/CD pipelines stopped working.

After some investigation, I isolated the issue to the following command:

$ echo "FROM ubuntu" | docker build . --build-arg BUILDKIT_INLINE_CACHE=1 -f-
[+] Building 1.6s (5/6)
 => [internal] load .dockerignore                                                                  0.0s
 => => transferring context: 2B                                                                    0.0s
 => [internal] load build definition from Dockerfile                                               0.0s
 => => transferring dockerfile: 49B                                                                0.0s
 => [internal] load metadata for docker.io/library/ubuntu:latest                                   1.6s
 => [auth] library/ubuntu:pull token for registry-1.docker.io                                      0.0s
 => CACHED [1/1] FROM docker.io/library/ubuntu@sha256:9a0bdde4188b896a372804be2384015e90e3f84906b  0.0s
 => preparing layers for inline cache                                                              0.0s
ERROR: failed to solve: Unavailable: error reading from server: EOF

I initially opened this issue in Buildkit repository, but as Docker daemon crashes, I decided to open it here instead. (moby/buildkit#3576 (comment))

Reproduce

echo "FROM ubuntu" | docker build . --build-arg BUILDKIT_INLINE_CACHE=1 -f-

Expected behavior

To work as it worked before 23.0.0.

docker version

Client: Docker Engine - Community
 Version:           23.0.0
 API version:       1.42
 Go version:        go1.19.5
 Git commit:        e92dd87
 Built:             Wed Feb  1 17:47:51 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.0
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.5
  Git commit:       d7573ab
  Built:            Wed Feb  1 17:47:51 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.16
  GitCommit:        31aa4358a36870b21a992d3ad2bef29e1d693bec
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.2
    Path:     /home/felipecrs/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.15.1
    Path:     /home/felipecrs/.docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 2
  Running: 0
  Paused: 0
  Stopped: 2
 Images: 6
 Server Version: 23.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 31aa4358a36870b21a992d3ad2bef29e1d693bec
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.15.83.1-microsoft-standard-WSL2
 Operating System: Ubuntu 22.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 15.58GiB
 Name: FELIPE-MSI
 ID: EKD4:EAZA:6IFV:FFDZ:35NR:23ZD:3ZCV:QB72:T6BK:XVUF:Y6PT:JDWI
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

Additional Info

I captured the following in my docker daemon logs:

Apparently, Docker daemon crashes.

Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.338482781-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.338518578-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.338541170-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.338555858-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.338572278-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.338584261-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.338615339-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.338640275-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.338670542-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.338728350-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.338784264-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.338872399-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.338904058-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.338913756-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.340002046-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.340055416-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.348819671-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.348849416-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.348868752-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.348875755-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.348885013-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.348891234-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.348927542-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.348958039-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.348979198-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.349030915-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.349072653-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.349177529-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.349210069-03:00" level=info msg="[c>
Feb 03 10:48:11 FELIPE-MSI dockerd[2622]: time="2023-02-03T10:48:11.349227412-03:00" level=info msg="[c>
Feb 03 10:48:13 FELIPE-MSI dockerd[2622]: panic: runtime error: invalid memory address or nil pointer d>
Feb 03 10:48:13 FELIPE-MSI dockerd[2622]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc>
Feb 03 10:48:13 FELIPE-MSI dockerd[2622]: goroutine 694 [running]:
Feb 03 10:48:13 FELIPE-MSI dockerd[2622]: github.com/docker/docker/vendor/github.com/moby/buildkit/cach>
Feb 03 10:48:13 FELIPE-MSI dockerd[2622]:         /go/src/github.com/docker/docker/vendor/github.com/mo>
Feb 03 10:48:13 FELIPE-MSI dockerd[2622]: github.com/docker/docker/vendor/github.com/moby/buildkit/util>
Feb 03 10:48:13 FELIPE-MSI dockerd[2622]:         /go/src/github.com/docker/docker/vendor/github.com/mo>
Feb 03 10:48:13 FELIPE-MSI dockerd[2622]: sync.(*Once).doSlow(0x55aaad2bc885?, 0xa?)
Feb 03 10:48:13 FELIPE-MSI dockerd[2622]:         /usr/local/go/src/sync/once.go:74 +0xc2
Feb 03 10:48:13 FELIPE-MSI dockerd[2622]: sync.(*Once).Do(0xc0014c8190?, 0xc0004ea000?)
Feb 03 10:48:13 FELIPE-MSI dockerd[2622]:         /usr/local/go/src/sync/once.go:65 +0x1f
Feb 03 10:48:13 FELIPE-MSI dockerd[2622]: created by github.com/docker/docker/vendor/github.com/moby/bu>
Feb 03 10:48:13 FELIPE-MSI dockerd[2622]:         /go/src/github.com/docker/docker/vendor/github.com/mo>
Feb 03 10:48:13 FELIPE-MSI systemd[1]: docker.service: Main process exited, code=exited, status=2/INVAL>
Feb 03 10:48:13 FELIPE-MSI systemd[1]: docker.service: Failed with result 'exit-code'.
Feb 03 10:48:15 FELIPE-MSI systemd[1]: docker.service: Scheduled restart job, restart counter is at 2.
Feb 03 10:48:15 FELIPE-MSI systemd[1]: Stopped Docker Application Container Engine.
Feb 03 10:48:15 FELIPE-MSI systemd[1]: Starting Docker Application Container Engine...
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.430026714-03:00" level=info msg="St>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.432788798-03:00" level=info msg="[c>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.432818314-03:00" level=info msg="[c>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.432845855-03:00" level=info msg="[c>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.432859520-03:00" level=info msg="[c>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.432974365-03:00" level=info msg="[c>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.433044977-03:00" level=info msg="[c>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.433111020-03:00" level=info msg="[c>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.433164089-03:00" level=info msg="[c>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.433193073-03:00" level=info msg="[c>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.433211447-03:00" level=info msg="[c>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.433587168-03:00" level=info msg="[c>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.433634066-03:00" level=info msg="[c>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.434258752-03:00" level=info msg="[c>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.434288708-03:00" level=info msg="[c>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.434306501-03:00" level=info msg="[c>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.434333431-03:00" level=info msg="[c>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.434378816-03:00" level=info msg="[c>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.434403732-03:00" level=info msg="[c>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.434436143-03:00" level=info msg="[c>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.434456420-03:00" level=info msg="[c>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.434492628-03:00" level=info msg="[c>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.434526822-03:00" level=info msg="[c>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.434732636-03:00" level=info msg="[c>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.434744658-03:00" level=info msg="[c>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.443321532-03:00" level=info msg="[g>
Feb 03 10:48:15 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:15.448322224-03:00" level=info msg="Lo>
Feb 03 10:48:18 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:18.128413769-03:00" level=info msg="De>
Feb 03 10:48:18 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:18.945982966-03:00" level=info msg="Lo>
Feb 03 10:48:18 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:18.955400146-03:00" level=warning msg=>
Feb 03 10:48:18 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:18.955431004-03:00" level=warning msg=>
Feb 03 10:48:18 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:18.955435733-03:00" level=warning msg=>
Feb 03 10:48:18 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:18.955438718-03:00" level=warning msg=>
Feb 03 10:48:18 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:18.955450570-03:00" level=info msg="Do>
Feb 03 10:48:18 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:18.955473122-03:00" level=info msg="Da>
Feb 03 10:48:19 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:19.161730146-03:00" level=info msg="[c>
Feb 03 10:48:19 FELIPE-MSI systemd[1]: Started Docker Application Container Engine.
Feb 03 10:48:19 FELIPE-MSI dockerd[3648]: time="2023-02-03T10:48:19.164307827-03:00" level=info msg="AP>
li
Reactions are currently unavailable

Metadata

Metadata

Assignees

Labels

area/builder/buildkitBuildBuildkind/bugBugs are bugs. The cause may or may not be known at triage time so debugging may be needed.Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed.priority/P1Important: P1 issues are a top priority and a must-have for the next release.Important: P1 issues are a top priority and a must-have for the next release.status/0-triageversion/23.0

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions
Morty Proxy This is a proxified and sanitized view of the page, visit original site.