Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

libnetwork: clean up inDelete network atomically#45310

Merged
neersighted merged 1 commit into
moby:mastermoby/moby:masterfrom
corhere:libn/delete-network-more-atomicallycorhere/moby:libn/delete-network-more-atomicallyCopy head branch name to clipboard
Apr 20, 2023
Merged

libnetwork: clean up inDelete network atomically#45310
neersighted merged 1 commit into
moby:mastermoby/moby:masterfrom
corhere:libn/delete-network-more-atomicallycorhere/moby:libn/delete-network-more-atomicallyCopy head branch name to clipboard

Conversation

@corhere

@corhere corhere commented Apr 11, 2023

Copy link
Copy Markdown
Contributor

The (*network).ipamRelease function nils out the network's IPAM info fields, putting the network struct into an inconsistent state. The network-restore startup code panics if it tries to restore a network from a struct which has fewer IPAM config entries than IPAM info entries. Therefore (*network).delete contains a critical section: by persisting the network to the store after ipamRelease(), the datastore will contain an inconsistent network until the deletion operation completes and finishes deleting the network from the datastore. If for any reason the deletion operation is interrupted between ipamRelease() and deleteFromStore(), the daemon will crash on startup when it tries to restore the network.

Updating the datastore after releasing the network's IPAM pools may have served a purpose in the past, when a global datastore was used for intra-cluster communication and the IPAM allocator had persistent global state, but nowadays there is no global datastore and the IPAM allocator has no persistent state whatsoever. Remove the vestigial datastore update as it is no longer necessary and only serves to cause problems. If the network deletion is interrupted before the network is deleted from the datastore, the deletion will resume during the next daemon startup, including releasing the IPAM pools.

- What I did

- How I did it

- How to verify it
Race a network delete with shutting down the daemon. Verify that the daemon starts back up without panicking.

$ docker network create foo
$ docker network rm foo & sleep 0.01; pkill -TERM dockerd

You will have hit the race window if the docker network rm command logs an error indicative of the daemon shutting down before the API request completes, such as

error during connect: Delete http://%2Fvar%2Frun%2Fdocker.sock/v1.30/networks/foo: EOF

and on next startup, the daemon logs an INFO message Removing stale network foo.

- Description for the changelog

  • Fixed an issue which could corrupt the daemon state, crashing on next start if the daemon is terminated while a container network is being deleted

- A picture of a cute animal (not mandatory but encouraged)

The (*network).ipamRelease function nils out the network's IPAM info
fields, putting the network struct into an inconsistent state. The
network-restore startup code panics if it tries to restore a network
from a struct which has fewer IPAM config entries than IPAM info
entries. Therefore (*network).delete contains a critical section: by
persisting the network to the store after ipamRelease(), the datastore
will contain an inconsistent network until the deletion operation
completes and finishes deleting the network from the datastore. If for
any reason the deletion operation is interrupted between ipamRelease()
and deleteFromStore(), the daemon will crash on startup when it tries to
restore the network.

Updating the datastore after releasing the network's IPAM pools may have
served a purpose in the past, when a global datastore was used for
intra-cluster communication and the IPAM allocator had persistent global
state, but nowadays there is no global datastore and the IPAM allocator
has no persistent state whatsoever. Remove the vestigial datastore
update as it is no longer necessary and only serves to cause problems.
If the network deletion is interrupted before the network is deleted
from the datastore, the deletion will resume during the next daemon
startup, including releasing the IPAM pools.

Signed-off-by: Cory Snider <csnider@mirantis.com>

@akerouanton akerouanton left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@neersighted neersighted left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like I missed my LGTM after we discussed this offline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Panic in libnetwork during daemon start (panic: runtime error: index out of range [0] with length 0)

4 participants

Morty Proxy This is a proxified and sanitized view of the page, visit original site.