Opening as a new issue, per #24941 (comment)
Note I just re-ran this on current master (Docker version 1.12.0-dev, build 42f2205),
and ended up with the same result
Something I just ran into, and can reproduce reliably;
- Create 4 droplets on DigitalOcean, and install 1.12-RC4
- First node creates swarm (
docker swarm init)
- Nodes 2, and 3 join the swarm as manager
- Node 4 joins the swarm as worker
Then; create a service, and scale to 16
docker service create --name web --replicas=16 -p 80:80 nginx:alpine
On one of the manager nodes (swarm-test-02), watch docker node ls;
ID HOSTNAME MEMBERSHIP STATUS AVAILABILITY MANAGER STATUS
0j5yrijzbkgwl3bmjzp1aag1f swarm-test-03 Accepted Ready Active Reachable
36oy4vdvxxmnuw72pfli5dv2i swarm-test-01 Accepted Ready Active Leader
37argq2z05d5719olaiqpauvh * swarm-test-02 Accepted Ready Active Reachable
bpcy1dxk7v6md85qg3m7jn8il swarm-test-04 Accepted Ready Active
And on all nodes, watch docker ps;
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
72ea95d66177 nginx:alpine "nginx -g 'daemon off" About a minute ago Up About a minute 80/tcp, 443/tcp web.2.7h8zuthv89jr2
ao4c1x4r9qce
dd58f8757a06 nginx:alpine "nginx -g 'daemon off" About a minute ago Up About a minute 80/tcp, 443/tcp web.10.8bolssicpwc2
5op18dleducvg
59966a937870 nginx:alpine "nginx -g 'daemon off" About a minute ago Up About a minute 80/tcp, 443/tcp web.11.0shxh3ccvzpw
zvjvphjrzefkh
49040cf8f0a6 nginx:alpine "nginx -g 'daemon off" About a minute ago Up About a minute 80/tcp, 443/tcp web.9.18f9rtrhyamjd
mxnmd9hmokj0
Kill the leader node
From the DigitalOcean control panel, destroy the leader node, meanwhile, on the nodes,
watch what happens
1. Initial state (before killing leader)
ID HOSTNAME MEMBERSHIP STATUS AVAILABILITY MANAGER STATUS
1msvix0hu4vz7czqcwvzi3poq * ubuntu-2gb-ams3-01 Accepted Ready Active Reachable
264qo2mcvkuagirx68iluzi88 ubuntu-2gb-ams3-02 Accepted Ready Active Reachable
4z3arwkhuwvvqiqwm9rxqg165 ubuntu-2gb-ams3-01 Accepted Ready Active Leader
en76v2a9mlf12sjwdtw3hyb58 ubuntu-2gb-ams3-01 Accepted Ready Active
2. Status "unknown" for all nodes
Just after killing the leader, an rpc deadline error is presented, then, the
node status goes through the following stages:
ID HOSTNAME MEMBERSHIP STATUS AVAILABILITY MANAGER STATUS
0j5yrijzbkgwl3bmjzp1aag1f swarm-test-03 Accepted Unknown Active Reachable
36oy4vdvxxmnuw72pfli5dv2i swarm-test-01 Accepted Unknown Active Unreachable
37argq2z05d5719olaiqpauvh * swarm-test-02 Accepted Unknown Active Leader
bpcy1dxk7v6md85qg3m7jn8il swarm-test-04 Accepted Unknown Active
3. Status "down" for all nodes
ID HOSTNAME MEMBERSHIP STATUS AVAILABILITY MANAGER STATUS
0j5yrijzbkgwl3bmjzp1aag1f swarm-test-03 Accepted Down Active Reachable
36oy4vdvxxmnuw72pfli5dv2i swarm-test-01 Accepted Down Active Unreachable
37argq2z05d5719olaiqpauvh * swarm-test-02 Accepted Down Active Leader
bpcy1dxk7v6md85qg3m7jn8il swarm-test-04 Accepted Down Active
4. Status "down" for the manager that did not become leader
ID HOSTNAME MEMBERSHIP STATUS AVAILABILITY MANAGER STATUS
0j5yrijzbkgwl3bmjzp1aag1f swarm-test-03 Accepted Down Active Reachable
36oy4vdvxxmnuw72pfli5dv2i swarm-test-01 Accepted Down Active Unreachable
37argq2z05d5719olaiqpauvh * swarm-test-02 Accepted Ready Active Leader
bpcy1dxk7v6md85qg3m7jn8il swarm-test-04 Accepted Ready Active
5. Status "ready"
ID HOSTNAME MEMBERSHIP STATUS AVAILABILITY MANAGER STATUS
0j5yrijzbkgwl3bmjzp1aag1f swarm-test-03 Accepted Ready Active Reachable
36oy4vdvxxmnuw72pfli5dv2i swarm-test-01 Accepted Down Active Unreachable
37argq2z05d5719olaiqpauvh * swarm-test-02 Accepted Ready Active Leader
bpcy1dxk7v6md85qg3m7jn8il swarm-test-04 Accepted Ready Active
However, at stage 5, all containers ended up on a single node:
swarm-test-02:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
swarm-test-03:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
swarm-test-04:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
06b59fd80536 nginx:alpine "nginx -g 'daemon off" 5 minutes ago Up 4 minutes 80/tcp, 443/tcp web.1.7599uowuwx8kln46jzzlzrsej
5e751507a919 nginx:alpine "nginx -g 'daemon off" 5 minutes ago Up 4 minutes 80/tcp, 443/tcp web.8.6wcxeir8kw4misv5hm6tbt1gb
9f871df4c0e6 nginx:alpine "nginx -g 'daemon off" 5 minutes ago Up 4 minutes 80/tcp, 443/tcp web.9.6348tx8lisem60tge1n9h105p
34d922f5627b nginx:alpine "nginx -g 'daemon off" 5 minutes ago Up 4 minutes 80/tcp, 443/tcp web.10.18x3x83ay20moo4wfk6uqssuk
fe00d09420fe nginx:alpine "nginx -g 'daemon off" 5 minutes ago Up 5 minutes 80/tcp, 443/tcp web.15.7ajzwkx125w0ilbv09kluxsh5
1a365c02c807 nginx:alpine "nginx -g 'daemon off" 5 minutes ago Up 4 minutes 80/tcp, 443/tcp web.2.9g06i0ji6oby1a681hrnr7fuv
780f69e79646 nginx:alpine "nginx -g 'daemon off" 5 minutes ago Up 4 minutes 80/tcp, 443/tcp web.12.0952zl8ejgp7glfr991xmiqsu
93002b1dcfe7 nginx:alpine "nginx -g 'daemon off" 5 minutes ago Up 4 minutes 80/tcp, 443/tcp web.14.62cwuiykimiz6dlmiognotmcn
c96a48711179 nginx:alpine "nginx -g 'daemon off" 5 minutes ago Up 4 minutes 80/tcp, 443/tcp web.3.96zsnc2bfgd9dj71bicgds0w1
7634285e896a nginx:alpine "nginx -g 'daemon off" 5 minutes ago Up 4 minutes 80/tcp, 443/tcp web.6.bqwqso18dqrkr80jc51aumonq
3ed44fe7e021 nginx:alpine "nginx -g 'daemon off" 5 minutes ago Up 4 minutes 80/tcp, 443/tcp web.11.8s2va2tvmpw187s7v8ce6h3r2
23a083a91b6b nginx:alpine "nginx -g 'daemon off" 5 minutes ago Up 4 minutes 80/tcp, 443/tcp web.13.709wueza0j9631eibdm7ildu3
6d6871c931ce nginx:alpine "nginx -g 'daemon off" 5 minutes ago Up 4 minutes 80/tcp, 443/tcp web.4.1o0ay75v1y8amya2hnhnfb8y0
2a4cb84bd3ee nginx:alpine "nginx -g 'daemon off" 5 minutes ago Up 4 minutes 80/tcp, 443/tcp web.7.7ijasltqhbalctuahsly3n2cy
18bcc8cb66a1 nginx:alpine "nginx -g 'daemon off" 5 minutes ago Up 5 minutes 80/tcp, 443/tcp web.16.b7z9i8lo97fzxmtvwd06ufpxp
67fa07227b58 nginx:alpine "nginx -g 'daemon off" 5 minutes ago Up 4 minutes 80/tcp, 443/tcp web.5.6ln0iyt24qxnic2nu0rg3ttxx
Opening as a new issue, per #24941 (comment)
Something I just ran into, and can reproduce reliably;
docker swarm init)Then; create a service, and scale to 16
On one of the manager nodes (
swarm-test-02), watchdocker node ls;And on all nodes, watch
docker ps;Kill the leader node
From the DigitalOcean control panel, destroy the leader node, meanwhile, on the nodes,
watch what happens
1. Initial state (before killing leader)
2. Status "unknown" for all nodes
Just after killing the leader, an
rpc deadlineerror is presented, then, thenode status goes through the following stages:
3. Status "down" for all nodes
4. Status "down" for the manager that did not become leader
5. Status "ready"
However, at stage 5, all containers ended up on a single node:
swarm-test-02:
swarm-test-03:
swarm-test-04: