Description
I'm getting inconsistent results with placement-pref where an empty 3-node swarm will schedule tasks on nodes that don't have the label before the spread is satisfied. 17.10 and the simple Docker Docs example causes my containers to run on nodes that don't have the Preference label.
Steps to reproduce the issue:
- create three node swarm, (node1, node2, node3) then:
docker node update --label-add=azone=1 node1docker node update --label-add=azone=2 node2docker service create --placement-pref spread=node.labels.azone --replicas 2 --name spread nginx- Check results, remove service, and re-create it several times to see how tasks fall on nodes.
Describe the results you received:
Randomly one of the two tasks requested by the service will land on node3, leaving one of the azone labeled nodes without a task. For me this happened at least 1 out of 5 times. Again, no other containers running on the three nodes.
Describe the results you expected:
You would expect the two replicas to always land on node1 and node2, in order to satisfy the spread preference.
Additional information you deem important (e.g. issue happens only occasionally):
I hope it's not user error 😱
Output of docker version:
Client:
Version: 17.10.0-ce
API version: 1.33
Go version: go1.8.3
Git commit: f4ffd25
Built: Tue Oct 17 19:04:16 2017
OS/Arch: linux/amd64
Server:
Version: 17.10.0-ce
API version: 1.33 (minimum version 1.12)
Go version: go1.8.3
Git commit: f4ffd25
Built: Tue Oct 17 19:02:56 2017
OS/Arch: linux/amd64
Experimental: false
Output of docker info:
Containers: 0
Running: 0
Paused: 0
Stopped: 4
Images: 10
Server Version: 17.10.0-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 65
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: 6u77atijtco59mxztkm3tyafk
Is Manager: true
ClusterID: vadxmt0dcf18jchgh4yydmdy2
Managers: 3
Nodes: 3
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: REDACTED
Manager Addresses:
REDACTED:2377
REDACTED:2377
REDACTED:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 0351df1c5a66838d0c392b4ac4cf9450de844e2d
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-93-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 488.3MiB
Name: node1
ID: REDACTED
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional environment details (AWS, VirtualBox, physical, etc.):
docker service inspect spread
[
{
"ID": "53dmcskbuoif9qy0mdf0vs4z0",
"Version": {
"Index": 5874
},
"CreatedAt": "2017-11-08T01:07:15.385965515Z",
"UpdatedAt": "2017-11-08T01:07:15.385965515Z",
"Spec": {
"Name": "spread",
"Labels": {},
"TaskTemplate": {
"ContainerSpec": {
"Image": "nginx:latest@sha256:9fca103a62af6db7f188ac3376c60927db41f88b8d2354bf02d2290a672dc425",
"StopGracePeriod": 10000000000,
"DNSConfig": {}
},
"Resources": {
"Limits": {},
"Reservations": {}
},
"RestartPolicy": {
"Condition": "any",
"Delay": 5000000000,
"MaxAttempts": 0
},
"Placement": {
"Preferences": [
{
"Spread": {
"SpreadDescriptor": "node.labels.azone"
}
}
],
"Platforms": [
{
"Architecture": "amd64",
"OS": "linux"
},
{
"OS": "linux"
},
{
"Architecture": "arm64",
"OS": "linux"
},
{
"Architecture": "386",
"OS": "linux"
},
{
"Architecture": "ppc64le",
"OS": "linux"
},
{
"Architecture": "s390x",
"OS": "linux"
}
]
},
"ForceUpdate": 0,
"Runtime": "container"
},
"Mode": {
"Replicated": {
"Replicas": 2
}
},
"UpdateConfig": {
"Parallelism": 1,
"FailureAction": "pause",
"Monitor": 5000000000,
"MaxFailureRatio": 0,
"Order": "stop-first"
},
"RollbackConfig": {
"Parallelism": 1,
"FailureAction": "pause",
"Monitor": 5000000000,
"MaxFailureRatio": 0,
"Order": "stop-first"
},
"EndpointSpec": {
"Mode": "vip"
}
},
"Endpoint": {
"Spec": {}
}
}
]docker node inspect node1
[
{
"ID": "6u77atijtco59mxztkm3tyafk",
"Version": {
"Index": 5810
},
"CreatedAt": "2017-09-22T07:35:42.844166962Z",
"UpdatedAt": "2017-11-07T22:34:47.577592187Z",
"Spec": {
"Labels": {
"azone": "1"
},
"Role": "manager",
"Availability": "active"
},
"Description": {
"Hostname": "node1",
"Platform": {
"Architecture": "x86_64",
"OS": "linux"
},
"Resources": {
"NanoCPUs": 1000000000,
"MemoryBytes": 512049152
},
"Engine": {
"EngineVersion": "17.10.0-ce",
"Plugins": "REDACTED-default-plugins"
},
"TLSInfo": {
"TrustRoot": "REDACTED",
"CertIssuerSubject": "REDACTED",
"CertIssuerPublicKey": "REDACTED"
}
},
"Status": {
"State": "ready",
"Addr": "REDACTED"
},
"ManagerStatus": {
"Leader": true,
"Reachability": "reachable",
"Addr": "REDACTED:2377"
}
}
]docker node inspect node2
[
{
"ID": "kpjxoefz8fkswtr0av9to1kpg",
"Version": {
"Index": 5811
},
"CreatedAt": "2017-09-22T07:36:00.0793046Z",
"UpdatedAt": "2017-11-07T22:34:47.603925533Z",
"Spec": {
"Labels": {
"azone": "2"
},
"Role": "manager",
"Availability": "active"
},
"Description": {
"Hostname": "node2",
"Platform": {
"Architecture": "x86_64",
"OS": "linux"
},
"Resources": {
"NanoCPUs": 1000000000,
"MemoryBytes": 512049152
},
"Engine": {
"EngineVersion": "17.10.0-ce",
"Plugins": "REDACTED-default-plugins"
},
"TLSInfo": {
"TrustRoot": "REDACTED",
"CertIssuerSubject": "REDACTED",
"CertIssuerPublicKey": "REDACTED"
}
},
"Status": {
"State": "ready",
"Addr": "REDACTED"
},
"ManagerStatus": {
"Reachability": "reachable",
"Addr": "REDACTED:2377"
}
}
]docker node inspect node3
[
{
"ID": "p1fr4z8gp74gaxpgykyplazyy",
"Version": {
"Index": 773
},
"CreatedAt": "2017-09-22T07:36:01.915696367Z",
"UpdatedAt": "2017-10-19T08:26:34.456319595Z",
"Spec": {
"Labels": {},
"Role": "manager",
"Availability": "active"
},
"Description": {
"Hostname": "node3",
"Platform": {
"Architecture": "x86_64",
"OS": "linux"
},
"Resources": {
"NanoCPUs": 1000000000,
"MemoryBytes": 512049152
},
"Engine": {
"EngineVersion": "17.10.0-ce",
"Plugins": "REDACTED-default-plugins"
},
"TLSInfo": {
"TrustRoot": "REDACTED",
"CertIssuerSubject": "REDACTED",
"CertIssuerPublicKey": "REDACTED"
}
},
"Status": {
"State": "ready",
"Addr": "REDACTED"
},
"ManagerStatus": {
"Reachability": "reachable",
"Addr": "REDACTED:2377"
}
}
]
Description
I'm getting inconsistent results with
placement-prefwhere an empty 3-node swarm will schedule tasks on nodes that don't have the label before thespreadis satisfied.17.10and the simple Docker Docs example causes my containers to run on nodes that don't have the Preference label.Steps to reproduce the issue:
docker node update --label-add=azone=1 node1docker node update --label-add=azone=2 node2docker service create --placement-pref spread=node.labels.azone --replicas 2 --name spread nginxDescribe the results you received:
Randomly one of the two tasks requested by the service will land on node3, leaving one of the
azonelabeled nodes without a task. For me this happened at least 1 out of 5 times. Again, no other containers running on the three nodes.Describe the results you expected:
You would expect the two replicas to always land on node1 and node2, in order to satisfy the
spreadpreference.Additional information you deem important (e.g. issue happens only occasionally):
I hope it's not user error 😱
Output of
docker version:Output of
docker info:Additional environment details (AWS, VirtualBox, physical, etc.):
docker service inspect spread[ { "ID": "53dmcskbuoif9qy0mdf0vs4z0", "Version": { "Index": 5874 }, "CreatedAt": "2017-11-08T01:07:15.385965515Z", "UpdatedAt": "2017-11-08T01:07:15.385965515Z", "Spec": { "Name": "spread", "Labels": {}, "TaskTemplate": { "ContainerSpec": { "Image": "nginx:latest@sha256:9fca103a62af6db7f188ac3376c60927db41f88b8d2354bf02d2290a672dc425", "StopGracePeriod": 10000000000, "DNSConfig": {} }, "Resources": { "Limits": {}, "Reservations": {} }, "RestartPolicy": { "Condition": "any", "Delay": 5000000000, "MaxAttempts": 0 }, "Placement": { "Preferences": [ { "Spread": { "SpreadDescriptor": "node.labels.azone" } } ], "Platforms": [ { "Architecture": "amd64", "OS": "linux" }, { "OS": "linux" }, { "Architecture": "arm64", "OS": "linux" }, { "Architecture": "386", "OS": "linux" }, { "Architecture": "ppc64le", "OS": "linux" }, { "Architecture": "s390x", "OS": "linux" } ] }, "ForceUpdate": 0, "Runtime": "container" }, "Mode": { "Replicated": { "Replicas": 2 } }, "UpdateConfig": { "Parallelism": 1, "FailureAction": "pause", "Monitor": 5000000000, "MaxFailureRatio": 0, "Order": "stop-first" }, "RollbackConfig": { "Parallelism": 1, "FailureAction": "pause", "Monitor": 5000000000, "MaxFailureRatio": 0, "Order": "stop-first" }, "EndpointSpec": { "Mode": "vip" } }, "Endpoint": { "Spec": {} } } ]docker node inspect node1[ { "ID": "6u77atijtco59mxztkm3tyafk", "Version": { "Index": 5810 }, "CreatedAt": "2017-09-22T07:35:42.844166962Z", "UpdatedAt": "2017-11-07T22:34:47.577592187Z", "Spec": { "Labels": { "azone": "1" }, "Role": "manager", "Availability": "active" }, "Description": { "Hostname": "node1", "Platform": { "Architecture": "x86_64", "OS": "linux" }, "Resources": { "NanoCPUs": 1000000000, "MemoryBytes": 512049152 }, "Engine": { "EngineVersion": "17.10.0-ce", "Plugins": "REDACTED-default-plugins" }, "TLSInfo": { "TrustRoot": "REDACTED", "CertIssuerSubject": "REDACTED", "CertIssuerPublicKey": "REDACTED" } }, "Status": { "State": "ready", "Addr": "REDACTED" }, "ManagerStatus": { "Leader": true, "Reachability": "reachable", "Addr": "REDACTED:2377" } } ]docker node inspect node2[ { "ID": "kpjxoefz8fkswtr0av9to1kpg", "Version": { "Index": 5811 }, "CreatedAt": "2017-09-22T07:36:00.0793046Z", "UpdatedAt": "2017-11-07T22:34:47.603925533Z", "Spec": { "Labels": { "azone": "2" }, "Role": "manager", "Availability": "active" }, "Description": { "Hostname": "node2", "Platform": { "Architecture": "x86_64", "OS": "linux" }, "Resources": { "NanoCPUs": 1000000000, "MemoryBytes": 512049152 }, "Engine": { "EngineVersion": "17.10.0-ce", "Plugins": "REDACTED-default-plugins" }, "TLSInfo": { "TrustRoot": "REDACTED", "CertIssuerSubject": "REDACTED", "CertIssuerPublicKey": "REDACTED" } }, "Status": { "State": "ready", "Addr": "REDACTED" }, "ManagerStatus": { "Reachability": "reachable", "Addr": "REDACTED:2377" } } ]docker node inspect node3[ { "ID": "p1fr4z8gp74gaxpgykyplazyy", "Version": { "Index": 773 }, "CreatedAt": "2017-09-22T07:36:01.915696367Z", "UpdatedAt": "2017-10-19T08:26:34.456319595Z", "Spec": { "Labels": {}, "Role": "manager", "Availability": "active" }, "Description": { "Hostname": "node3", "Platform": { "Architecture": "x86_64", "OS": "linux" }, "Resources": { "NanoCPUs": 1000000000, "MemoryBytes": 512049152 }, "Engine": { "EngineVersion": "17.10.0-ce", "Plugins": "REDACTED-default-plugins" }, "TLSInfo": { "TrustRoot": "REDACTED", "CertIssuerSubject": "REDACTED", "CertIssuerPublicKey": "REDACTED" } }, "Status": { "State": "ready", "Addr": "REDACTED" }, "ManagerStatus": { "Reachability": "reachable", "Addr": "REDACTED:2377" } } ]