moby SWARM Mode - Docker binds ingress ports always on 0.0.0.0 - Go

Description

We have multiple hosts using swarm mode.

Some of this hosts have two or more external IP addresses. Docker swarm services with published ports always bind to 0.0.0.0:. Even after launching dockerd with --ip=xxx.xxx.xxx.xx.

Container not started in swarm mode, are binding to the correct IP. Services in Swarm are always binding published ports to every IP known by the Host (eno1,eno2 and lo). We cannot use the second IP with a port already taken by docker swarm.

Steps to reproduce the issue: 1. Create a Docker Service in Swarm with published Ports 2. netstat -utelpna | grep 3. Port is bound on 0.0.0.0:

Describe the results you received:

Started dockerd with --ip argument:

root 4869 1.1 10.0 2749972 102920 ? Ssl 12:36 0:12 /usr/bin/dockerd -H fd:// --ip=139.59.215.247 --swarm-default-advertise-addr=139.59.215.247

Swarm services bind published ports to every IP known by the system (0.0.0.0).

Describe the results you expected:

Expected swarm services to bind published ports only on IP 139.59.215.247.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:09 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:40:48 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 29
 Running: 4
 Paused: 0
 Stopped: 25
Images: 65
Server Version: 17.09.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 554
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: lbvavrurdt9edmfb44eshjbqk
 Is Manager: true
 ClusterID: 3uthkj04fn1rv2htzljm2imo9
 Managers: 1
 Nodes: 2
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: 139.59.215.247
 Manager Addresses:
  139.59.215.247:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.0-4-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 996.6MiB
Name: dev1
ID: CDVW:ZY3R:JMYQ:AW3L:CC6K:F7RL:JRZX:GPXQ:FOD6:4VTI:7TZV:7JDC
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Asked Oct 29 '21 16:10
avatar eyenx
eyenx

6 Answer:

@fcrisciani Thanks for the answer. My use case is that my servers are part of two networks. An internal private network which can not be accessed from the internet (in which all servers are in and I via VPN) and an external one which can be accessed via the internet. I wanted to have a very simple but still secure way of accessing docker swarm services in the private network only (without having to deal with the complicated IP table stuff).

1
Answered Nov 15 '17 at 08:33
avatar  of janober
janober

I have the same problem.

my solution was to run in host mode.

To be sure it would bind in the address I want in each docker node.

I have to remove docker_gwbridge from each node before join swarm and do something like

docker network create -o "com.docker.network.bridge.hostbindingipv4"="192.168.1.151" docker_gwbridge

192.168.1.151 is the ip I want to expose in this node

On compose I was using

ports:
  - target: 80
    published: 80
    protocol: tcp
    mode: host

It would be GREAT have something like that for ingress.

And it sounds really simple to implement.

Hope we have it.

1
Answered Apr 11 '18 at 12:52
avatar  of surfdude75
surfdude75

I am a little bit confused now. Checked now a lot of tickets but could not find what I was looking for. So hope somebody here can help me as it seems very closely related. What I want is that all services of a swarm node do not listen to all IP addresses and to a specific one instead. So my server has the IP 10.0.0.1 and 192.0.0.1 I want that it only responds to 10.0.0.1 . That would be for all services running on that node. I can understand that it is complicated on a per service level but for the whole node that should be not that hard. Can that be done or is that also not possible?

1
Answered Nov 09 '17 at 20:19
avatar  of janober
janober

@janober on recent versions of docker you can specify a --data-path-addr when initialising the swarm (docker swarm init), or joining the swarm (docker swarm join). That option allows you to specify either an IP-address, or interface to use for networking data (separate from the "control" networking, that's used by managers to control the swarm)

It was implemented in this pull request: https://github.com/moby/moby/pull/32717

Note that this option can only be set on init or join, and cannot be modified afterwards

1
Answered Nov 09 '17 at 21:10
avatar  of thaJeztah
thaJeztah

@thaJeztah thanks a lot for the fast reply. That sounds like exactly what I am looking for. Sadly does it not work for me. I am running Docker version 17.09.0-ce, build afdb6d4. So that option is already there.

However when I start swarm like this:

docker swarm init --advertise-addr=x.x.x.x--data-path-addr=x.x.x.x

and then a container, for example ngnix like that:

docker service create \
  --name nginx \
  --publish 80:80 \
  nginx

I can not just access the web page via http://x.x.x.x I can still access it via the other IP address http://y.y.y.y

Do I understand or do something wrong?

1
Answered Nov 10 '17 at 06:02
avatar  of janober
janober

Maybe also my later question is not very clear. I am also not talking about the "--listen-addr" because also that one does not change the "listening" I am talking about. I want to change the IP the docker services respond to.

1
Answered Nov 11 '17 at 08:48
avatar  of janober
janober