IPv6 Problems with Docker
- hosting
- servers
- infrastructure
- docker
- updates
Where to start¶
Docker networking is, interesting.
IPv4 ✅
IPv4 and IPv6 🟡
IPV6 💥
Short and Sweet¶
I'm running n8n to automate things between various api's, it's pretty awesome. But now I want to migrate my n8n vm to be ipv6 only.
After turning off ipv4 and rebooting, suddenly none of the api calls worked. DNS resolution is failing.
I've got DNS resolution working, JANKILY and it takes ~3 seconds to respond but it does respond.
Still doesn't work as n8n tries to connect to the IPv4 responded with.
Some info¶
When IPv4 is available on the host system, the docker daemon will create a DNS server for the containers that's hardcoded to 127.0.0.11
It resolves container names first, then searches the external dns servers.
When IPv6 is enabled, the ip6 server reported on the host system via whatever dhcp client (using /run/hook-state/resolv.conf/eth0.dhcp6 || eth0.ra for example) is simply appended to the generated /etc/resolv.conf inside the docker containers.
When both IPv4 and IPv6 are enabled on the host system, this means there are two DNS entries generated in /etc/resolv.conf - which creates and fully explains the intermittent dns resolution problems I was seeing with the Nextcloud-AIO.
Musl and GLIBC¶
The containers that end up having issues are the ones based on Musl instead of GlibC - eg - anything Alpine based.
Musl does dns resolution in parrallel, it will query all the servers in /etc/resolv.conf and whichever responds the fastest wins and is reported back to whichever application requested the lookup. When ipv6 is enabled, the ipv6 dns server can and fairly often does respond faster than the internal docker one, but it's less likely than the docker one responding faster so it can easily appear on the surface as if things are working as intended. ** This explains the DNS resolution issues I was having with Nextcloud AIO!**
Some of the issue here has been alleviated in Alpine 3.18 ?
When ipv4 is removed¶
Things break. I SWEAR to god I've seen it not even put 127.0.0.11 into /etc/resolv.conf a few times , but I can't seem to reproduce that right now.
Testing¶
dockernet ip6 configuration.¶
networks:
dockernet:
name: app-dockernet
enable_ipv6: true
driver: bridge
internal: false
driver_opts:
com.docker.network.enable_ipv6: "true"
com.docker.network.bridge.enable_ip_masquerade: "true"
ipam:
driver: default
config:
- subnet: 172.19.238.0/24
gateway: 172.19.238.1
- subnet: fd49:1::/64
gateway: fd49:1::1
mongonet:
name: mongonet
driver: bridge
internal: true
Test configurations:¶
/home/deadc0de # cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.17.4
PRETTY_NAME="Alpine Linux v3.17"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"
/home/deadc0de # docker version
Client:
Version: 24.0.5
API version: 1.43
Go version: go1.20.6
Git commit: ced0996
Built: Fri Jul 21 20:34:32 2023
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 24.0.5
API version: 1.43 (minimum version 1.12)
Go version: go1.20.6
Git commit: a61e2b4
Built: Fri Jul 21 20:35:56 2023
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: v1.6.18
GitCommit: 2456e983eb9e37e47538f59ea18f2043c9a73640
runc:
Version: 1.1.5
GitCommit: f19387a6bec4944c770f7668ab51c4348d9c2f38
docker-init:
Version: 0.19.0
GitCommit: de40ad0
A¶
Only dockernet No mongonet¶
n8n with mongo, mongo express, and traefik as an ingress to n8n and mongo express.
This can really be any images, I just had these ready. It can be 3x nginx's with different names, or alpine's doing a while sleep loop. Anything to create a container and give it a hostname on the test network.
/home/deadc0de # docker exec -it app-n8n sh
~ $ nslookup mongo
Server: 127.0.0.11
Address: 127.0.0.11:53
Non-authoritative answer:
Name: mongo
Address: 172.19.238.5
Non-authoritative answer:
Name: mongo
Address: fd49:1::5
~ $ nslookup google.com
Server: 127.0.0.11
Address: 127.0.0.11:53
** server can't find google.com: SERVFAIL
** server can't find google.com: SERVFAIL
~ $ cat /etc/resolv.conf
nameserver 127.0.0.11
nameserver fd60::3
options ndots:0
Add DNS Entries¶
/etc/docker/daemon.json add dns: ["fd60::3"] the internal network dns ip6 address,
/home/deadc0de # nano /etc/docker/daemon.json
/home/deadc0de # service docker restart
* Stopping Docker Daemon ... [ ok ]
* Starting Docker Daemon ... [ ok ]
/home/deadc0de # docker exec -it app-n8n sh
~ $ nslookup mongo
Server: 127.0.0.11
Address: 127.0.0.11:53
Non-authoritative answer:
Name: mongo
Address: 172.19.238.3
Non-authoritative answer:
Name: mongo
Address: fd49:1::3
~ $ nslookup google.com
Server: 127.0.0.11
Address: 127.0.0.11:53
** server can't find google.com: SERVFAIL
** server can't find google.com: SERVFAIL
~ $
Add DNS Entries¶
/etc/docker/daemon.json add dns: ["fd60::3","192.168.255.1"] the internal network dns ip6 address,
/home/deadc0de # docker exec -it app-n8n sh
~ $ nslookup mongo
Server: 127.0.0.11
Address: 127.0.0.11:53
Non-authoritative answer:
Name: mongo
Address: 172.19.238.5
Non-authoritative answer:
Name: mongo
Address: fd49:1::5
~ $ nslookup google.com
Server: 127.0.0.11
Address: 127.0.0.11:53
Server: fd60::3
Address: [fd60::3]:53
Non-authoritative answer:
Name: google.com
Address: 142.250.69.238
Non-authoritative answer:
Name: google.com
Address: 2607:f8b0:4002:c05::65
Name: google.com
Address: 2607:f8b0:4002:c05::66
Name: google.com
Address: 2607:f8b0:4002:c05::8b
Name: google.com
Address: 2607:f8b0:4002:c05::64
~ $
Current Results¶
n8n now resolves external addresses, but it tries to connect via IPv4 ERROR: connect ENETUNREACH 192.168.255.5:443
With mongonet added to every container too¶
/app # nano docker-compose.yml
/app # docker compose down && docker compose up -d
[+] Running 6/6
✔ Container traefik Removed 0.5s
✔ Container app-n8n Removed 4.2s
✔ Container mongo Removed 0.4s
✔ Container mongo-express Removed 0.5s
✔ Network app-dockernet Removed 0.1s
✔ Network mongonet Removed 0.2s
[+] Running 6/6
✔ Network app-dockernet Created 0.1s
✔ Network mongonet Created 0.0s
✔ Container mongo-express Started 2.1s
✔ Container traefik Started 2.1s
✔ Container app-n8n Started 1.9s
✔ Container mongo Started 1.2s
/app # docker exec -it app-n8n sh
~ $ nslookup mongo
Server: 127.0.0.11
Address: 127.0.0.11:53
Non-authoritative answer:
Name: mongo
Address: 172.19.238.2
Non-authoritative answer:
Name: mongo
Address: fd49:1::2
~ $ nslookup google.com
Server: 127.0.0.11
Address: 127.0.0.11:53
Server: fd60::3
Address: [fd60::3]:53
Non-authoritative answer:
Name: google.com
Address: 142.250.72.46
Non-authoritative answer:
Name: google.com
Address: 2607:f8b0:400f:804::200e
Current Results¶
No Change
Disclaimer¶
I'm no super genius, I just have a lot of time on my hands and I've dived as deep as I can go into this.
External Links¶
- Docker Enable ipv6
- IPv6 Embedded DNS Server Feature Request
- Docker in IPv6 only - network problem #32675
- Intermittent faults of docker-internal DNS with IPv6 #2492