Network Connectivity Issues from Docker Containers on AWS ipv6-only Lightsail Ubuntu Instance

0

Hello AWS Community,

I'm currently experiencing network connectivity issues with Docker containers running on an Ubuntu instance on AWS Lightsail. Here's the situation:

Problem Description:

  • I have an AWS Lightsail instance running Ubuntu.
  • Inside this instance, I’m running a Debian-based Docker container.
  • The host machine (Lightsail instance) can successfully ping external sites like google.com and resolve domain names.
  • However, when I attempt to ping or curl external sites (e.g., google.com, github.com) from within the Docker container, I receive 100% packet loss or connection timeouts.

Commands and Outputs:

  • Ping from Docker Container ping google.com -c 4: Output:
     PING google.com (172.217.167.110) 56(84) bytes of data.
     --- google.com ping statistics ---
    4 packets transmitted, 0 received, 100% packet loss, time 3057ms
  • Curl from Docker Container curl github.com: Output:
    curl: (28) Failed to connect to github.com port 80 after 135265 ms: Couldn't connect to server
  • nslookup from Docker Container nslookup github.com: Output:
    Server:         127.0.0.11
    Address:        127.0.0.11#53
    Non-authoritative answer:
    Name:   github.com
    Address: 4.237.22.38

Troubleshooting Steps Taken:

Firewall & IPTables:

I checked iptables on the host and noticed the FORWARD chain was set to DROP. I tried setting it to ACCEPT and added rules to allow outbound traffic from Docker containers, but the issue persists. Here’s the output of iptables -L -v -n:

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
Chain FORWARD (policy DROP 0 packets, 0 bytes)
  pkts bytes target     prot opt in     out     source               destination
  82   5280 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0
  ...
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
Chain DOCKER (4 references)
  pkts bytes target     prot opt in     out     source               destination
  ...

Has anyone faced a similar issue with network connectivity from Docker containers on AWS ipv6-only Lightsail instance? Are there specific Docker or AWS configurations that I might be overlooking? Any advice or troubleshooting steps would be greatly appreciated.

Thank you in advance for your help

3 Answers
1
Accepted Answer

Thanks for the help, Riku_Kobayashi. I think I found the issue.

For ping, if I don't specify the -6 flag (which tells ping to use IPv6), the command prioritizes IPv4. Since I'm on an IPv6-only instance, that doesn’t work. I tried ping google.com -6, and it works now.

As for curl github.com, I discovered that GitHub does not support IPv6, so it simply won't work on this instance.

byond
answered 13 days ago
profile picture
EXPERT
reviewed 12 days ago
profile picture
EXPERT
reviewed 13 days ago
1

Hello.

Since the IPv4 address can be confirmed when name resolution is performed within the container, we suspect that the Docker container is not configured for IPv6.
Have you configured it to support IPv6 using the steps in the document below?
https://docs.docker.com/engine/daemon/ipv6/

profile picture
EXPERT
answered 13 days ago
profile picture
EXPERT
reviewed 12 days ago
  • How about running the following command to enable NAT on the host OS side?

    ip6tables -t nat -A POSTROUTING -s 2001:db8:1::/64 ! -o docker0 -j MASQUERADE
    
0

Hi,

Yes, I have configure Docker for IPv6 following the document:

  • My /etc/docker/daemon.json looks like this:
{
  "ipv6": true,
  "fixed-cidr-v6": "2001:db8:1::/64"
}
  • I also set enable_ipv6: true inside my docker compose files:
# compose.traefik.yml
name: traefik
services:
  traefik:
    command:
      - --providers.docker=true
      - --providers.docker.exposedbydefault=false
      - --providers.file.directory=/etc/traefik/dynamic_conf
      - --providers.file.watch=true
      - --entrypoints.http.address=:80
      - --entrypoints.https.address=:443
      - --accesslog
      - --log
      - --api
      - --certificatesresolvers.myresolver.acme.tlschallenge=true
      - --certificatesresolvers.myresolver.acme.email=johndoe@example.com
      - --certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json
    image: traefik:v2.6
    restart: unless-stopped
    labels:
      traefik.docker.network: traefik-public
      traefik.enable: "true"
      traefik.http.middlewares.https-redirect.redirectscheme.permanent: "true"
      traefik.http.middlewares.https-redirect.redirectscheme.scheme: https
      traefik.http.services.traefik-public.loadbalancer.server.port: "8080"
    networks:
      traefik-public: null
    ports:
      - mode: ingress
        target: 80
        published: "80"
        protocol: tcp
      - mode: ingress
        target: 443
        published: "443"
        protocol: tcp
    volumes:
      - type: bind
        source: /home/ubuntu/traefik/letsencrypt
        target: /letsencrypt
        bind:
          create_host_path: true
      - type: bind
        source: /home/ubuntu/traefik/certificates
        target: /certificates
        bind:
          create_host_path: true
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock
        read_only: true
        bind:
          create_host_path: true
      - type: bind
        source: ./traefik.config.yml
        target: /etc/traefik/dynamic_conf/conf.yml
        read_only: true
        bind:
          create_host_path: true
networks:
  traefik-public:
    name: traefik-public
    enable_ipv6: true
# docker-compose.yml
services:
  redis:
    image: redis:alpine
    restart: unless-stopped
    networks:
      bench-network: null
  frappe:
    image: jk1117/bench:erpnext-version-15
    restart: unless-stopped
    command: sleep infinity
    environment:
      - SHELL=/bin/bash
    volumes:
      - bench-dir:/home/frappe/frappe-bench
      - sites:/home/frappe/frappe-bench/sites:rw
      - assets:/home/frappe/frappe-bench/apps:rw
      - templates:/home/frappe/.bench/bench/config/templates:rw
    networks:
      bench-network: null
    extra_hosts:
      - "actual.domain.com:[2406:da1c:31d:2d00:33df:130:e1a2:8478]"
  proxy:
    depends_on:
      frappe:
        condition: service_started
    restart: unless-stopped
    environment:
      BACKEND: frappe:8000
      FRAPPE_SITE_NAME_HEADER: $$host
      SOCKETIO: frappe:443
      UPSTREAM_REAL_IP_ADDRESS: 2406:da1c:31d:2d00:33df:130:e1a2:8478
      UPSTREAM_REAL_IP_HEADER: X-Forwarded-For
      UPSTREAM_REAL_IP_RECURSIVE: "off"
      SERVER_NAME: actual.domain.com
    image: nginx:1.23.1
    labels:
      traefik.docker.network: traefik-public
      traefik.enable: 'true'
      traefik.http.services.uat.loadbalancer.server.port: '8080'
      traefik.http.routers.uat-http.entrypoints: http
      traefik.http.routers.uat-http.rule: Host(`actual.domain.com`)
      traefik.http.routers.uat-http.service: uat
      traefik.http.routers.uat-http.middlewares: https-redirect
      traefik.http.routers.uat-https.rule: Host(`actual.domain.com`)
      traefik.http.routers.uat-https.entrypoints: https
      traefik.http.routers.uat-https.tls: true
      traefik.http.routers.uat-https.service: uat
      traefik.http.routers.uat-https.tls.certresolver: myresolver
    networks:
      traefik-public: null
      bench-network: null
    volumes:
      - ./nginx_templates:/etc/nginx/templates
      - sites:/usr/share/nginx/html/sites:rw
      - assets:/usr/share/nginx/html/assets:ro
      - templates:/usr/share/nginx/html/templates:ro
volumes:
  bench-dir:
  assets:
  sites:
  templates:
networks:
  bench-network:
    enable_ipv6: true
  traefik-public:
    name: traefik-public
    external: true
    enable_ipv6: true
  mariadb-network:
    name: mariadb-network
    external: true
    enable_ipv6: true
  • I also verify the setup by running docker run --rm --network ip6net -p 80:80 traefik/whoami and the result is exactly the same as the documentation provided.

Am I missing any setting to enable IPv6 for Docker Compose?

byond
answered 13 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions