IPv6-only Lightsail Ubuntu + Debian outbound call to Let's Encrypt timing out

Question

I have an off the shelf Lightsail VPS IPv6-only, Ubuntu 22.04, with basic Docker and Caddy installation. Edit : confirmed to fail also with Debian 12.4.
When Caddy calls out to the the Let's Encrypt API to get a certificate, it fails with a timeout. I tried also with a Certbot container to rule out Caddy and it behaves the same.
It is behaving as if it's preferring IPv4 call when of course those fail with a timeout - only IPv6 calls work. Let's Encrypt do fully support IPv6 and the majority of their requests are resolved by IPv6. Is there a way to force IPV6 transport when it makes the outbound calls?

This is the log extract, I've redacted the domains and email. I have left out the issuer "2/2" as they don't support IPv6:

```
caddy-1  | {"level":"warn","ts":1707861337.1362386,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://acme-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-v02.api.letsencrypt.org/directory\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
caddy-1  | {"level":"info","ts":1707861366.8875773,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"redacted.url.com"}
caddy-1  | {"level":"debug","ts":1707861366.887625,"logger":"events","msg":"event","name":"cert_obtaining","id":"74e7ebd6-95c8-4a43-b5ee-3538a4135960","origin":"tls","data":{"identifier":"redacted.url.com"}}
caddy-1  | {"level":"debug","ts":1707861366.887937,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
caddy-1  | {"level":"warn","ts":1707861367.3869283,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://acme-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-v02.api.letsencrypt.org/directory\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
caddy-1  | {"level":"error","ts":1707861367.3869965,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"www.redacted.url.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"registering account [mailto:email@example.com] with server: provisioning client: performing request: Get \"https://acme-v02.api.letsencrypt.org/directory\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
```

Outbound calls as expected timeout with IPv4, but should work with IPv6.
```
$ curl -I6 https://acme-v02.api.letsencrypt.org/directory
HTTP/2 200 
server: nginx
date: Tue, 13 Feb 2024 22:08:33 GMT
content-type: application/json
content-length: 752
cache-control: public, max-age=0, no-cache
replay-nonce: 6HUcA-o5ZppmkLjLO_19dfEDelVgGJ3hF8nvxcVt3uR1IyLYAbs
x-frame-options: DENY
strict-transport-security: max-age=604800
$ curl -I4 https://acme-v02.api.letsencrypt.org/directory
curl: (28) Failed to connect to acme-v02.api.letsencrypt.org port 443 after 131057 ms: Connection timed out
$
```
Any help much appreciated!

Accepted Answer

Hi,

If I am understanding correctly, you are running a docker container within an IPv6-only Lightsail instance.   Have you already configured IPv6 networking for the docker daemon?

https://docs.docker.com/config/daemon/ipv6/

Answer

Thank you this was the issue. I got working by creating a new file /etc/docker/daemon.json with these contents:
```
{
  "experimental": true,
  "ip6tables": true
}
```
..and updating the docker compose file to include:
```
 networks:
   ip6net:
     enable_ipv6: true
     ipam:
       config:
         - subnet: 2001:0DB8::/112
```
This sort of thing should be turned on by default in Docker, but no....

IPv6-only Lightsail Ubuntu + Debian outbound call to Let's Encrypt timing out

Relevanter Inhalt