Redirect web socket to an external service

Question

I want to forward a Web Socket (WS) connection from a domain-1 to domain-2.

The goal is that domain-1 uses weak cypher algos, TLS 1.2 and shorter URL and forward to domain2 which has TLS 1.3 only strong cypher algos and a longer domain name.

![schema](/media/postImages/original/IMFT9gQQDiR9CJlg-JkIg6kg)

I'm getting a 400 error in step (3)

Log of NGINX in domain-1:

2023/10/06 13:17:01 [notice] 1152#1152: *37717323 "(?i)/std(/|$)(.*)" matches "/std/base/key1/key2/name1", client: 10.0.179.122, server: domain-1, request: "GET /std/base/key1/key2/name1 HTTP/1.1", host: "domain-1"
2023/10/06 13:17:01 [notice] 1152#1152: *37717323 rewritten data: "/base/key1/key2/name1", args: "", client: 10.0.179.122, server: domain-1, request: "GET /std/base/key1/key2/name1 HTTP/1.1", host: "domain-1"
10.0.179.122 - - [06/Oct/2023:13:17:01 +0000] "GET /std/base/key1/key2/name1 HTTP/1.1" 400 248 "-" "-" 336 0.004 [standard-external-service-443] [] 3.79.135.121:443 248 0.005 400 3111ad9748961e237011dd0449dafe08

Log of NGINX in domain-2:

2023/10/10 13:17:01 [info] 140#140: *40730708 client sent plain HTTP request to HTTPS port while reading client request headers, client: 10.0.62.0, server: _, request: "GET /base/key1/key2/CS-SIEMENS HTTP/1.1", host: "domain-1"

We are in k8s env, with NGINX ingress + External Service:

External service:

```
    apiVersion: v1
    kind: Service
    metadata:
      name: standard-external-service
      namespace: qa
    spec:
      type: ExternalName
      externalName: domain-2

```

Ingress:

```
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: ingress-standard-external
      namespace: qa
      annotations:
        cert-manager.io/issuer: letsencrypt-prod
        nginx.ingress.kubernetes.io/enable-access-log: "true"
        nginx.ingress.kubernetes.io/enable-rewrite-log: "true"
        nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
        nginx.ingress.kubernetes.io/proxy-body-size: 1m
        nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
        nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
        nginx.ingress.kubernetes.io/rewrite-target: /$2
        nginx.ingress.kubernetes.io/secure-backends: "true"
        nginx.ingress.kubernetes.io/ssl-redirect: "true"
        nginx.ingress.kubernetes.io/use-regex: "true"
        nginx.ingress.kubernetes.io/websocket-services: standard-external-service
        nginx.org/websocket-services: standard-external-service
    spec:
      ingressClassName: nginx
      tls:
        - hosts:
            - "domain-1"
            - "*.domain-1"
          secretName: tls-secret-qa
      rules:
        - host: "domain-1"
          http:
            paths:
              # e-Mobility Standard
              - path: /std(/|$)(.*)
                pathType: Prefix
                backend:
                  service:
                    name: standard-external-service
                    port:
                      number: 443
        - host: "*.domain-1"
          http:
            paths:
              # e-Mobility Standard
              - path: /std(/|$)(.*)
                pathType: Prefix
                backend:
                  service:
                    name: standard-external-service
                    port:
                      number: 443
```

NGINX controller:

```
    {
      "apiVersion": "v1",
      "data": {
        "allow-snippet-annotations": "true",
        "keep-alive": "349",
        "large-client-header-buffers": "4 16k",
        "ssl-ciphers": "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:AES256-SHA256",
        "ssl-protocols": "TLSv1.3 TLSv1.2"
      },
      "kind": "ConfigMap",
      "metadata": {
        "labels": {
          "app.kubernetes.io/component": "controller",
          "app.kubernetes.io/instance": "ingress-nginx",
          "app.kubernetes.io/managed-by": "Helm",
          "app.kubernetes.io/name": "ingress-nginx",
          "app.kubernetes.io/part-of": "ingress-nginx",
          "app.kubernetes.io/version": "1.8.1",
          "argocd.argoproj.io/instance": "ingress-nginx",
          "helm.sh/chart": "ingress-nginx-4.7.1",
        },
        "name": "ingress-nginx-controller",
        "namespace": "ingress-nginx",
      },
    }
```

NGINX rewrites the URL and redirect the call and I got the error 400: client sent plain HTTP request to HTTPS port

I think the SSL terminaison occurs at NGINX level (not NLB) and forwarded to the domain-2, but I don't get why I have this error as SSL should be used all along!

Answer

The logs generated for both domains suggest that the requests have been processed and forwarded to reach the server of domain 2. Although, as the 400 error indicates a bad request, this would lead to the following scenario being a possible root cause of the error :

Considering the following setup at your NLB 2 where :
Listener (TLS:443) ----> Target group (TCP:443)

The NLB is configured with a TLS listener, the TLS is terminated at the NLB and only HTTP content remains to be forwarded and if the target groups are configured with the TCP protocol, meaning there is no additional TLS encryption happening at the NLB to the targets. With the HTTP traffic being forwarded to an HTTPS endpoint (port 443 on the webserver) you will find the webserver respond back with a 400 error generally with a message letting you know that a HTTP request was received on a HTTPS port.

Therefore, the web server of domain 2 would be receiving HTTP traffic from the NLB instead of the expected HTTPS traffic. The TLS encryption would make HTTPS traffic unique from HTTP traffic.

Hence, it is recommended to configure the NLB target group to establish TLS connections from the NLB to the targets, resulting in end-to-end encryption in transit from your clients to the targets.

Please refer to the following documentations for more information on the same :

1. https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-target-group.html
2. https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#target-group-routing-configuration

I would also recommend reaching out to us via support case incase you would like to troubleshoot the issue further.

Answer

Hello,
Thanks a lot for your reply!!!
I think that the TLS is terminated at NGINX level not at NLB.
I haven't attached any certificate to the NLB but there is a Let's Encrypt cert sercret for NGINX deployed by cert-manager.
So I think that the HTTP request is sent by NGINX and it should be configured to not terminate SSL or reopen a new SSL connexion toward the domain-2.
I'll open a ticket to the support.
Thanks,
Serge.

Redirect web socket to an external service

Relevant content