如何在我的 Amazon EKS 叢集上允許從外部存取多個 Kubernetes 服務?

7 分的閱讀內容
0

我想在我的 Amazon Elastic Kubernetes Service (Amazon EKS) 叢集上允許從外部存取多個 Kubernetes 服務。

簡短說明

使用 NGINX 輸入控制器或適用於 Kubernetes 的 AWS 負載平衡器控制器,從外部存取 Amazon EKS 叢集中的多個 Kubernetes 服務。NGINX 輸入控制器主要由 NGINX 進行維護。如要檢查 NGINX 輸入控制器是否有問題,請參閱 GitHub 網站的問題清單。AWS 負載平衡器控制器由 Amazon Web Services (AWS) 進行維護。如要檢查 AWS 負載平衡器控制器是否有問題,請參閱 GitHub 網站的問題清單

**重要事項:**輸入控制器和 IngressClass (來自 Kubernetes 網站) 與 輸入 (來自 Kubernetes 網站) 不同。輸入是 Kubernetes 資源,可從叢集外將 HTTP 和 HTTPS 路由公開至叢集中的服務。輸入控制器通常透過負載平衡器履行輸入。您必須擁有輸入控制器才能使用輸入。IngressClass 用於識別要使用哪個輸入控制器以履行輸入物件請求。

先決條件: 安裝 AWS 負載平衡器控制器。最佳實務是使用 AWS 負載平衡器控制器來建立和管理 Amazon EKS 中適用於 LoadBalancer 類型服務物件的 Network Load Balancer。

解決方法

下列解決方法會在 Kubernetes GitHub 網站上使用 kubernetes/ingress-nginx 輸入控制器。另一個可公開使用的輸入控制器為 NGINX GitHub 網站上的 nginxinc/kubernetes-ingress

部署適用於 Kubernetes 的 NGINX 輸入控制器

您可以透過 Transmission Control Protocol (TCP) 或 Transport Layer Security (TLS),部署適用於 Kubernetes 的 NGINX 輸入控制器。

**注意:**下列解決方法使用 Amazon EKS 1.22 版、NGINX 輸入控制器 1.3.0 版及 AWS 負載平衡器控制器 2.4.3 版進行測試。

(選項 1) 使用 Network Load Balancer 的 TCP 與 NGINX 輸入控制器搭配使用

1.    讓 YAML 檔案部署下列 Kubernetes 物件:命名空間serviceaccountsconfigmapclusterrolesclusterrolebindings角色rolebindings服務部署ingressclassesvalidatingwebhookconfigurations

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/aws/deploy.yaml

2.    編輯檔案。接著,在 ingress-nginx-controller 服務物件區段中,將所有 service.beta.kubernetes.io 註釋替換為下列內容:

service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-type: "external"
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "instance"
service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"

3.    套用清單檔案:

kubectl apply -f deploy.yaml

輸出範例:

namespace/ingress-nginx created
serviceaccount/ingress-nginx created
configmap/ingress-nginx-controller created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
service/ingress-nginx-controller-admission created
service/ingress-nginx-controller created
deployment.apps/ingress-nginx-controller created
ingressclass.networking.k8s.io/nginx created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
serviceaccount/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created

(選項2) Network Load Balancer 上的 NGINX 輸入控制器 TLS 終止

根據預設,上一個解決方案在 NGINX 輸入控制器中終止 TLS。您也可以設定 NGINX 輸入服務,在 Network Load Balancer 終止 TLS。

1.    下載 deploy.yaml 範例:

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/aws/nlb-with-tls-termination/deploy.yaml

2.    編輯檔案。接著,在 ingress-nginx-controller 服務物件區段中,將所有 service.beta.kubernetes.io 註釋替換為下列內容:

service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-west-2:XXXXXXXX:certificate/XXXXXX-XXXXXXX-XXXXXXX-XXXXXXXX
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
service.beta.kubernetes.io/aws-load-balancer-type: "external"
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "instance"
service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"

**注意:**請務必包含您 service.beta.kubernetes.io/aws-load-balancer-ssl-cert 的 ARN。

3.    編輯檔案,並變更 Kubernetes 叢集的 Amazon Virtual Private Cloud (Amazon VPC) CIDR:

proxy-real-ip-cidr: XXX.XXX.XXX/XX

4.    套用清單檔案:

kubectl apply -f deploy.yaml

注意:先前的清單檔案使用 ExternalTrafficPolicy 作為本機以保留來源 (用戶端) IP 地址。在 Amazon VPC 使用此組態搭配自訂 DHCP 名稱會發生問題。如要避免發生問題,請將下列修補程式套用至 kube-proxy

kubectl edit daemonset kube-proxy -n kube-system

5.    編輯清單檔案以包含下列程式碼片段:

spec:
  template:
    spec:
      containers:
        - name: kube-proxy
          command:
            - kube-proxy
            - --hostname-override=$(NODE_NAME)
            - --v=2
            - --config=/var/lib/kube-proxy-config/config
           env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName

確認部署的資源

AWS 負載平衡器控制器

命令:

kubectl get all -n kube-system --selector app.kubernetes.io/instance=aws-load-balancer-controller

輸出範例:

NAME                                                READY   STATUS    RESTARTS   AGE   IP               NODE                                           NOMINATED NODE   READINESS GATES
pod/aws-load-balancer-controller-85cd8965dc-ctkjt   1/1     Running   0          48m   192.168.37.36    ip-192-168-59-225.us-east-2.compute.internal   none             none
pod/aws-load-balancer-controller-85cd8965dc-wpwx9   1/1     Running   0          48m   192.168.53.110   ip-192-168-59-225.us-east-2.compute.internal   none>         none
NAME                                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/aws-load-balancer-webhook-service   ClusterIP   10.100.154.44   none          443/TCP   19h   app.kubernetes.io/instance=aws-load-balancer-controller,app.kubernetes.io/name=aws-load-balancer-controller
NAME                                           READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS                   IMAGES                                                                                    SELECTOR
deployment.apps/aws-load-balancer-controller   2/2     2            2           19h   aws-load-balancer-controller 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-load-balancer-controller:v2.4.0   app.kubernetes.io/instance=aws-load-balancer-controller,app.kubernetes.io/name=aws-load-balancer-controller
NAME                                                      DESIRED   CURRENT   READY   AGE   CONTAINERS                     IMAGES                                                                                    SELECTOR
replicaset.apps/aws-load-balancer-controller-85cd8965dc   2         2         2       19h   aws-load-balancer-controller   602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-load-balancer-controller:v2.4.0   app.kubernetes.io/instance=aws-load-balancer-controller,app.kubernetes.io/name=aws-load-balancer-controller,pod-template-hash=85cd8965dc

NGINX 輸入控制器

命令:

kubectl get all -n ingress-nginx --selector app.kubernetes.io/instance=ingress-nginx

輸出範例:

NAME                                           READY  STATUS   RESTARTS  AGE   IP               NODE                                           NOMINATED NODE  READINESS GATES
pod/ingress-nginx-controller-54d8b558d4-k4pdf  1/1    Running  0         56m   192.168.46.241   ip-192-168-59-225.us-east-2.compute.internal   none            none
NAME                                        TYPE           CLUSTER-IP      EXTERNAL-IP                                                                     PORT(S)                      AGE   SELECTOR
service/ingress-nginx-controller            LoadBalancer   10.100.99.129   ad9bba7a8239a475297d24bd2f617782-a579e639079f8270.elb.us-east-2.amazonaws.com   80:32578/TCP,443:30724/TCP   15h   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
service/ingress-nginx-controller-admission  ClusterIP      10.100.190.61   none                                                                            443/TCP                      15h   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
NAME                                       READY   UP-TO-DATE   AVAILABLE    AGE   CONTAINERS   IMAGES                                                                                                               SELECTOR
deployment.apps/ingress-nginx-controller   1/1     1            1            15h   controller   k8s.gcr.io/ingress-nginx/controller:v1.1.1@sha256:0bc88eb15f9e7f84e8e56c14fa5735aaa488b840983f87bd79b1054190e660de   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
NAME                                                  DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES                                                                                                               SELECTOR
replicaset.apps/ingress-nginx-controller-54d8b558d4   1         1         1       15h   controller   k8s.gcr.io/ingress-nginx/controller:v1.1.1@sha256:0bc88eb15f9e7f84e8e56c14fa5735aaa488b840983f87bd79b1054190e660de   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=54d8b558d4
NAME                                       COMPLETIONS   DURATION   AGE   CONTAINERS   IMAGES                                                                                                                         SELECTOR
job.batch/ingress-nginx-admission-create   1/1           2s         15h   create       k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1@sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660   controller-uid=242bdf56-de16-471d-a691-1ca1dbc10a41
job.batch/ingress-nginx-admission-patch    1/1           2s         15h   patch        k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1@sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660   controller-uid=a9e710d2-5001-4d40-a435-ddc8993bfe42

IngressClass

命令:

kubectl get ingressclass

輸出範例:

NAME    CONTROLLER             PARAMETERS                             AGE
alb     ingress.k8s.aws/alb    IngressClassParams.elbv2.k8s.aws/alb   19h
nginx   k8s.io/ingress-nginx   none                                   15h

測試部署設定

**注意:**下列步驟可執行兩個微型服務。微型服務會以預設類型透過 Kubernetes 在內部公開。

1.    設定您的部署或微型服務。例如,hostname-appapache-app

hostname-apphostname-app-svc.yaml 檔案範例:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hostname-app
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: hostname-app
  template:
    metadata:
      labels:
        app: hostname-app
    spec:
      containers:
      - name: hostname-app
        image: k8s.gcr.io/serve_hostname:1.1

---
apiVersion: v1
kind: Service
metadata:
  name: hostname-svc
  namespace: default
spec:
  ports:
  - port: 80
    targetPort: 9376
    protocol: TCP
  selector:
    app: hostname-app

apache-appapache-app-svc.yaml 檔案範例:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: apache-app
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: apache-app
  template:
    metadata:
      labels:
        app: apache-app
    spec:
      containers:
      - name: apache-app
        image: httpd:latest
        ports:
        - containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: apache-svc
  namespace: default
  labels:
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
  selector:
    app: apache-app

2.    套用您的組態。

hostname-app:

kubectl apply -f hostname-app-svc.yaml

apache-app:

kubectl apply -f apache-app-svc.yaml

3.    確認已建立資源。

部署

命令:

kubectl get deployment hostname-app apache-app -n default

輸出範例:

NAME           READY   UP-TO-DATE   AVAILABLE   AGE
hostname-app   2/2     2            2           29m
apache-app     2/2     2            2           29m

服務

命令:

kubectl get svc apache-svc hostname-svc -n default

輸出範例:

NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
apache-svc     ClusterIP   10.100.73.51    none          80/TCP    29m
hostname-svc   ClusterIP   10.100.100.44   none          80/TCP    29m

測試 NGINX 輸入控制器

1.    存取您從命令列擷取的負載平衡器 DNS 網址:

curl -I ad9bba7a8239a475297d24bd2f617782-a579e639079f8270.elb.us-east-2.amazonaws.com

輸出範例:

HTTP/1.1 404 Not Found
Date: Thu, 03 Mar 2022 14:03:11 GMT
Content-Type: text/html
Content-Length: 146
Connection: keep-alive

注意:預設伺服器會對所有未定義輸入規則的網域請求傳回找不到 404 頁面。根據定義的規則,除非請求與組態相符,否則輸入控制器不會將流量導向特定後端服務。由於託管欄位設定為輸入物件,您必須為請求的託管標頭提供相同託管名稱。在測試環境中,請使用 curl 旗標提供託管標頭。在生產環境中,將負載平衡器 DNS 名稱對應至任何 DNS 供應商的託管名稱,例如 Amazon Route 53。

2.    實作輸入,以透過 NGINX 輸入控制器提供的單一負載平衡器連接您的服務。

範例 micro-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: micro-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
    - host: hostname.mydomain.com
      http:
        paths:
        - backend:
            service:
              name: hostname-svc
              port:
                number: 80
          path: /
          pathType: Prefix
  - host: apache.mydomain.com
    http:
      paths:
      - backend:
          service:
            name: apache-svc
            port:
              number: 80
        path: /
        pathType: Prefix

**注意:**如需詳細資訊,請參閱名稱型虛擬主機 (來自 Kubernetes 網站)。

3.    確認已建立資源。

輸入

命令:

kubectl get ingress -n default

輸出範例:

NAME           CLASS   HOSTS                                       ADDRESS                                                                         PORTS   AGE
micro-ingress  none    hostname.mydomain.com,apache.mydomain.com   ad9bba7a8239a475297d24bd2f617782-a579e639079f8270.elb.us-east-2.amazonaws.com   80      29m

4.    為請求新增託管標頭。

第一個已設定的網域:

curl -i -H "Host: hostname.mydomain.com" http://aaa71bxxxxx-11xxxxx10.us-east-2.elb.amazonaws.com/

輸出範例:

HTTP/1.1 200 OK Date: Sat, 26 Mar 2022 18:50:38 GMT Content-Type: text/plain; charset=utf-8 Content-Length: 29 Connection: keep-alive

第二個已設定的網域:

curl -i -H "Host: apache.mydomain.com" http://aaa71bxxxxx-11xxxxx10.us-east-2.elb.amazonaws.com/

輸出範例:

HTTP/1.1 200 OK
Date: Sat, 26 Mar 2022 18:51:00 GMT
Content-Type: text/html
Content-Length: 45
Connection: keep-alive
Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT
ETag: "2d-432a5e4a73a80"
Accept-Ranges: bytes

新增託管標頭後,如果流量與 Ingress 定義的組態相符,輸入控制器就會將流量重新導向後端設定服務。

如要保留相同的網域名稱,但轉移存取該路徑的流量,您必須透過輸入新增路徑路由。

例如:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: path-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: nginxd
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: mydomain.com
    http:
      paths:
      - backend:
          service:
            name: hostname-svc
            port:
              number: 80
        path: /hostname
        pathType: Prefix
  - host: mydomain.com
    http:
      paths:
      - backend:
          service:
            name: apache-svc
            port:
              number: 80
        path: /apache
        pathType: Prefix

注意:如果要求的託管標頭為 mydomain.com,則前述範例只會傳回 200 回應。請求可以透過 /hostname/apache 路徑存取。所有其他請求則會傳回 404 回應。

5.    確認已新增路徑路由:

命令:

kubectl get ingress -n default

輸出:

NAME            CLASS  HOSTS                                       ADDRESS                                                                         PORTS  AGE
micro-ingress   none   hostname.mydomain.com,apache.mydomain.com   ad9bba7a8239a475297d24bd2f617782-a579e639079f8270.elb.us-east-2.amazonaws.com   80     164m
path-ingress    none   mydomain.com,mydomain.com                   ad9bba7a8239a475297d24bd2f617782-a579e639079f8270.elb.us-east-2.amazonaws.com   80     120m

命令:

curl -i -H "Host: mydomain.com" http://aaa71bxxxxx-11xxxxx10.us-east-2.elb.amazonaws.com/hostname
  • 或 -
curl -i -H "Host: mydomain.com" http://aaa71bxxxxx-11xxxxx10.us-east-2.elb.amazonaws.com/apache

使用 AWS 負載平衡器控制器測試輸入

1.    使用下列輸入清單檔案範例來啟動 Application Load Balancer:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: micro-ingress-alb
  namespace: default
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
  - host: alb.hostname.mydomain.com
    http:
      paths:
      - backend:
          service:
            name: hostname-svc
            port:
              number: 80
        path: /
        pathType: Prefix
   - host: alb.apache.mydomain.com
     http:
       - backend:
            service:
              name: apache-svc
              port:
                number: 80
         path: /
         pathType: Prefix

2.    確認啟動 Application Load Balancer。

命令:

kubectl get ingress -n default

輸出:

NAME               CLASS  HOSTS                                               ADDRESS                                                                         PORTS  AGE
micro-ingress      none   hostname.mydomain.com,apache.mydomain.com           ad9bba7a8239a475297d24bd2f617782-a579e639079f8270.elb.us-east-2.amazonaws.com   80     164m
micro-ingress-alb  none   alb.hostname.mydomain.com,alb.apache.mydomain.com   k8s-default-microing-8a252bde81-1907206594.us-east-2.elb.amazonaws.com          80     18m
path-ingress       none   mydomain.com,mydomain.com                           ad9bba7a8239a475297d24bd2f617782-a579e639079f8270.elb.us-east-2.amazonaws.com   80     120m

根據第一個已設定網域的請求:

curl -i -H "Host: alb.hostname.mydomain.com" http://k8s-default-microing-8a252bde81-1907206594.us-east-2.elb.amazonaws.com

輸出範例:

HTTP/1.1 200 OK Date: Sat, 26 Mar 2022 20:46:02 GMT Content-Type: text/plain; charset=utf-8 Content-Length: 29 Connection: keep-alive

根據第二個已設定網域的請求:

curl -i -H "Host: alb.apache.mydomain.com" http://k8s-default-microing-8a252bde81-1907206594.us-east-2.elb.amazonaws.com

輸出範例:

HTTP/1.1 200 OK
Date: Sat, 26 Mar 2022 20:46:14 GMT
Content-Type: text/html Content-Length: 45
Connection: keep-alive
Server: Apache/2.4.53 (Unix)
Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT
ETag: "2d-432a5e4a73a80"
Accept-Ranges: bytes

AWS 官方
AWS 官方已更新 2 年前