NLB using private subnet with direct connect

Yes, you could use AWS PrivateLink to expose your service via a Network Interface (ENI) in your VPC that can be accessed over Direct Connect. This approach is useful if you want to restrict access to your service to specific IP addresses in your corporate network. However, PrivateLink is typically used for secure, private communication between different services within AWS, and may not be necessary if you're already using Direct Connect.

If you want to use an internal NLB, you can specify the subnets to associate with the NLB using the service.beta.kubernetes.io/aws-load-balancer-subnets annotation. To specify a static IP for each subnet, you can use the service.beta.kubernetes.io/aws-load-balancer-private-ipv4-addresses annotation. To choose these IP addresses, you can simply pick any unused IP address within the CIDR block of each subnet. However, you must ensure that these IP addresses are not used by any other resources within the subnet.

AWS EKS manages the IP addresses of your worker nodes independently of the IP addresses associated with your NLB. When you specify a static IP for your NLB, AWS ensures that this IP is not used by any other resources within the subnet. Therefore, restarting your EKS nodes should not affect the IP addresses associated with your NLB.

Here's how you could modify your existing service definition to create an internal NLB with static IP addresses:

apiVersion: v1
kind: Service
metadata:
  name: my-service
  namespace: my-namespace
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internal
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: "HTTP"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: "/healthz"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "8080"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-private-ipv4-addresses: "${PRIVATE_IPS}"
    service.beta.kubernetes.io/aws-load-balancer-subnets: "${PRIVATE_SUBNETS}"

This configuration creates an internal NLB that uses the specified static IP addresses and is associated with the specified subnets. The NLB will route traffic to the backend pods of your EKS service, and can be accessed over Direct Connect from your corporate data center.

Please note that you need to replace ${PRIVATE_IPS} and ${PRIVATE_SUBNETS} with a comma-separated list of IP addresses and subnet IDs, respectively. The order of the IP addresses and subnet IDs should match, i.e., the first IP address should be in the first subnet, the second IP address should be in the second subnet, etc.

The best way to achieve what you need is to delete the existing public-facing NLB and create it as a private NLB. That will give you an NLB with only private IP addresses that will not change for the lifetime of the load balancer.

I would not go with option (1) - you might be able to make that work (I've not tested it and there are some subtleties there) and in any case it would increase the cost of your solution for no particular gain.

For (3): Most things in a VPC that don't have static IP addresses (such as NLB) get their addresses from DHCP so there should never be a conflict like that.