Workaround ideas for EKS Node Monitoring Agent listening port

While the manual update of the DaemonSet is one workaround for the EKS Node Monitoring Agent port issue, there are indeed more sustainable approaches that could be better suited for production workloads and maintain automation. Here are some alternative ideas:

1. Custom Prometheus Configuration: Instead of modifying the Node Monitoring Agent, you could configure Prometheus to scrape metrics from a different port. This approach keeps the agent untouched while still allowing you to collect the necessary metrics.

2. Service Monitor Approach: Implement a ServiceMonitor custom resource, which is part of the Prometheus Operator. This allows you to define how Prometheus should discover and scrape metrics from your services, including the Node Monitoring Agent, without directly modifying the agent's configuration.

3. Proxy Solution: Deploy a lightweight proxy (like Envoy or NGINX) as a sidecar container alongside the Node Monitoring Agent. Configure this proxy to listen on a different port and forward requests to the agent's 8080 port. This way, you can expose a different port externally without changing the agent itself.

4. Custom Init Container: Use an init container in the Node Monitoring Agent's pod to dynamically modify the agent's configuration at startup. This could change the listening port without manually editing the DaemonSet.

5. GitOps Approach: If you're using a GitOps workflow, you could maintain a customized version of the Node Monitoring Agent manifest in your repository. This allows you to apply your modifications automatically as part of your deployment process.

6. Kubernetes Operator: Develop a custom Kubernetes operator that manages the Node Monitoring Agent deployment, including any necessary modifications. This operator could automatically apply the port change whenever the agent is deployed or updated.

7. Network Policy: If the port conflict is due to security concerns, implement Kubernetes Network Policies to restrict access to the 8080 port rather than changing the port itself.

These approaches aim to provide more automated, scalable solutions that can be integrated into your existing workflows and are better suited for production environments. The best choice will depend on your specific infrastructure setup and operational requirements.
Sources
Monitoring - Amazon EKS
Amazon EKS enhances Kubernetes control plane observability | Containers
View the health status of your nodes - Amazon EKS