How can I reduce the resource requests/limits for DaemonSets created by the AWS CloudWatch Observability add-on in EKS?

Question

I’ve enabled Advanced CloudWatch Observability for my Amazon EKS cluster using the AWS-provided add-on. This automatically deploys several DaemonSets such as Fluent Bit and CLoudwatch Agent Collector for logs and metrics.

I’ve noticed that the resource requests and limits (CPU/memory) for these DaemonSets are higher than what I want to allocate, especially on smaller nodes.

However, I’m unable to directly modify these DaemonSets or update the resource values using kubectl because the changes get reverted or are managed by the add-on.

Is there a recommended way to override or reduce the resource settings (requests/limits) for these DaemonSets without disabling the add-on entirely?

Things I’ve tried/considered:

Editing the deployment manually (gets reverted)

Searching for Helm values or ConfigMaps to override settings (couldn’t find any)

Any advice on how to customize resource settings while keeping the add-on managed would be helpful.

Thanks in advance!

re:Post Agent · Answer

To reduce the resource requests and limits for DaemonSets created by the AWS CloudWatch Observability add-on in your EKS cluster, you have a few options:

1. Update the DaemonSet configuration file: You can extract the current DaemonSet configuration, modify the resource settings, and apply the changes. For example:
```
kubectl get pod cloudwatch-agent-xyz123 -n amazon-cloudwatch -o yaml | grep -A10 'resources:'
```
Then update the resources section in the DaemonSet YAML file with lower values:
```
resources:
requests:
cpu: 100m
memory: 200Mi
limits:
cpu: 200m
memory: 400Mi
```
And apply the changes:
```
kubectl apply -f cloudwatch-agent-daemonset.yaml
```

2. Use the conflict resolution option: When installing or updating the CloudWatch Observability EKS add-on, you can use the conflict resolution configuration option with "OVERWRITE" parameter. This allows you to proceed with customized settings by overwriting the conflicts on the cluster. In the Amazon EKS console, you'll find this under "Optional configuration settings" when creating or updating the add-on. If using AWS CLI, you can add the `--resolve-conflicts OVERWRITE` parameter to your command.

3. Remove existing setup and reinstall: If you're experiencing issues with the current setup, you can delete the existing CloudWatch agent setup from your cluster, back up any customizations you've made, and then reinstall the EKS add-on with your preferred resource settings. Make sure to back up any custom agent configurations before doing this.

4. Use node labels to control DaemonSet deployment: You can use node labels and the NodeAffinity setting to control which nodes the DaemonSets run on, potentially avoiding deployment on smaller nodes where resources are more constrained.

Remember to monitor your pods after making these changes to ensure they're not entering a CrashLoopBackOff state due to insufficient resources. You can use tools like `kubectl top pods` to monitor resource usage and further adjust your settings as needed.  
**Sources**  
[Resolve CrashLoopBackOff state for pods in Amazon EKS cluster | AWS re:Post](https://repost.aws/knowledge-center/eks-crashloopbackoff-cloudwatch-pod-identity)  
[How to monitor high resource usage in Amazon EKS  tasks and services | AWS re:Post](https://repost.aws/articles/AR6-LmkFXTS2W-zmXgUwY4PQ/how-to-monitor-high-resource-usage-in-amazon-eks-tasks-and-services)  
[Kubernetes Control Plane - Amazon EKS](https://docs.aws.amazon.com/eks/latest/best-practices/scale-control-plane.html)  
[Troubleshooting your Application Signals installation - Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Application-Signals-Enable-Troubleshoot.html)

How can I reduce the resource requests/limits for DaemonSets created by the AWS CloudWatch Observability add-on in EKS?

Add your answer

Relevant content