EKS Addon Guardduty agent can not start

Question

I'm using EKS 1.32 with  seft managed nodegroup (ami-008f1bfea3518dc59)  and  enable addons kubeproxy, coredns , guardduty agent.
* when using guardduty agent version 1.8.1. it running OK.
* when upgrade or create new with version 1.10.0, the daemonset aws-guardduty-agent can not running and raise error

```
2025-06-30T17:11:35.665623Z  INFO amzn_guardduty_agent: GuardDuty agent starting with 8 worker thread(s) and 100 max blocking threads.
2025-06-30T17:11:35.693141Z  INFO amzn_guardduty_agent: Agent fingerprint: 91c321fb96defd24ed9504c9622e39f9491d222fbce0e13c9cc5b9a4e8c81d14
2025-06-30T17:11:35.693161Z  INFO amzn_guardduty_agent: Set the log file retention from default value to 12.0 hour.
2025-06-30T17:11:35.693236Z  INFO amzn_guardduty_agent: Agent config for Component(s): AgentConfig { component_channel_size: 100, ingestion_endpoint: None, generate_test_messages: false, test_messages_size: 1000, integrity_check_file: None, agent_version: Some("91c321fb96defd24ed9504c9622e39f9491d222fbce0e13c9cc5b9a4e8c81d14"), raw_os_version: "Amazon_Linux_2023.7.20250609", os_version: OSUnknown, kernel_version: LinuxKernel6_1, region: None, stage: None, agent_id: "2af8b460-b914-45b7-b5fd-6388d1a7508e", agent_publish_metrics_to_end_point_period: 300s, process_cache_capacity: 16384, container_cache_capacity: 16384, pod_cache_capacity: 16384, task_cache_capacity: 16384, pid_namespace_cache_capacity: 1024, pre_suppressor_cache_capacity: 32768, post_suppressor_cache_capacity: 32768, docker_socket_file_path: "/agent_run/docker.sock", containerd_socket_file_path: "/agent_run/containerd/containerd.sock", scan_for_process_metadata_period: 900s, scan_for_container_metadata_period: 600s, scan_for_pod_metadata_period: 600s, scan_for_task_metadata_period: 600s, eks_cluster_name_env_var: "CLUSTER_NAME", max_file_size_to_hash: 536870912, proc_folder_path: "/host_proc", agent_runtime_environment: EksEc2, ecs_fargate_endpoint_env_var: "ECS_CONTAINER_METADATA_URI_V4", ecs_ec2_endpoint: "http://localhost:51678", log_file_retention: 12.0, agent_log_dir: "", file_hash_calculation_timeout: 5s, file_open_timeout: 1.5s, mnt_ns_pids_map_capacity: 1024, pid_set_cap: 32 }

2025-06-30T17:11:35.710358Z  INFO amzn_guardduty_agent::dependency_checks: Dependency checks complete.
2025-06-30T17:11:35.713432Z  INFO amzn_guardduty_agent_data_model::schema: Event schema rabin fingerprint = "d2ef81ab24449fad"
2025-06-30T17:11:35.713528Z  INFO amzn_guardduty_agent_data_model::schema: Container schema rabin fingerprint = "5a675cafcf525a5d"
2025-06-30T17:11:35.713566Z  INFO amzn_guardduty_agent_data_model::schema: Pod schema rabin fingerprint = "8504ac7492b3cb8b"
2025-06-30T17:11:35.713619Z  INFO amzn_guardduty_agent_data_model::schema: Task schema rabin fingerprint = "195a73fa36862425"
2025-06-30T17:11:35.716077Z  INFO amzn_guardduty_agent: GuardDuty agent started ...
2025-06-30T17:11:35.716086Z  INFO amzn_guardduty_agent: Type Ctrl+C to terminate
2025-06-30T17:11:35.820652Z ERROR amzn_guardduty_agent_publisher::publisher: Agent exits due to AccessDeniedException
2025-06-30T17:11:35.821416Z  INFO amzn_guardduty_agent: Agent receives SHUTDOWN signal from Publisher
2025-06-30T17:11:35.821427Z  INFO amzn_guardduty_agent: Shutdown initiated ...
2025-06-30T17:11:35.821448Z  INFO amzn_guardduty_agent::controller: Controller terminated ...
2025-06-30T17:11:35.821784Z  INFO amzn_guardduty_agent: GuardDuty agent terminated
2025-06-30T17:11:35.821789Z  INFO amzn_guardduty_agent_metrics::metrics_manager: Metrics Manager terminated ...
```
It's also occur with EKS 1.31 and 1.33

re:Post Agent · Answer

The issue you're experiencing with the GuardDuty agent version 1.10.0 failing to start while version 1.8.1 works correctly appears to be related to an access permission problem, as indicated by the "AccessDeniedException" error in your logs.

Here are some troubleshooting steps to resolve this issue:

1. **VPC Endpoint Configuration**: When managing the GuardDuty agent manually, you need to create a VPC endpoint as a prerequisite. This endpoint helps GuardDuty receive runtime events from your EKS cluster. Verify that your VPC endpoint is properly configured and accessible from your EKS nodes.

2. **Admission Controller Configuration**: Before updating to a newer agent version (like 1.10.0), you need to add the GuardDuty image registry to the `allowed-container-registries` in your admission controller. The error might be occurring because your admission controller is blocking the new agent version.

3. **Resolve Conflicts**: If you're seeing configuration conflicts when updating the addon, you might need to use the `OVERWRITE` resolve conflict flag. This will overwrite any changes made directly to the related resources in Kubernetes.

4. **Security Policies**: Review any security policies that might be blocking the addon update. You may need to either disable the controller (webhook) or configure the controller to accept requests from Amazon EKS.

5. **Network Connectivity**: Ensure that the required network connectivity is available between your EKS nodes and the GuardDuty service. The "AccessDeniedException" suggests that the agent cannot communicate with the GuardDuty service properly.

If the issue persists, you might want to consider removing the Amazon EKS add-on from your cluster and then reinstalling it with the desired version. This approach can help resolve configuration conflicts that might be preventing the newer agent version from starting correctly.  
**Sources**  
[Runtime coverage and troubleshooting for Amazon EKS clusters - Amazon GuardDuty](https://docs.aws.amazon.com/guardduty/latest/ug/eks-runtime-monitoring-coverage.html)  
[GuardDuty security agent release versions - Amazon GuardDuty](https://docs.aws.amazon.com/guardduty/latest/ug/runtime-monitoring-agent-release-history.html)  
[Managing security agent manually for Amazon EKS cluster - Amazon GuardDuty](https://docs.aws.amazon.com/guardduty/latest/ug/managing-gdu-agent-eks-manually.html)

bryan · Answer

I encountered the same issue as described by OP when upgrading the GuardDuty addon.

To further debug this issue, you can increase the log level for the AWS GuardDuty pod to identify the root cause. To enable debug logging, update the `aws-guardduty-agent` DaemonSet with an additional environment variable for the container:

```
- name: RUST_LOG
   value: debug
```

This revealed the following error:

`User: arn:aws:sts::123456789:assumed-role/aws:ec2-instance/i-yyyyyyyyyy is not authorized to perform: guardduty:SendSecurityTelemetry on resource: arn:aws:guardduty:us-east-1:123456789:detector/zzzzzzzzzzzzzzzzz with an explicit deny in a VPC endpoint policy`

After seeing this error, I discovered that I had incorrectly used the condition `"aws:Principal"` instead of the correct `"aws:PrincipalAccount"` as documented in:

https://docs.aws.amazon.com/guardduty/latest/ug/eksrunmon-prereq-deploy-security-agent.html

EKS Addon Guardduty agent can not start

Add your answer

Relevant content