How to know from which Pod DNS request is originated?

Question

I am using AWS GuardDuty and EKS. Recently I got couple of alerts from AWS GuardDuty for DefenseEvasion:EC2/UnusualDNSResolver mentioning one of the EKS nodes are connecting to 1.1.1.1. When I check the running pods in the node, I couldn't find any pods suspicious.

Is there a way to identify which pod or which process is exactly making the request to public DNS server?

Answer

Because the data source of GuardDuty finding **DefenseEvasion:EC2/UnusualDNSResolver** is VPC flow logs[1], so you can firstly collect the VPC flow logs into S3 [2] to trace the events (which is not collected by default), the query tool for VPC flow logs could be Athena[3] (using SQL), and then figure out which and when the source IP address(s) of EC2 instance and Pod established connection to 1.1.1.1 [4].

[1] https://docs.aws.amazon.com/guardduty/latest/ug/guardduty_finding-types-ec2.html#defenseevasion-ec2-unusualdnsresolver  
[2] https://docs.aws.amazon.com/vpc/latest/userguide/working-with-flow-logs.html  
[3] https://aws.amazon.com/blogs/networking-and-content-delivery/analyze-vpc-flow-logs-with-point-and-click-amazon-athena-integration/  
[4] https://aws.amazon.com/blogs/networking-and-content-delivery/using-vpc-flow-logs-to-capture-and-query-eks-network-communications/

How to know from which Pod DNS request is originated?

Relevant content