EC2 of private subnet to S3 bucket flow log
Topology: I have EC2 instances hosted in a private subnet. This EC2 instance communicates to S3 bucket through NAT gateway.
Requirement: Analyse "which EC2 communicates with which S3 bucket (bucket name)".
Failed experiments to get EC2 to S3 bucket log:
- Using VPC flow logs of subnet: It has S3 bucket IP as destination IP. But as S3 bucket IP is ephemeral. So, it cannot be co-related to S3 bucket name.
- Using NAT flow logs: It has EC2 IP, S3 IP and NAT IP. Again as S3 IP is ephemeral, it cannot be co-related to S3 bucket name
- Using Cloudtrail logs of S3 bucket: It has NAT IP as Source IP instead of EC2 IP.
Question: Is there any way/logs by which I can get which EC2 is talking to which S3 bucket information directly or by co-relating 2 or more logs? Is there any setting that needs to be done to get this information.
- Newest
- Most votes
- Most comments
Flow logs are not the appropriate tool to use here because (as you've discovered) the bucket name isn't related to the IP address. Instead, it is part of the higher-level protocol (it's transmitted as part of the HTTPS request to the S3 API) - so even if you were doing deep-packet inspection it would still be encrypted which makes it difficult (again) to determine.
To get the information you require, you might try the following:
- Create a S3 Gateway Endpoint - this has no extra charge (in fact, it may reduce your NAT Gateway charges) and is transparent to your application(s).
- Enable S3 access logging - note that you will pay for storage of the logs so I'd recommend deleting old logs to save costs.
- The access log format has the source IP address of the requester in it.
Followup question: What do you need this information for?
Relevant content
- asked 3 years ago
AWS OFFICIALUpdated 3 years ago- AWS OFFICIALUpdated 7 months ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 years ago
Thanks for the answer! We need this information to monitor communication between EC2 and S3 for some analytics purpose.