Proactive Strategies for Optimizing AWS CloudTrail Lake Usage

The Usage Spike: What Happened?

A recent inquiry from a customer highlighted a serious issue: Their AWS CloudTrail usage had unexpectedly spiked to hundreds of thousands of dollars.
Identifying the root cause became a top priority.

By leveraging AWS Cost Explorer, and filtering by CloudTrail service, grouping by Usage Type, we quickly pinpointed that 90% of their CloudTrail usage came from the following parameters:

• Region: EU (Ireland)
• Usage Type: EU-Ingestion-Bytes

---

Understanding EU-Ingestion-Bytes

The EU-Ingestion-Bytes usage type is directly linked to the ingestion of events in AWS CloudTrail Lake under the Seven-Year Retention Pricing Option. CloudTrail Lake charges based on the volume of ingested data, and every type of event can drive this cost up.

AWS CloudTrail Lake encompasses a variety of event categories:

• CloudTrail Events – Management and data events.
• Insights Events – Behavioral anomalies detected by CloudTrail Insights.
• Configuration Items – Changes recorded in AWS Config.
• Integration Events – Custom audit events ingested via PutAuditEvents.
• AWS Audit Manager Events – Evidence collection for compliance tracking.

With this understanding, we moved on to analyze the event sources contributing to this spike in ingestion costs.

---

Identifying High-Volume CloudTrail Lake Event data stores

To identify the specific events driving up ingestion costs, we turned to Lake Query, a powerful tool for querying Event data stores.

Here’s how we approached the analysis:

1. Sign in to the AWS Management Console and open the CloudTrail console at https://console.aws.amazon.com/cloudtrail/.
2. From the navigation pane, under Lake, choose Query.
3. On the Editor tab, choose the event data store for which you want to run the query. When you choose the event data store from the list, CloudTrail automatically populates the event data store ID in the FROM line of the query editor.
4. Execute an Lake Query – We used the below query to isolate the top event sources and event names consuming the most CloudTrail ingestion bytes:
5. You can also Choose a Sample Query that can be helpful to edit and adapt to your needs, from the Sample queries tab.

SELECT eventName, count(eventName) AS apiCallCount, eventSource 
FROM $EDS_ID 
WHERE eventTime > DATE_ADD('month', -1, CURRENT_TIMESTAMP) 
GROUP BY eventName, eventSource 
ORDER BY apiCallCount DESC;

Alternatively, for a broader search, you can find events by their event ID and event name across multiple event data stores. The results are selected from each event data store first, then results are concatenated, ordered by event ID, and limited to ten events.

Select eventId, eventName from $EDS_ID1
UNION
Select eventId, eventName from $EDS_ID2
UNION ALL
Select eventId, eventName from $EDS_ID3 
ORDER BY eventId;

3. Findings: Upon executing these queries, we discovered that AWS KMS Events and Amazon RDS Data API Events were the primary factor driving excessive ingestion costs.

---

Optimizing CloudTrail to Reduce Usage

To address the high CloudTrail ingestion usage, we recommended several optimizations, including the exclusion of certain high-volume event types from the CloudTrail configuration:

1. Excluding AWS KMS Events

AWS Key Management Service (KMS) operations such as Encrypt, Decrypt, and GenerateDataKey generate a significant volume of read events, often accounting for more than 99% of total logs. While read events are crucial for tracking, the write events (e.g., Disable, Delete, and ScheduleKey) are essential for maintaining security and compliance.

Cost Optimization Recommendations:

• Exclude all KMS read events if they are not required.
• Alternatively, log only write events to focus on critical security changes while reducing the volume of logs.

2. Excluding Amazon RDS Data API Events

By default, CloudTrail logs all Amazon RDS Data API events, which can significantly increase storage and ingestion costs. If auditing these events is not essential, we recommended disabling logging for RDS Data API calls.

---

The Results: A Dramatic Usage Reduction

After implementing these changes, the customer saw a significant drop in their CloudTrail usage. By refining their CloudTrail configuration, they managed to:

• Eliminate unnecessary, high-volume events from their logs.
• Retain essential audit logs for security and compliance purposes.
• Avoid a potential six-figure CloudTrail bill.

---

Key Takeaways

• Monitor CloudTrail Costs Regularly – Use AWS Cost Explorer and AWS Budgets to track and identify unusual cost spikes early.
• Leverage Lake Query for In-Depth Insights – Query your CloudTrail logs, including Event data stores, to find and address the events contributing to the highest costs.
• Exclude Unnecessary High-Volume Events – By filtering out non-essential events, you can drastically reduce ingestion and storage costs.
• Continuously Adjust Configurations – CloudTrail settings should be reviewed and adjusted over time to align with your evolving audit requirements.

By adopting a proactive approach to managing CloudTrail logs, organizations can significantly cut costs while still maintaining the security and compliance visibility they need.

---

For additional details, please refer to:

• Optimize CloudTrail Lake Queries
• AWS CloudTrail Lake Announces Enhanced Event Filtering