How to monitor athena resource usage per user identity

Question

I can monitor Athena resource usage using different approaches:

- I can log [Athena API calls with AWS CloudTrail](https://docs.aws.amazon.com/athena/latest/ug/monitor-with-cloudtrail.html) and use `StartQueryExecution` to track the number of queries per user identity
- I can monitor [Athena queries with CloudWatch metrics](https://docs.aws.amazon.com/athena/latest/ug/query-metrics-viewing.html) to track number of queries, execution time, data scanned etc... per workgroup

How can I easily monitor Athena resource usage per user identity, for example data scanned and execution time? Should I use the Athena API [`GetQueryExecution`](https://docs.aws.amazon.com/athena/latest/APIReference/API_GetQueryExecution.html) ? Is there a way to easily add/query GetQueryExecution from CloudWatch Logs? Can you post an example data log insight query to join the `StartQueryExecution` with `GetQueryExecution`?  I can't use workgroup per user because of the 1000 workgroup limit per account. We could perhaps use workgroup per teams but it would need a lot of implementation because people can work on projects that cross team boundaries.

Giovanni Lauria · Answer

You can monitor Athena resource usage per user identity using AWS CloudTrail with StartQueryExecution and the Athena API GetQueryExecution. CloudWatch Logs Insights can be used to analyze your logs. Despite the limit, workgroups can still be used to separate users, teams, applications, or different workloads.

Soumaya · Answer

Hello Giovanni
Thanks for getting back to me. Unfortunately we have about 800 users and this is expected to grow so we can’t use workgroups which has a 1000 hard limit. With regards to using  AWS CloudTrail with StartQueryExecution and the Athena API GetQueryExecution, can you please provide more details? As I explained in my question I am familiar with Athena API GetQueryExecution, but would like some advice on how to easily query that info from cloudtrail/cloudwatch. I have seen https://aws.amazon.com/blogs/big-data/auditing-inspecting-and-visualizing-amazon-athena-usage-and-cost/ but it seems overly complicated and there would be additional costs incurred from the lambda function.

Regards
Soumaya

How to monitor athena resource usage per user identity

Add your answer

Relevant content