MWAA metrics export for data analytics

Question

Apart from the Cloudwatch monitor we use tableau to report the ETLs processess stats.
So, is there a way to export the metrics (DAG start time, DAG End time, DAG task start time, endtime, status, last run etc)  from MWAA as a file or any other format?

Answer

There are methods to query the metadata in the back-end database used by Amazon MWAA.

I recommend reviewing this blog, but not utilizing the AWS CloudFormation template as it will create an Amazon MWAA Enviroment.

https://aws.amazon.com/blogs/big-data/persist-and-analyze-metadata-in-a-transient-amazon-mwaa-environment/

By utilizing AWS Glue along with the example DAGs, your metadata will existing within Amazon S3 and can then be downloaded or queried by Amazon Athena.
If you decide to utilize Amazon Athena, then you can then connect from Tableau by following this blog post example: https://aws.amazon.com/blogs/big-data/building-aws-data-lake-visualizations-with-amazon-athena-and-tableau/

MWAA metrics export for data analytics

Contenuto pertinente