Redirecting MWAA metrics to external tools and bypassing CloudWatch

Question

We are experimenting a significant increase in CloudWatch metrics costs since deploying Amazon MWAA. We noticed that AWS MWAA utilizes CloudWatch as the default metrics destination via Statsd.

We made an attempt to modify the Airflow metrics configuration[1] by utilizing Apache Airflow Configuration options on MWAA[2]. Unfortunately, that didn't work. Upon MWAA startup, a log message appears indicating `statsd_host: None` and the metrics continue to be sent to CloudWatch

Is there a method to reroute the MWAA metrics to an external monitoring tool, such as Prometheus or Datadog, while bypassing CloudWatch as an intermediary?

For reference:
1- https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html
2 - https://docs.aws.amazon.com/mwaa/latest/userguide/configuring-env-variables.html?icmpid=docs_console_unmapped

Best regards

Answer

Hi,
Amazon Managed Workflows for Apache Airflow [monitoring overview](https://docs.aws.amazon.com/mwaa/latest/userguide/monitoring-overview.html) states that Apache Airflow is already set-up to send StatsD metrics for an Amazon Managed Workflows for Apache Airflow environment to Amazon CloudWatch.
If you are getting more metrics than you need, you may want to [choose the metrics that Airflow reports](https://docs.aws.amazon.com/mwaa/latest/userguide/access-metrics-cw-202.html#choosing-metrics). Likewise, if you are using Logs, you can [control the level of logs you want to report](https://docs.aws.amazon.com/mwaa/latest/userguide/monitoring-airflow.html#monitoring-airflow-enable).

Redirecting MWAA metrics to external tools and bypassing CloudWatch

相關內容