MWAA Metrics to OpenTelemetry Endpoint

0

Airflow supports sending metrics to an OpenTelemetry endpoint as described here

I tried configuring my MWAA airflow instance with the following environment params. but it doesn't seem to work:

metrics.otel_on = True
metrics.otel_host = <my otel endpoint>
metrics.otel_port = 4318
metrics.otel_prefix = airflow
metrics.otel_ssl_active = False

I assume this is because MWAA supports statsd by default, but even if I use metrics.statsd_on = False, it looks like OTel metrics still doesn't work. The same config works on a K8s hosted Airflow installation so this issue is specific to MWAA.

Since I also have the OTel Tracing integration for Airflow enabled, I know that the endpoint configuration is fine and the issue is only with metrics workflow. How do I go about enabling MWAA metrics to OTel integration instead of Cloudwatch?

Thanks

1 Answer
-1

I apologize, but based on the information provided in the available sources, I cannot confirm that Amazon Managed Workflows for Apache Airflow (MWAA) directly supports sending metrics to an OpenTelemetry endpoint as you've described.

By default, MWAA publishes Airflow metrics to Amazon CloudWatch Metrics. The configuration options you've tried to set for OpenTelemetry metrics integration are not mentioned in the MWAA documentation provided.

MWAA does allow you to change some Apache Airflow configuration options directly on the Amazon MWAA console. However, these changes are primarily focused on overriding default Airflow settings and attaching them as environment variables to the AWS Fargate container for your environment.

If you need to integrate MWAA metrics with OpenTelemetry, you might need to explore alternative approaches. One possibility could be to use the metrics published to CloudWatch as a starting point and then use AWS services or custom solutions to forward these metrics to your OpenTelemetry endpoint.

For the most up-to-date and accurate information on MWAA's capabilities and supported integrations, I recommend reaching out to AWS support or consulting the latest MWAA documentation. They would be best positioned to advise on whether there's a supported way to send MWAA metrics directly to an OpenTelemetry endpoint or if there are plans to support this in the future.
Sources
Using Apache Airflow configuration options on Amazon MWAA - Amazon Managed Workflows for Apache Airflow
Amazon Managed Workflows for Apache Airflow (MWAA) FAQs – Amazon Web Services

profile picture
answered 7 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions