Configure MWAA to use json logging

Question

It seems like it is possible in Apache Airflow to override loggers to use specific json logging.

I've been noticing it is very difficult to trace logs in cloud watch for airflow since they are poorly formatted from all sources (scheduler, processor, task instance, etc).

I'd like to configure MWAA to use a json logger instead that will output all log info into a single json-formatted string, however, I did not find anything in documentation for this.

I believe I can add logging handler within DAGs and Operators, however, it will most likely not apply to the global logger in the scheduler and other components.

Is there any way to configure such logging?

Answer

Hello,
It is the default behaviour of MWAA wherein it sends the logs to CloudWatch in the said format. As mentioned by you, it is possible to make use of the **logging** python package within your DAGs and operators individually to edit the format of your logs. Please note that MWAA being a managed service, it is currently not possible to edit the logs format on the global level. I apologise for the inconvenience that this may cause to you. If you would like to get more details about this with respect to your use case then please feel free to reach out to AWS Premium Support team. They would be more than happy to help you out.

For more details about logging in MWAA, please refer [this](https://docs.aws.amazon.com/mwaa/latest/userguide/monitoring-airflow.html)

Configure MWAA to use json logging

Relevant content