Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

MWAA psycopg2 SSL Error After Upgrading Environment from Small to Medium

0

Hello AWS Community,

I'm experiencing an SSL connection issue with my Amazon Managed Workflows for Apache Airflow (MWAA) environment after upgrading the environment class. Looking for guidance on resolving this error.

Issue Description: After upgrading my MWAA environment from small to medium class, one of my DAGs is failing with the following error:

psycopg2.OperationalError: SSL error: decryption failed or bad record mac

Environment Details:

Service: Amazon MWAA (Managed Workflows for Apache Airflow) Environment Class: Upgraded from Small to Medium Error Type: psycopg2 SSL connection error Affected: One specific DAG (others working normally) Steps to Reproduce:

Upgraded MWAA environment from small to medium environment class Attempted to run the previously working DAG DAG now fails with the SSL error during database operations What I've Tried:

Verified the DAG worked correctly before the upgrade Checked that other DAGs in the same environment are functioning normally Reviewed CloudWatch logs for additional error context Questions:

Is this a known issue when upgrading MWAA environment classes? Could the upgrade have affected SSL configurations between workers and the metadata database? Are there specific worker configuration adjustments needed for medium environments? Should I review my requirements.txt file for psycopg2 version compatibility? Are there any network connectivity settings that might need adjustment after the upgrade? Additional Context: The error appears to be related to SSL communication between Airflow workers and the metadata database. The timing coincides exactly with the environment class upgrade, suggesting a configuration or compatibility issue introduced during the upgrade process.

Note: I think it has something to do to the DAG being running while we had an upgrade running. I can't access XCOM of this dag via UI, but it works for others. Seems like a specific issue, though I can't remove XCOMs of this corrupted DAG. Are my only option is to rename the DAG and hope for the best or it's something on the infrastructure side of MWAA?

Any insights, troubleshooting steps, or similar experiences would be greatly appreciated!

Topics
Application Integration
Tags
Amazon Managed Workflows for Apache Airflow (MWAA)
Language
English

asked a year ago299 views

1 Answer
-1

This error occurs when the worker overloads or loses connection to the metadata database, or when a task tries to reuse a closed database connection. The "psycopg2.OperationalError: SSL connection has been closed unexpectedly" error is a common issue in Amazon MWAA environments.

When you upgraded from a small to medium environment class, it's possible that the change in resources affected how your DAG interacts with the metadata database. Here are some troubleshooting steps to resolve this issue:

  1. Reduce CPU and memory utilization:
  • Spread out task schedules
  • Narrow imports in your DAG
  • Use external services for heavy processing
  • Reduce data passed between tasks
  1. Customize Apache Airflow configurations:
  • Adjust celery.worker_autoscale
  • Set core.execute_tasks_new_python_interpreter to True
  • Set worker, scheduler, and web server logs to at least WARNING level
  1. Optimize your DAG:
  • Use deferrable operators for tasks that submit jobs to external systems
  • Remove top-level code that uses unnecessary resources
  • Don't use Apache Airflow variables in top-level Python code
  • Simplify the DAG structure
  • Create an .airflowignore file to disregard non-DAG files during parsing

  1. If you're using Apache Airflow v1.x, consider upgrading to Apache Airflow v2.0.2 or later, which allows you to specify up to 5 Schedulers, providing better resource management.

  2. For PostgreSQL-specific issues, ensure you have the proper PostgreSQL dependencies in your requirements.txt file:

apache-airflow[postgres]==<your-airflow-version>
  1. Test your DAGs, custom plugins, and Python dependencies locally using the aws-mwaa-local-runner tool to identify any compatibility issues.

The error is likely not directly caused by the environment class change itself, but rather by how your DAG's resource usage interacts with the new environment configuration. By optimizing your DAG and adjusting configurations, you should be able to resolve the SSL connection issue.
Sources
Troubleshoot error in Amazon MWAA | AWS re:Post
Troubleshooting: CloudWatch Logs and CloudTrail errors - Amazon Managed Workflows for Apache Airflow
Troubleshooting: DAGs, Operators, Connections, and other issues in Apache Airflow v1 - Amazon Managed Workflows for Apache Airflow

answered a year ago

  • Tomas Peluritis
    a year ago

    Dag's are developed according to best practices (At least o my knowledge). celery_worker_autoscale was adjusted to "8,4".

    what does "core.execute_tasks_new_python_interpreter" do?

    Main question still stands, why it worked OK on small environment, but doesn't on medium, I do feel there are some changes in SSL after upgrade.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content