Job ID value not pushed into xcom

As you have mentioned that the EmrServerlessStartJobOperator was pushing the job Id in XCOM, with which the job submits to EMR, till yesterday but after that you're not seeing any return value from this Operator, can be due to an issue with the job submission or the XCom push operation.

Here are a few things you can try to troubleshoot the issue:

• Check the logs: Look for any error messages or exceptions in the Airflow task logs for the EmrServerlessStartJobOperator. This might provide insights into what's causing the issue.

• Verify the EMR Serverless configuration: Double-check your EMR Serverless configuration, such as the Amazon Resource Name (ARN) of the execution role, the application configuration, and other required parameters. Ensure that the configuration is correct and that the Airflow user has the necessary permissions to submit jobs to EMR Serverless.

• Check network connectivity: Ensure that the Airflow worker has network connectivity to the Amazon EMR Serverless service. If there are any network issues or proxy configurations, it might prevent the job submission or XCom push operation from working correctly.

• Try a simpler example: Create a simple DAG with just the EmrServerlessStartJobOperator and a dummy job configuration. This will help you isolate the issue and determine if it's related to your specific job configuration or if it's a more general problem with the operator.

For specific troubleshooting, you can create a case with the AWS Technical Support and provide the respective logs.