spark env issues post custom ami creation for EMR on EC2 cluster stuck in steps failure

The issue you're experiencing is likely related to the custom AMI creation process and its impact on the Spark environment configuration. When you create a custom AMI for EMR, especially when upgrading components like Python, it's important to ensure that all necessary environment variables and configurations are properly set.

The error message suggests that the EMR File System (EMRFS) class is not being found when submitting Spark jobs through the step function. This is likely because the custom AMI creation process has altered the default environment setup.

To resolve this issue, you have a few options:

1. Bootstrap Action: You can create a bootstrap action script that sources the Spark environment file before running any Spark jobs. Add this bootstrap action when creating your EMR cluster:

   #!/bin/bash
   sudo tee -a /etc/spark/conf/spark-env.sh << EOL
   source /etc/spark/conf/spark-env.sh
   EOL
   

2. Custom Step Script: Instead of directly using spark-submit in your step function, create a wrapper script that sources the Spark environment file before running the Spark job. Your step would then call this wrapper script instead.

3. Modify EMR Configuration: You can try to set the Spark classpath explicitly in the EMR configuration when creating the cluster. Add a configuration like this:

   [
     {
       "Classification": "spark-env",
       "Properties": {},
       "Configurations": [
         {
           "Classification": "export",
           "Properties": {
             "SPARK_CLASSPATH": "/usr/lib/spark/jars/*:/usr/share/aws/emr/emrfs/lib/*"
           }
         }
       ]
     }
   ]
   

4. Review Custom AMI Process: Ensure that your custom AMI creation process is not inadvertently removing or modifying critical EMR configurations. You may need to manually copy over some configuration files from the base EMR AMI to your custom AMI.

Remember, when creating a custom AMI for EMR, it's crucial to maintain compatibility with EMR's expected environment. Always test thoroughly after making changes to ensure all components work as expected.
Sources
Manual submitted step jobs failing | AWS re:Post
Spark application specifics with earlier AMI versions of Amazon EMR - Amazon EMR
EMR Cluster failure with "Failed to start the job flow due to an internal error" | AWS re:Post
ClassNotFoundException EmrFileSystem error while setting Spark --driver-class-path config in EMR Serverless | AWS re:Post