We have a pyspark job which we are executing to connect with MongoDB using the mongo-spark-connector. The job is executed successfully with no errors in the stdout log file and in the stderr log file we get following error:

INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(1, [2600:1f13:e3d:5e02:5016:e800:a87c:5cd4], 42357, None)
INFO BlockManagerMaster: Removed 1 successfully in removeExecutor
INFO DAGScheduler: Shuffle files lost for executor: 1 (epoch 1)
WARN DefaultEmrServerlessRMClient: Encountered errors when releasing containers: [{ContainerGroupId: 8cc55db8-8f45-6671-d09e-dd1e0088906d,ContainerId: 04c55db8-8fbb-7666-3f7e-626522a44608,ErrorCode: INTERNAL_ERROR}]
WARN DefaultEmrServerlessRMClient: Encountered errors when releasing containers: [{ContainerGroupId: 8cc55db8-8f45-6671-d09e-dd1e0088906d,ContainerId: 20c55db8-8fab-0c85-b530-2f5b44dc48cd,ErrorCode: INTERNAL_ERROR}, {ContainerGroupId: 8cc55db8-8f45-6671-d09e-dd1e0088906d,ContainerId: 04c55db8-8fbb-7666-3f7e-626522a44608,ErrorCode: INTERNAL_ERROR}]
WARN DefaultEmrServerlessRMClient: Encountered errors when releasing containers: [{ContainerGroupId: 8cc55db8-8f45-6671-d09e-dd1e0088906d,ContainerId: 20c55db8-8fab-0c85-b530-2f5b44dc48cd,ErrorCode: INTERNAL_ERROR}, {ContainerGroupId: 8cc55db8-8f45-6671-d09e-dd1e0088906d,ContainerId: 04c55db8-8fbb-7666-3f7e-626522a44608,ErrorCode: INTERNAL_ERROR}]

Due to this the pyspark job is not getting stopped and is running indefinitely.

pyspark code snippet that we use for MongoDB connection which is getting executed successfully but the pyspark job is not terminating after execution :

spark = SparkSession.builder.appName(
            "test-app"
        ).config(
            "spark.jars.packages", "org.mongodb.spark:mongo-spark-connector:10.0.2"
        ).config(
            "spark.mongodb.read.connection.uri", mongo_url
        ).config(
            "spark.mongodb.write.connection.uri", mongo_url
        ).getOrCreate()

dataDF = spark.read.format("mongodb").option(
        "uri", mongo_url
    ).option(
        "database", database
    ).option(
        "collection", readFromCollection
    ).load()
dataDF.printSchema()
dataDF.show()
print(dataDF.count())

Can anyone please help me to understand why we are getting this error and any ways to avoid it. Also is there any way we can forcefully exit from a pyspark job in emr serverless. We already have tried with following options but none have worked so far:

• spark.stop()
• os._exit(0)