Questions tagged with AWS Glue

Content language: English

Sort by most recent

Browse through the questions and answers listed below or filter and sort to narrow down your results.

Glue Job started failing after working for weeks with Py4JJavaError: An error occurred while calling o111.pyWriteDynamicFrame. java.lang.reflect.InvocationTargetException

Hi, I am a security architect not a programmer so I am the blind squirrel looking for a nut in this situation but could use some help. We have a glue job failing and the error bounces around from Py4JJavaError: An error occurred while calling o117.pyWriteDynamicFrame. java.lang.reflect.InvocationTargetException and JavaError: An error occurred while calling o116.pyWriteDynamicFrame. the full error I believe would be the below... Any help would be awesome and greatly appreciated "Event": "GlueETLJobExceptionEvent", "Timestamp": 1666211480954, "Failure Reason": "Traceback (most recent call last):\n File \"/tmp/governed_tables.py\", line 68, in <module>\n sink.writeFrame(dy_df)\n File \"/opt/amazon/lib/python3.6/site-packages/awsglue/data_sink.py\", line 32, in writeFrame\n return DynamicFrame(self._jsink.pyWriteDynamicFrame(dynamic_frame._jdf, callsite(), info), dynamic_frame.glue_ctx, dynamic_frame.name + \"_errors\")\n File \"/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py\", line 1305, in __call__\n answer, self.gateway_client, self.target_id, self.name)\n File \"/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py\", line 111, in deco\n return f(*a, **kw)\n File \"/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py\", line 328, in get_return_value\n format(target_id, \".\", name), value)\npy4j.protocol.Py4JJavaError: An error occurred while calling o117.pyWriteDynamicFrame.\n: java.lang.reflect.InvocationTargetException\n\tat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n\tat sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n\tat sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n\tat java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n\tat com.amazonaws.services.glue.remote.MichiganAWSCredentialProviderProxy$.get(MichiganAWSCredentialProviderProxy.scala:14)\n\tat com.amazonaws.services.glue.util.LakeformationClientWrapper.$anonfun$lakeformationUpdateTableObjectsInternal$1(LakeformationGovernedWrapper.scala:126)\n\tat scala.collection.immutable.List.map(List.scala:282)\n\tat com.amazonaws.services.glue.util.LakeformationClientWrapper.lakeformationUpdateTableObjectsInternal(LakeformationGovernedWrapper.scala:115)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native .Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat com.amazonaws.services.glue.GlueUtility$.callLakeformationMethod(GlueUtility.scala:44)\n\tat com.amazonaws.services.glue.sinks.HadoopDataSink.lakeformationUpdateTableObjects(HadoopDataSink.scala:344)\n\tat com.amazonaws.services.glue.sinks.HadoopDataSink.$anonfun$writeDynamicFrame$2(HadoopDataSink.scala:313)\n\tat com.amazonaws.services.glue.util.FileSchemeWrapper.$anonfun$executeWithQualifiedScheme$1(FileSchemeWrapper.scala:90)\n\tat com.amazonaws.services.glue.util.FileSchemeWrapper.executeWith(FileSchemeWrapper.scala:83)\n\tat com.amazonaws.services.glue.util.FileSchemeWrapper.executeWithQualifiedScheme(FileSchemeWrapper.scala:90)\n\tat com.amazonaws.services.glue.sinks.HadoopDataSink.$anonfun$writeDynamicFrame$1(HadoopDataSink.scala:158)\n\tat org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:253)\n\tat com.amazonaws.services.glue.sinks.HadoopDataSink.writeDynamicFrame(HadoopDataSink.scala:152)\n\tat com.amazonaws.services.glue.DataSink.pyWriteDynamicFrame(DataSink.scala:72)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)\n\tat py4j.Gateway.invoke(Gateway.java:282)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:238)\n\tat java.lang.Thread.run(Thread.java:750)\nCaused by: java.lang.RuntimeException: class com.amazonaws.services.gluejobexecutor.model.EntityNotFoundException:Database null not found. (Service: AWSLakeFormation; Status Code: 400; Error Code: EntityNotFoundException; Request ID: 37ecd5ab-d47a-4c25-ad7a-80a070ed1558; Proxy: null) (Service: AWSGlueJobExecutor; Status Code: 400; Error Code: EntityNotFoundException; Request ID: b3862a0e-3bb4-440e-9801-452716b82337; Proxy: null)\n\tat com.amazonaws.services.glue.remote.LakeformationCredentialsProvider.refresh(LakeformationCredentialsProvider.scala:50)\n\tat com.amazonaws.services.glue.remote.LakeformationCredentialsProvider.<init>(LakeformationCredentialsProvider.scala:77)\n\t... 33 more\n", "Stack Trace": [ { "Declaring Class": "get_return_value", "Method Name": "format(target_id, \".\", name), value)", "File Name": "/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", "Line Number": 328 }, { "Declaring Class": "deco", "Method Name": "return f(*a, **kw)", "File Name": "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", "Line Number": 111 }, { "Declaring Class": "__call__", "Method Name": "answer, self.gateway_client, self.target_id, self.name)", "File Name": "/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", "Line Number": 1305 }, { "Declaring Class": "writeFrame", "Method Name": "return DynamicFrame(self._jsink.pyWriteDynamicFrame(dynamic_frame._jdf, callsite(), info), dynamic_frame.glue_ctx, dynamic_frame.name + \"_errors\")", "File Name": "/opt/amazon/lib/python3.6/site-packages/awsglue/data_sink.py", "Line Number": 32 }, { "Declaring Class": "<module>", "Method Name": "sink.writeFrame(dy_df)", "File Name": "/tmp/governed_tables.py", "Line Number": 68 } ], "Last Executed Line number": 68, "script": "governed_tables.py"
1
answers
0
votes
75
views
asked a month ago