AWS Glue Job Error - Exception in User Class

Question

I'm trying to run a simple AWS Glue job to convert a file from CSV to Parquet. Does anyone know what this error means?  
  
2021-08-11 05:40:28,288 ERROR glue.ProcessLauncher (Logging.scala:logError(91)): Exception in User Class  
java.lang.reflect.UndeclaredThrowableException  
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1862)  
    at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:64)  
    at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:188)  
    at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:281)  
    at org.apache.spark.executor.CoarseGrainedExecutorBackendPlugin$class.launch(CoarseGrainedExecutorBackendWrapper.scala:10)  
    at org.apache.spark.executor.CoarseGrainedExecutorBackendWrapper$$anon$1.launch(CoarseGrainedExecutorBackendWrapper.scala:15)  
    at org.apache.spark.executor.CoarseGrainedExecutorBackendWrapper.launch(CoarseGrainedExecutorBackendWrapper.scala:19)  
    at org.apache.spark.executor.CoarseGrainedExecutorBackendWrapper$.main(CoarseGrainedExecutorBackendWrapper.scala:5)  
    at org.apache.spark.executor.CoarseGrainedExecutorBackendWrapper.main(CoarseGrainedExecutorBackendWrapper.scala)  
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
    at java.lang.reflect.Method.invoke(Method.java:498)  
    at com.amazonaws.services.glue.SparkProcessLauncherPlugin$class.invoke(ProcessLauncher.scala:47)  
    at com.amazonaws.services.glue.ProcessLauncher$$anon$1.invoke(ProcessLauncher.scala:75)  
    at com.amazonaws.services.glue.ProcessLauncher.launch(ProcessLauncher.scala:123)  
    at com.amazonaws.services.glue.ProcessLauncher$.main(ProcessLauncher.scala:29)  
    at com.amazonaws.services.glue.ProcessLauncher.main(ProcessLauncher.scala)  
Caused by: org.apache.spark.SparkException: Exception thrown in awaitResult:   
    at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:226)  
    at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)  
    at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)  
    at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:201)  
    at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:65)  
    at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:64)  
    at java.security.AccessController.doPrivileged(Native Method)  
    at javax.security.auth.Subject.doAs(Subject.java:422)  
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)  
    ... 17 more  
Caused by: java.io.IOException: Failed to connect to /172.36.215.7:35553  
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:245)  
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:187)  
    at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:198)  
    at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:194)  
    at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190)  
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)  
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  
    at java.lang.Thread.run(Thread.java:748)  
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.36.215.7:35553  
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)  
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323)  
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)  
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)  
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)  
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)  
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)  
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)  
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)  
    ... 1 more  
Caused by: java.net.ConnectException: Connection refused  
    ... 11 more  
  
Edited by: jinsanz on Aug 11, 2021 8:54 AM  
  
Edited by: jinsanz on Aug 11, 2021 8:55 AM  
  
Edited by: jinsanz on Aug 11, 2021 8:56 AM

Answer

I'm having the same issue... The thing is that it worked before. I made a few changes to my CF and after that it started failing

Answer

Right now I have same problems as your. So please tell me if you passed it and how do you do? Thank you.

Answer

I had this exact problem/error.  
  
I solved the problem by upgrading the job to use Glue 3.0 with Spark 3.1.  Glue version 2.0 with Spark version 2.4 gets this error.  I am using Python as the ETL language.  My connection is JDBC to a SQL Server database.  The Crawler works just fine so the connection and database and tables are in good order.  
  
Add a little more detail for @enricu :  
I have the older configuration running in production but have not touched it.  It still runs ok.  I was replicating that job in a lower environment to test changes to the ETL.   So if you need to make changes to an existing implementation you will probably need to upgrade.  This may or  may not be impactful depending on your implementation.  
  
Job that errors:  
Name	  
IAM role	  
Type	Spark  
Glue version	2.0  
Python version	3  
Spark version	2.4  
ETL language	python  
  
New Job that does not experience that error:  
Name	  
IAM role	  
Type	Spark  
Glue version	3.0  
Python version	3  
Spark version	3.1  
ETL language	python  
  
Edited by: mystic on Sep 22, 2021 4:40 PM

Answer

After going through all of the logs ("All logs" link in the runs list), I've found the real cause of the issue, which was a typo in SQL statement. Try to go through all of the logs, so you might find more details about this issue

AWS Glue Job Error - Exception in User Class

相关内容