Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Access Denied Error on Glue Tutorial for Flight Data

0

I'm getting an access denied error when trying to complete the flight data tutorial. Is there something specific I should be assigning to the role I'm using?

9/04/23 22:14:48 INFO client.RMProxy: Connecting to ResourceManager at ip-172-32-19-124.us-west-2.compute.internal/172.32.19.124:8032
Container: container_1556057109087_0002_01_000001 on ip-172-32-19-25.us-west-2.compute.internal_8041

LogType:stdout

Log Upload Time:Tue Apr 23 22:14:48 +0000 2019
LogLength:9590
Log Contents:
null_fields []
Traceback (most recent call last):

File "script_2019-04-23-22-14-04.py", line 40, in <module>

datasink4 = glueContext.write_dynamic_frame.from_options(frame = dropnullfields3, connection_type = "s3", connection_options =
{
"path": "s3://369016067933-us-west-2-datalakes3curated"
}
, format = "parquet", transformation_ctx = "datasink4")

File "/mnt/yarn/usercache/root/appcache/application_1556057109087_0002/container_1556057109087_0002_01_000001/PyGlue.zip/awsglue/dynamicframe.py", line 584, in from_options

File "/mnt/yarn/usercache/root/appcache/application_1556057109087_0002/container_1556057109087_0002_01_000001/PyGlue.zip/awsglue/context.py", line 191, in write_dynamic_frame_from_options

File "/mnt/yarn/usercache/root/appcache/application_1556057109087_0002/container_1556057109087_0002_01_000001/PyGlue.zip/awsglue/context.py", line 214, in write_from_options

File "/mnt/yarn/usercache/root/appcache/application_1556057109087_0002/container_1556057109087_0002_01_000001/PyGlue.zip/awsglue/data_sink.py", line 32, in write

File "/mnt/yarn/usercache/root/appcache/application_1556057109087_0002/container_1556057109087_0002_01_000001/PyGlue.zip/awsglue/data_sink.py", line 28, in writeFrame

File "/mnt/yarn/usercache/root/appcache/application_1556057109087_0002/container_1556057109087_0002_01_000001/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in call

File "/mnt/yarn/usercache/root/appcache/application_1556057109087_0002/container_1556057109087_0002_01_000001/pyspark.zip/pyspark/sql/utils.py", line 63, in deco

File "/mnt/yarn/usercache/root/appcache/application_1556057109087_0002/container_1556057109087_0002_01_000001/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value

py4j.protocol.Py4JJavaError: An error occurred while calling o206.pyWriteDynamicFrame.

: com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 981F622DF5FD582D), S3 Extended Request ID: PKaNaPlcVc2tAsVXQ8yafNC6W6EsSXMGSWcLf8R6AFDNfYHqhJ9juawK7Eyau3I4oSgV6MD8QGM=

at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)

at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)

at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)

at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)

at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)

at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)

at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)

at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)

at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)

at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4169)
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4116)
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1700)
at com.amazon.ws.emr.hadoop.fs.s3.lite.call.PutObjectCall.performCall(PutObjectCall.java:34)
at com.amazon.ws.emr.hadoop.fs.s3.lite.call.PutObjectCall.performCall(PutObjectCall.java:9)
at com.amazon.ws.emr.hadoop.fs.s3.lite.call.AbstractUploadingS3Call.perform(AbstractUploadingS3Call.java:62)
at com.amazon.ws.emr.hadoop.fs.s3.lite.executor.GlobalS3Executor.execute(GlobalS3Executor.java:80)
at com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.invoke(AmazonS3LiteClient.java:176)
at com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.putObject(AmazonS3LiteClient.java:104)
at com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.storeEmptyFile(Jets3tNativeFileSystemStore.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy56.storeEmptyFile(Unknown Source)
at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.mkdir(S3NativeFileSystem.java:1198)
at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.mkdirs(S3NativeFileSystem.java:1180)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1929)
at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.mkdirs(EmrFileSystem.java:397)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.setupJob(FileOutputCommitter.java:313)
at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.setupJob(HadoopMapReduceCommitProtocol.scala:138)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply$mcV$sp(FileFormatWriter.scala:169)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:166)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:166)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:65)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:166)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:145)

at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)
at org.apache.spark.sql.execution.datasources.DataSource.writeInFileFormat(DataSource.scala:435)

at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:471)

at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:50)

at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:609)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:233)
at com.amazonaws.services.glue.SparkSQLDataSink.writeDynamicFrame(DataSink.scala:123)
at com.amazonaws.services.glue.DataSink.pyWriteDynamicFrame(DataSink.scala:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:748)

End of LogType:stdout
Container: container_1556057109087_0002_01_000002 on ip-172-32-20-12.us-west-2.compute.internal_8041
LogType:stdout
Log Upload Time:Tue Apr 23 22:14:48 +0000 2019
LogLength:0
Log Contents:
End of LogType:stdout

Topics
Analytics
Tags
AWS Glue
Language
English

asked 7 years ago1.2K views

4 Answers
0

Hi ngalloway,

I know it's been a while, but have you since been able to resolve the issue? I am getting the same error message and have absolutely no clue on how to resolve the issue.
I already made sure to use a IAM role with sufficient permissions, but still no luck...

Looking forward to your reply, or anybody else's :)

Best regards
thsc0036

answered 7 years ago

0

Sorry for inconvenience, but could you elaborate more about this?
Are you trying this tutorial? https://aws.amazon.com/blogs/aws/launch-aws-glue-now-generally-available/

AWS

answered 7 years ago

0

I'm not exactly doing the Glue Tutorial for Flight Data, but something similar - creatign a Glue job that points to Data Catalog table as source and JDBC connection to Redshift as target, using proposed script generated by AWS Glue with IAM Role that includes S3FullAccess permission. I'm getting this error:

py4j.protocol.Py4JJavaError: An error occurred while calling o140.pyWriteDynamicFrame.
: java.io.IOException: com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: A802C91454CDC80D; S3 Extended Request ID: 4kEjfvhVmYiJCs_QMQQAAsm_gwurWgxbTuHkBdA6DpO6_HeiVuD_FE7n2OWr1_zy4A7JTpOJV24=), S3 Extended Request ID: 4kEjfvhVmYiJCs_QMQQAAsm_gwurWgxbTuHkBdA6DpO6_HeiVuD_FE7n2OWr1_zy4A7JTpOJV24=
at com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.list(Jets3tNativeFileSystemStore.java:286)
at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.getFileStatus(S3NativeFileSystem.java:781)
........

It looks like the under the hood EMR resource is having problem with S3 access. The same error shows up in EMR forum a lot too, and there seems to be workaround there. However since Glue is serverless the workarounds can't be applied here. Please help!

answered 7 years ago

0

There are some possibilities that might cause access denied even if you are using IAM role with S3 full access.

  • Bucket owner: Is the target bucket located in your account?
    • If it is in other account, you will make the bucket writable from your account/role.
  • Bucket policy: Are there any bucket policy which might block writes from Glue job?
    • If it has bucket policy that blocks writes into specific paths, writes from specific source, etc., then you might need to modify the policy.
  • VPC endpoint: Are you using Glue Connections? If yes, are there any VPC S3 endpoint with specific policy in the middle of network route from your Glue job to S3?
  • Lake Formation permission: Did you enable Lake Formation permission and register this target bucket as a data location?
    • You need to ensure that you have enough Lake Formation permission.

I guess it was due to lack of Lake Formation permission.
It would be good for you to go through Lake Formation permissions and grant required permissions to the role.
If you are not using Lake Formation now and you do not have a plan to use Lake Formation permission in near future, it might be okay for you to use a tool to revert permissions. https://github.com/aws-samples/aws-glue-samples/tree/master/utilities/use_only_IAM_access_controls

Since it is a kind of resource-specific troubleshooting, I would recommend you to contact AWS premium support if the issue still remains and you need further help.
https://aws.amazon.com/premiumsupport/

AWS

answered 7 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content