Read data from AWS Redshift using AWS Glue job

0

I set up a connection between AWS Glue and AWS Redshift, created an AWS Glue job, in the job when trying to execute a valid SQL query:

select distinct user_id from user_api.payment_made

source code of Glue job is presented below:

import sys from awsglue.transforms import * from awsglue.utils import getResolvedOptions from pyspark.context import SparkContext from awsglue.context import GlueContext from awsglue.job import Job from awsglue import DynamicFrame

args = getResolvedOptions(sys.argv, ["JOB_NAME"]) sc = SparkContext() glueContext = GlueContext(sc) spark = glueContext.spark_session job = Job(glueContext) job.init(args["JOB_NAME"], args)

AmazonRedshift_node1688056230525 = glueContext.create_dynamic_frame.from_options( connection_type="redshift", connection_options={ "sampleQuery": "select distinct user_id from user_api.payment_made", "redshiftTmpDir": "s3://aws-glue-assets-840773940442-us-east-1/temporary/", "useConnectionProperties": "true", "aws_iam_role": "arn:aws:iam::840773940442:role/service-role/AWSGlueServiceRole-RDSExportS3", "connectionName": "aws_glue_to_aws_redshift_connection", }, transformation_ctx="AmazonRedshift_node1688056230525", )

AmazonRedshift_node1688056290381 = glueContext.write_dynamic_frame.from_options( frame=AmazonRedshift_node1688056230525, connection_type="redshift", connection_options={ "redshiftTmpDir": "s3://aws-glue-assets-840773940442-us-east-1/temporary/", "useConnectionProperties": "true", "aws_iam_role": "arn:aws:iam::840773940442:role/service-role/AWSGlueServiceRole-RDSExportS3", "dbtable": "business_intellegence_data.user_path_info", "connectionName": "aws_glue_to_aws_redshift_connection", "preactions": "CREATE TABLE IF NOT EXISTS business_intellegence_data.test_t1 (user_id VARCHAR); TRUNCATE TABLE business_intellegence_data.test_t1;", }, transformation_ctx="AmazonRedshift_node1688056290381", )

job.commit()

I get an error:

Invalid operation: exception name : UnauthorizedException, error type : 135, message: Not authorized to get credentials of role arn:aws:iam::840773940442:role/service-role/AWSGlueServiceRole-RDSExportS3

role AWSGlueServiceRole-RDSExportS3 was assigned to Redshift cluster.

Tell me, please, how can I solve this problem?

demandé il y a 10 mois590 vues
1 réponse
0

The role you mention AWSGlueServiceRole-RDSExportS3 is not AWS managed role but customer managed role instead. Double check if you are giving permission to Redshift or RDS (as the name suggests).

Refer to Configuring Redshift connections in AWS Glue which provides the necessary details in the Configuring IAM roles section.

profile pictureAWS
répondu il y a 10 mois

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions