I created an AWS Glue extract, transform, and load (ETL) job to send messages to an Amazon Simple Queue Service (Amazon SQS) queue. But the Amazon SQS queue is in another AWS account that's in a different AWS Region. When I run the job, I get the error "The specified queue does not exist or you do not have access to it."
Short description
If the Amazon SQS queue is in a different AWS Region than the AWS Glue job, then messages to the queue must contain Region information. If you don't pass Region information in your message, then your ETL job fails with the following error:
ERROR [main] glue.ProcessLauncher (Logging.scala:logError(70)): Exception in User Class: com.amazonaws.services.sqs.model.QueueDoesNotExistException :The specified queue does not exist or you do not have access to it. (Service: AmazonSQS; Status Code: 400; Error Code: AWS.SimpleQueueService.NonExistentQueue; Request ID: 3861e4c0-9b49-5404-a4c6-bcd3ed43fe20)
Resolution
Suppose you have an AWS account called Account A in the AWS Region us-west-2, and another account called Account B in Region us-east-1. To create an AWS Glue Spark job in Account A that sends messages to Amazon SQS in Account B, complete the following steps:
- Create an Amazon SQS queue in Account B with the following access policy. This access policy provides access to the AWS Identity and Access Management (IAM) role that Account A attaches to the AWS Glue Spark job. You can also grant these required permissions to a specific IAM user (for example, testuser) in Account A. For more information, see Basic examples of Amazon SQS policies.
{
"Version": "2008-10-17",
"Id": "__default_policy_ID",
"Statement": [
{
"Sid": "__owner_statement",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::111122223333444:role/GlueSparkJobIAMRole",
"arn:aws:iam::111122223333444:user/testuser"
]
},
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:us-east-1:5555666677778888:test-queue"
}
]
}
Note: Replace the following in the preceding policy:
111122223333444 with the AWS account ID for Account A
5555666677778888 with the AWS account ID for Account B
testuser with the name of the IAM user in Account A
GlueSparkJobIAMRole with the IAM role that's attached to the AWS Glue Spark job in Account A
test-queue with the name of the queue that Account B created
- Create an AWS Glue ETL job in Account A. On the Configure the job properties page, select A new script to be authored by you. For more information, see Configuring job properties for Spark jobs in AWS Glue.
To send a message to the Amazon SQS queue in Account B, include the following Python script in the job:
import boto3
sqs = boto3.client('sqs', region_name="us-east-1")
queue_url = 'https://sqs.us-east-1.amazonaws.com/5555666677778888/glue-queue'
response = sqs.send_message(
QueueUrl=queue_url,
DelaySeconds=10,
MessageAttributes={
'Title': {
'DataType': 'String',
'StringValue': 'The Whistler'
},
'Author': {
'DataType': 'String',
'StringValue': 'John Doe'
},
'WeeksOn': {
'DataType': 'Number',
'StringValue': '6'
}
},
MessageBody=('Example message'))
print(response['MessageId'])
Note: Replace the following in the preceding script:
5555666677778888 with the AWS account ID of Account B
glue-queue with the name of the Amazon SQS queue
Example message with the message that the AWS Glue job will send to the Amazon SQS queue
The MessageAttributes keys and values with your own keys and values
- Identify the AWS IAM role attached to the AWS Glue Spark job in Account A. Then, grant Amazon SQS required permissions to that role. For example, attach the AWS managed policy AmazonSQSFullAccess to this AWS IAM role. For more information, see Setting up IAM permissions for AWS Glue.
- Run the AWS Glue ETL job created in Account A.
- To verify that the job completed, checking that the job sent the message to the Amazon SQS queue in Account B.
- To receive the message in the Amazon SQS queue in Account B, poll for the message in the queue. For more information, see Receiving and deleting a message in Amazon SQS.
- Verify that you can view the messages sent from Account A in the queue.
Related information
Managing an Amazon SQS queue