How do I integrate an API Gateway REST API with Amazon SQS and resolve common errors?

9 minute read
0

I want to integrate an Amazon API Gateway REST API with Amazon Simple Queue Service (Amazon SQS), and troubleshoot integration errors.

Resolution

To integrate an API Gateway REST API with Amazon SQS, choose one of the following methods:

Use the AWS query protocol

To integrate an API Gateway REST API with Amazon SQS, you can use the AWS query protocol. To do so, complete the following steps:

  1. Create an SQS queue.
  2. Create an AWS Identity and Access Management (IAM) role, and then attach an Amazon SQS policy with a SendMessage permission. The policy allows you to publish messages from the API to Amazon SQS:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Resource": [
        "arn:aws:sqs:example-region:example-account-id:example-sqs-queue-name"
      ],
      "Action": [
        "sqs:SendMessage"
      ]
    }
  ]
}

Note: Replace example-region with your AWS Region, example-account-id with your AWS account ID, and example-sqs-queue-name with your SQS queue name.

  1. Create a REST API in API Gateway.
  2. In the API Gateway console, create an Amazon SQS integration for your REST API.
  3. Create a REST API resource or a REST API method:
    On the Resources screen, choose Create method.
    For Method type, choose POST.
    For Integration type, choose AWS Service.
    For AWS Region, choose your Region.
    For AWS Service, choose Simple Queue Service (SQS).
    (Optional) For AWS Subdomain, enter the subdomain that the AWS service uses. Check the AWS service documentation to confirm the availability of a subdomain. For the Amazon SQS example setup, keep this blank.
    For HTTP method, choose POST.
    For Action Type, choose Use path override.
    For Path override (optional), enter your account ID and SQS queue name in the following format: example-account-id/example-sqs-queue-name. For example: 1234567890/MySQSStandardQueue.
    For Execution role, enter the ARN of the IAM role.
    For Default Timeout, choose an option for your setup.
    Continue to enter your REST API integration information.
    Choose the POST method Integration Request.
    Choose Edit.
    For Request body passthrough, select the option for your requirements.
    Expand URL request headers parameters.
    Choose Add request header parameter.
    For Name, enter Content-Type.
    For Mapped from, enter 'application/x-www-form-urlencoded'.
    Expand Mapping Templates.
    Choose Add mapping template.
    For Content-Type, enter application/json.
    For the template, enter Action=SendMessage&MessageBody=$input.body, and then choose Save.
  4. Deploy the REST API.
  5. To test the setup, send the following request to API Gateway:
curl --location --request POST 'https://example-api-id.execute-api.example-region.amazonaws.com/example-stage/example-resource' \  
   --header 'Content-Type: application/json' \  
   --data-raw '{  
    "message": "Hello World"  
  }'

Note: Replace example-api-id with your API ID, example-region with your Region, example-stage with your testing stage name, and example-resource with your resource name.

When your integration is successful, your response looks similar to the following:

{  
  "SendMessageResponse": {  
    "ResponseMetadata": {  
      "RequestId": "f879fb11-e736-52c0-bd29-a0f2d09ad90d"  
    },  
      "SendMessageResult": {  
        "MD5OfMessageAttributes": null,  
        "MD5OfMessageBody": "3fc759ac1733366f98ec4270c788fcd1",  
        "MD5OfMessageSystemAttributes": null,  
        "MessageId": "4c360c3c-08f4-4392-bc14-8b0c88e314a2",  
        "SequenceNumber": null  
    }  
  }  
}

Use the AWS JSON protocol

To integrate an API Gateway REST API with Amazon SQS, you can use the AWS JSON protocol. To do so, complete the following steps:

  1. Create an SQS queue.
  2. Create an AWS IAM role, and then attach an Amazon SQS policy with a SendMessage permission. The policy allows you to publish messages from the API to Amazon SQS:
{  
  "Version": "2012-10-17",  
  "Statement": [  
    {  
      "Effect": "Allow",  
      "Resource": [  
        "arn:aws:sqs:example-region:example-account-id:example-sqs-queue-name"  
      ],  
      "Action": [  
        "sqs:SendMessage"  
      ]  
    }  
  ]  
}

Note: Replace example-region with your AWS Region, example-account-id with your AWS account ID, and example-sqs-queue-name with your SQS queue name.

  1. Create a REST API in API Gateway.
  2. In the API Gateway console, create an Amazon SQS integration for your REST API.
  3. Create a REST API resource or a REST API method:
    On the Resources screen, choose Create method.
    For Method type, choose POST.
    For Integration type, choose AWS Service.
    For AWS Region, choose your Region.
    For AWS Service, choose Simple Queue Service (SQS).
    For AWS Subdomain, keep this blank. This is an optional parameter in which you enter the subdomain that an AWS service uses. Check the AWS service documentation to confirm the availability of a subdomain.
    For HTTP method, choose POST.
    For Action Type, choose Use path override.
    For Path override, enter the / character.
    For Execution role, enter the ARN of the IAM role.
    For Default Timeout, choose an option for your setup.
    Expand HTTP request headers.
    Choose Add header.
    For Name, enter Content-Type.
    Choose Add header.
    For Name, enter X-Amz-Target.
    Select Create method.
    Choose the POST method Integration Request.
    Choose Edit.
    For Request body passthrough, leave as the default option When no template matches the request content-type header.
    Expand URL request headers parameters.
    Choose Add request header parameter.
    For Name, enter Content-Type.
    For Mapped from, enter method.request.header.Content-Type.
    Choose Add request header parameter.
    For Name, enter X-Amz-Target.
    For Mapped from, enter method.request.header.X-Amz-Target.
    Choose Save.
  4. Deploy the REST API.
  5. To test the setup, send the following request to API Gateway:
curl --location --request POST 'https://example-api-id.execute-api.example-region.amazonaws.com/example-stage/example-resource' \
  --header 'Content-Type:application/x-amz-json-1.0' \
  --header 'X-Amz-Target:AmazonSQS.SendMessage' \
  --data-raw '{
    "QueueUrl": "https://sqs.<region>.<domain>/<awsAccountId>/<queueName>/",
    "MessageBody": "This is a test message"
}'

Note: Replace example-api-id with your API ID, example-region with your Region, example-stage with your testing stage name, and example-resource with your resource name. To find your QueueUrl value, check your Amazon SQS queue details.

If your integration is successful, then your response looks similar to the following:

{"MD5OfMessageBody":"fafb00f5732ab283681e124bf8747ed1","MessageId":"b5aef1f3-af31-49f2-9973-6f802f7753e6"}

Note: The expected response from AWS JSON protocol is different from AWS query protocol, even for the same API call. Refer to the SQS API reference for example responses from each protocol.

Resolve common errors

UnknownOperationException error

You can get an UnknownOperationException error from both AWS query protocol and AWS JSON protocol.

If you use the AWS query protocol, then you get an UnknownOperationException when you don't configure the Content-Type as "application/x-www-form-urlencoded" in the integration request HTTP header. You also get this error when you don't add the SendMessage action to the integration request's mapping template. To resolve this error, make sure to correctly format Content-Type and to include the SendMessage action in the mapping template.

If you use the AWS JSON protocol, then you get an UnknownOperationException error when you don't send or don't correctly configure the Content-Type and X-Amz-Target headers. To resolve this error, configure the Content-Type header as "application/x-amz-json-1.0" and configure the X-Amz-Target header as AmazonSQS.{SQS-Action}, and include both headers in the request.

AccessDenied error

You can get an AccessDenied error from both the AWS query protocol and the AWS JSON protocol.

You get an AccessDenied error occurs when the API integration execution role doesn't have the sqs:SendMessage permission set to send messages to the SQS queue.

You also get this error when you use the AWS query protocol and pass unsupported special characters in the request body payload string. You must encode special characters to avoid this error. Add the $util.urlEncode() function in the mapping template to convert the request body from a string to an encoded format. The following is an example mapping template:

Action=SendMessage&MessageBody=$util.urlEncode($input.body)

The following example includes the required permissions to send messages to the SQS queue:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Resource": [
        "arn:aws:sqs:example-region:example-account-id:example-sqs-queue-name"
      ],
      "Action": [
        "sqs:SendMessage"
      ]
    }
  ]
}

Note: Replace example-region with your Region, example-account-id with your account ID, and example-sqs-queue-name with your SQS queue name.

KMS.AccessDeniedException error

You can get an KMS.AccessDeniedException error from both AWS query protocol and AWS JSON protocol.

You get a KMS.AccessDeniedException error when the API integration execution role can't perform operations through AWS Key Management Service (AWS KMS). To resolve this error, configure permissions to perform operations on the AWS KMS keys that are attached to the Amazon SQS server-side encrypted queue.

The following example includes the required permissions to perform operations on the KMS keys that are attached to the SQS queue:

{
  "Sid": "Allow use of the key",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::example-account-id:role/example-api-gw-integration-execution-role"
  },
  "Action": [
    "kms:Encrypt",
    "kms:GenerateDataKey*",
    "kms:Decrypt"
  ],
  "Resource": "*"
}

Note: Replace example-account-id with your account ID and example-api-gw-integration-execution-role with your execution role name.

SignatureDoesNotMatch error

You can get a SignatureDoesNotMatch error when you use the AWS query protocol.

You get a SignatureDoesNotMatch error when the HTTP method of the integration request is set to GET instead of POST. To resolve this error, set the HTTP method to POST.

InvalidAddress error

You can get an InvalidAddress error when you use the AWS JSON protocol.

You get an InvalidAddress error when the SQS Queue URL in the body payload is incorrect. To resolve this error, check the Queue URL of the SQS queue that the API call targets.

SerializationException error

You can get a SerializationException error when you use the AWS JSON protocol.

You get a SerializationException error when the body payload is invalid JSON. For example, your JSON might have a missing or extra comma, or a missing or extra curly brace. To resolve this error, use a JSON formatter tool to find syntax mistakes in your JSON.

MissingRequiredParameterException error

You can get a MissingRequiredParameterException error when you use the AWS JSON protocol.

You get a MissingRequiredParameterException error when you don't include one or more of the required parameters in the body payload. The required parameters depend your API call. For example, you get this error from a SendMessage API call when the MessageBody parameter is missing. Check the SQS API Reference for required parameters and syntax.

Related information

Integrate Amazon API Gateway with Amazon SQS to handle asynchronous REST APIs

How do I use API Gateway as a proxy for another AWS service?

7 Comments

is there any way to make this work without overriding path?

replied 2 years ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
EXPERT
replied 2 years ago

Hi AWS, is there a way to check the constructed message size does not exceed 256 KiB and directly returning a bad request response using VTL?

replied a year ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied a year ago

If you receive this error:

The request must contain the paramter MessageGroupId.

modify your template body to

Action=SendMessage&MessageGroupId=$context.extendedRequestId&MessageBody=$input.body

This provides the unique request ID as the message group ID for SQS. You may also have to enable Content-based deduplication for your SQS queue.

replied a year ago

I'm receiving this error:

{"Error":{"Code":"SignatureDoesNotMatch","Message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

replied 6 months ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied 6 months ago