lambda - redshift - status = Query does not exist

0

The same code works in one account but not another account. In a step function, lambda calls a redshift query, returns, next step calls get status. The lambda returns but the status call errors with msg "Query does not exist". The query in the previous step has successfully run and a statement id returned. Its only the step with status call that fails. Other lambdas in the same step function using the same pattern work correctly. These other steps use the same get status lambda as the steps that fail. The common item is the call to the redshift query in the previous step. When ever that is used the status call fails. However: The query executes correctly and the same lambda works in the same step function in another account. Other points: The redshift clusters were built with a cloud formation template and should be identical. The entire step function (unchanged other than account number change) works correctly in another account. The lambdas have been manually copied and pasted from the account that works to ensure they are identical.

Does any one have any suggestions, as all obvious checks have been done.

For completeness the code for the redshift query call is below, however as stated above this works in another account.

import json
import boto3
import base64
import urllib.parse
import botocore.session as bc
from botocore.exceptions import ClientError

ssm_client = boto3.client('ssm')

def lambda_handler(event, context):

    environment             = event['environment']
    source_bucket           = event['source_bucket']
    processed_bucket        = event['processed_bucket']
    role                    = event['role']
    region                  = event['region']
    database_name           = event['database_name']
    secret_name             = event['secret_name']
    secret_arn              = event['secret_arn']
    cluster_id              = event['cluster_id']
    proc_name               = event['proc_name']
    
    ssm_redshift_proc_name  = ssm_client.get_parameter(Name=proc_name, WithDecryption=True)
    redshift_proc_name      = ssm_redshift_proc_name['Parameter']['Value']
    query_str               = "call "+redshift_proc_name+"();"

    bc_session = bc.get_session()
    session = boto3.Session(
            botocore_session    = bc_session,
            region_name         = region,
        )
        
    client_redshift = session.client("redshift-data")
    res = client_redshift.execute_statement(
            Database            = database_name,
			SecretArn           = secret_arn,
			Sql                 = query_str,
			ClusterIdentifier   = cluster_id
		)

    return {
		'environment': environment,
		'source_bucket': source_bucket,
		'processed_bucket': processed_bucket,
		'role': role,
		'region': region,
		'database_name': database_name,
		'secret_name': secret_name,
		'secret_arn': secret_arn,
		'cluster_id': cluster_id,
        'statementid': res['Id']
    }


1 Answer
0
Accepted Answer

Hi,

Have you checked the IAM roles used? Are they identical in both accounts?

Are the DescribeStatement and ExecuteStatement permissions attached to the used IAM Role?

Thanks,

Ziad
answered 23 days ago
  • Hi Ziad

    Thanks for your response. Both state machines are executing with a role that has identical polices and trust relationships, including RedshiftDataApiFullaccess and LambdaFullAccess.

    However, I have just checked the individual lambda's. There are 36 lambda's and 35 had the correct role!

    Thank you very much. I had checked the permissions before but missed this (the two role names are very similar). Thank you for prompting me to check again.

  • Thank you for the update. Good news.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions