SQS FiFo Deduplication

Possible Solutions to Track Deduplicated Messages:

• Custom Logging: Implement custom logging on the sender side to track the MessageDeduplicationId for each message before sending it to SQS. This can help you identify which messages might have been discarded by comparing the logs with the messages successfully received by the consumer.

• Use Unique Message Attributes: Attach a unique attribute to each message, such as a timestamp or a UUID, which can help in identifying duplicates when comparing the sent and received messages.

• Custom Deduplication Handling: Handle deduplication manually by maintaining a cache or a database that tracks the MessageDeduplicationId of recently sent messages. This way, you can keep a record of all messages and check which ones might have been discarded.

Example of Custom Logging and Unique Attributes:

• Sender Side:

import boto3
import uuid
import time

sqs = boto3.client('sqs')
queue_url = 'https://sqs.your-region.amazonaws.com/your-account-id/your-queue-name'

def send_message(message_body):
    deduplication_id = str(uuid.uuid4())
    message_group_id = 'your-group-id'  # Replace with your group ID logic
    timestamp = str(int(time.time()))
    
    response = sqs.send_message(
        QueueUrl=queue_url,
        MessageBody=message_body,
        MessageDeduplicationId=deduplication_id,
        MessageGroupId=message_group_id,
        MessageAttributes={
            'Timestamp': {
                'StringValue': timestamp,
                'DataType': 'String'
            }
        }
    )
    
    print(f'Sent message with DeduplicationId: {deduplication_id}, Timestamp: {timestamp}')
    return response

# Example usage
send_message('Your message body')

• Receiver Side:

import boto3

sqs = boto3.client('sqs')
queue_url = 'https://sqs.your-region.amazonaws.com/your-account-id/your-queue-name'

def receive_messages():
    response = sqs.receive_message(
        QueueUrl=queue_url,
        MaxNumberOfMessages=10,
        MessageAttributeNames=['All']
    )
    
    for message in response.get('Messages', []):
        deduplication_id = message['MessageDeduplicationId']
        timestamp = message['MessageAttributes'].get('Timestamp', {}).get('StringValue')
        
        print(f'Received message with DeduplicationId: {deduplication_id}, Timestamp: {timestamp}')
        # Process the message
        sqs.delete_message(
            QueueUrl=queue_url,
            ReceiptHandle=message['ReceiptHandle']
        )

# Example usage
receive_messages()

While there is no built-in feature to directly identify which messages were discarded due to deduplication in SQS FIFO queues, implementing custom logging and using unique message attributes can help track and identify duplicates. Additionally, monitoring CloudWatch metrics can provide further insights into message processing discrepancies.