The Challenge of Large Message Replication

By default, both AWS Database Migration Service (AWS DMS) and Amazon Managed Streaming for Apache Kafka (Amazon MSK) have a default 1MB limit on message sizes. There are scenarios where you might need to handle messages larger than this default limit. If you need to send messages larger than 1MB in Amazon MSK, you'll need to modify configurations at multiple levels.

Misconfigured parameters in large message scenarios result in data truncation or replication task failures. Understanding when and how to adjust these parameters is crucial for successful data migration.

AWS DMS

When replicating data to Amazon MSK, AWS DMS behaves as a Kafka producer. Two configuration parameters directly control the maximum size of the Kafka messages that AWS DMS can generate and send during replication:

MessageMaxBytes

The maximum size in bytes for records created on the endpoint. This setting is configured at the DMS Target Endpoint level.

• Default: 1MB (1,000,000 bytes)
• Maximum: 100MB (104,857,600 bytes)

Example: 10MB message

{
  "MessageMaxBytes": 10485760
}

Maximum LOB Size

Maximum LOB size defines how much LOB data AWS DMS can include in a single record, allowing DMS to pre-allocate memory and load LOB data in bulk. LOBs that exceed the maximum LOB size are truncated, and a warning is issued to the log file. This setting is configured at the DMS Replication Task level.

• Default: 32KB for Limited LOB mode
• Maximum: 100MB (102,400 KB)

Example: 10MB message

Maximum LOB size (KB): 10240

Amazon MSK

Amazon MSK parameters must be configured to accept, store, and replicate large messages across the cluster.

message.max.bytes (Broker Setting)

The largest record batch size allowed by Kafka

• Default: 1MB (1,048,576 bytes)

replica.fetch.max.bytes (Broker Setting)

The number of bytes of messages to attempt to fetch for each partition during replication.

• Default: 1MB (1,048,576 bytes)

max.message.bytes (Topic Setting)

The largest record batch size allowed by Kafka. This is a topic-level setting.

• Default: Inherits from broker setting

Configuration scope guidance :

Kafka allows large-message limits to be configured at either the broker level or the topic level.

In this configuration, message.max.bytes is set at the broker (cluster) level so that all topics automatically inherit the same maximum message size. This broker-level approach is appropriate when large messages are expected across most or all topics in the cluster.

However, it is generally recommended to configure max.message.bytes at the topic level to scope large-message support only to the topics that require it, reducing cluster-wide impact and limiting risk.

Important notes:

• max.message.bytes (topic) must be less than or equal to the broker-level message.max.bytes. If the topic already exists, changing the broker configuration alone is not enough.

• socket.request.max.bytes must be greater than or equal to message.max.bytes; otherwise, brokers may reject large produce requests. The default value is 104,857,600 bytes (100 MB).

• The parameters replica.fetch.max.bytes and socket.request.max.bytes must be configured at the broker level to ensure that large messages can be replicated successfully between brokers.

Configure broker-level and topic-level settings for large messages

Large-message support in Amazon MSK requires changes at both the broker (cluster) level and the topic level.

Amazon MSK uses cluster configurations to manage broker-level parameters. In the Amazon MSK console, navigate to Configurations and create the new or updated configuration.

Example: 10MB message:

• Broker-level (cluster) configuration

message.max.bytes=10485760
replica.fetch.max.bytes=10485760
socket.request.max.bytes=10485760

Topic-level parameters are managed separately, either via Kafka tools or automation. Using Kafka admin tools:

• Topic-level configuration

max.message.bytes=10485760

kafka-configs.sh \
  --bootstrap-server <broker-endpoint> \
  --alter \
  --entity-type topics \
  --entity-name <topic-name> \
  --add-config max.message.bytes=10485760

When to Change Parameters

Decision Criteria

Data Characteristics	Action Required	Parameters to Change
Max size < 1MB	No changes needed	Use defaults
Max size > 1MB	Change DMS Endpoint + DMS Task + MSK	MessageMaxBytes, Maximum LOB size, message.max.bytes, replica.fetch.max.bytes, socket.request.max.bytes

Conclusion

Successfully configuring AWS DMS and Amazon MSK for large message replication requires careful analysis of data characteristics, systematic parameter alignment, and ongoing performance monitoring.

The key principles are:

1. Analyze First: Understand your data size distribution before making changes
2. Align Parameters: Ensure consistency across DMS and MSK configurations
3. Monitor Continuously: Track performance and data integrity metrics
4. Plan for Scale: Consider memory, network, and storage implications
5. Test Thoroughly: Validate configurations in development environments

Important Notes:

• Full LOB mode is not supported for Kafka targets
• Limited LOB mode treats LOB columns as VARCHAR data types and pre-allocates memory for better performance
• A primary key is mandatory for tables containing LOB columns during Change Data Capture (CDC) operations

With proper configuration, DMS can reliably replicate messages up to 100MB to Amazon MSK while maintaining performance and data integrity.

References

Using Apache Kafka as a target for AWS Database Migration Service

Handling large messages in Amazon Managed Streaming for Apache Kafka (MSK)