DMS: Table suspended due to “overflow occurred” on varchar column with short values

Question

I’m using AWS DMS to replicate data from a MySQL source (version 5.7) to a MySQL target (version 8.4). 
During full load + CDC, DMS suspends certain tables with this error:
```
[SOURCE_CAPTURE] T: Truncation occurred. DataLen: 65535, BufferLen: 800  
[SOURCE_CAPTURE] W: Error parsing column 'col_name' in table 'table_name' - overflow occurred  
[SOURCE_CAPTURE] W: Suspending table 'table_name' 
```
Questions:
1.  Why is DMS reporting overflow on varchar columns with very short values like “jk”?
2.  Could this be caused by charset/collation mismatch between source and target?
3.  Could DMS’s internal buffer allocation or metadata caching be misinterpreting column lengths?
4.  Why does the error appear on different columns across retries even if the table schema is unchanged?

re:Post Agent · Answer

The error you're experiencing with AWS DMS suspending tables due to "overflow occurred" is likely related to character set and collation differences between your MySQL 5.7 source and MySQL 8.4 target.

1. DMS is reporting overflow on short values like "jk" because the issue isn't necessarily with the actual data length but with how DMS interprets the column metadata. The error shows a significant discrepancy between DataLen (65535) and BufferLen (800), suggesting DMS is incorrectly calculating the potential size needed for the column.

2. Yes, charset/collation mismatches are a common cause of this issue. MySQL 8.x uses utf8mb4_0900_bin collation by default, which may be incompatible with the collation used in your MySQL 5.7 source. Previous versions of DMS had known issues with the utf8mb4_0900_bin collation specifically.

3. DMS's internal buffer allocation could indeed be misinterpreting column lengths. When DMS encounters columns with unbounded numeric types or certain character sets, it may allocate buffer space incorrectly. This is particularly true when dealing with VARCHAR columns that might have different byte-per-character requirements between source and target due to character set differences.

4. The error appearing on different columns across retries despite unchanged schema suggests that DMS is encountering the issue during data processing rather than schema interpretation. As different data rows are processed in different orders during retries, columns containing certain character combinations might trigger the overflow condition.

To resolve this issue, you could:

- Use the CharsetMapping extra connection attribute to specify character set mapping between your source and target databases
- Consider upgrading your DMS version if you're not on the latest, as newer versions have fixed issues with utf8mb4_0900_bin collation
- For VARCHAR columns causing problems, you might need to modify them to use TEXT data type instead, which handles variable-length data differently
- Add the MapUnboundedNumericAsString=true extra connection attribute to both source and target endpoints if numeric columns are involved  
**Sources**  
[Get error in DMS Validation | AWS re:Post](https://repost.aws/questions/QUyuc4mqrsQdeozUfynoxfIw/get-error-in-dms-validation)  
[Troubleshooting - Amazon Aurora MySQL Migration Handbook](https://docs.aws.amazon.com/whitepapers/latest/amazon-aurora-mysql-migration-handbook/troubleshooting.html)  
[Converting Database Character Sets to utf8mb4 in AWS RDS/Aurora MySQL | AWS re:Post](https://repost.aws/articles/ARFx4npxdtT4-B-Wdeu_UNuw/converting-database-character-sets-to-utf8mb4-in-aws-rds-aurora-mysql)  
[AWS DMS release notes - AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_ReleaseNotes.html)

DMS: Table suspended due to “overflow occurred” on varchar column with short values

Add your answer

Relevant content