Getting error "DataFormatConversion.SchemaValidationTimeout"

I was able to identify the problem. Leaving it here in case others hit this same error message. It had to do with us running a Glue crawler. This Glue crawler creates a new "version" of the table, even if nothing changed. Eventually you can build up SO many versions (we had 21k) that the Firehouse times out trying to get the schema for that table (apparently the large number of versions causes a problem there). Additionally, there is NO way to check how many versions you have in the AWS Console UI. The only way to see your count is if you "compare" versions, in the "compare" page it will show your version "id" which is just an incrementing integer, so this "id" would indicate your number of versions (assuming you dont delete them, which you can't do either through the UI).

Our fix was to delete them manually, one-by-one, through the AWS CLI (assuming version "id" as an integer between 1 ... 21k (in our case)).

So, lastly, as a warning to others. Don't use the GLUE crawler in an automated way (via scheduling) unless you intend to manually delete old versions. The crawler will not do that for you.

Edited by: Jason Malcolm on Oct 22, 2020 1:41 PM

Edited by: Jason Malcolm on Oct 22, 2020 1:41 PM