When developing some Glue scripts from a successful Crawler run from a JDBC Oracle data source, I am encountering an error that I cannot resolve.
An error occurred while calling o102.pyWriteDynamicFrame. Decimal precision 136 exceeds max precision 38
I've tried specifically editing the schema in the catalog, but then printing the schema in the Glue job reveals that the changes did not actually work. I believe this is a "feature" of DynamicFrames where each row can have a slightly different schema.
I've tried using SelectFields so that the offending data columns aren't included, but the job fails with the same error.
I've tried using Lake Formation to generate the blueprint and workflow to see if that would use more default job options but it results in the same errors.
As I have hundreds of tables where this is going to be an issue, I'm trying to find the most robust way to solve without having to manually change code in each job.
Interestingly, the issue occurs only when writing to s3--the loading of the data into the DynamicFrame seems to be okay and the ApplyMapping also doesn't result in an error. It's just the last stage before job.commit() that is causing the problem.
Can anyone advise a workaround for this very frustrating issue?
Many thanks for reading.
