Error trying to load data from SQL Server "decimal" type


I have a Glue job that's pulling data from a SQL Server view and trying to create/update a table definition in the data catalog. If I include one of the "decimal" type fields, I keep getting this:

An error occurred while calling o132.pyWriteDynamicFrame. Schema cannot be updated: data type has been changed, new data type: [amountAllocated] optional int64 amountAllocated (DECIMAL(11,4)) does not match old data type: [amountAllocated] optional int64 amountAllocated (DECIMAL(10,4))

It varies from DECIMAL(9,4) / DECIMAL(10,4) and the error above, so it looks like it defines the type when it sees one record, and then when it finds another record that exceeds the previously defined type, it crashes.

Any suggestions of a way for me to define everything on the database side and just let Glue create a type that's big enough for the whole table to be loaded? I tried adding "CAST ... AS DECIMAL(38,4)" on the DB side but nothing changes.

asked 15 days ago9 views
1 Answer

I found a workaround which is to use a SQL Query filtering and forcing the conversion there, by specifying all the fields in the view and using CAST ... AS DECIMAL(38,4) there. Is there a way for me to do this by default to all fields of that type just so I don't have to hardcode field names?

answered 15 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions