GLUE DATALOAD TO JDBC

0

If there are 10 records in input file , Glue used to reject all the 10 transactions and not commit anything from an input file to jdbc of one record has an issue. BUT NOW ... it's committing 9 records... is there any recent change happened to glue that commits data partially? How can we make glue to load an entire file or nothing from input to jdbc.

And why is that glue changes behavior suddenly from version 3 to version 4 ? Is there any documentation for this change that happened ?

asked 9 months ago221 views
2 Answers
0

AWS Glue is a managed ETL (Extract, Transform, Load) service that enables you to move data among data stores. It appears that you're experiencing a change in behavior from one version of Glue to another, specifically in how transactions are handled during a load to a JDBC target.

  • Partial Commits: In some data processing systems, it's common to allow partial commits where some records are successfully processed and others are rejected. This can be a feature or a setting that can be configured. If this behavior has changed in a new version of Glue, it could be due to a change in the underlying engine or a change in default settings.

  • Version Changes: Changes between versions of software can be for various reasons such as improvements, bug fixes, or alignment with new standards. These changes should be documented in the release notes or version documentation.

  • Documentation: Typically, major changes in behavior like this would be documented in the release notes for the specific version. You can look into the AWS Glue release notes or documentation for the versions in question to find details about this change. AWS also provides forums and support channels where you can ask for clarification on specific changes.

  • Transactional Control: If you want to ensure that either all records are committed or none (all-or-nothing), you might want to look into transaction control within your ETL process. This might require using a particular JDBC option or configuration within Glue itself. Glue's scripting or job configuration might have options to enforce this behavior.

profile picture
answered 9 months ago
  • tried different ways to load , using Aws crawler (write from catalog) and using from options …. Both are loading data partially until the point the load hits the error record

0

Any update here ?

answered 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions