- Newest
- Most votes
- Most comments
The issue you're experiencing with AWS Schema Conversion Tool (SCT) during data extraction from BigQuery is likely related to the syntax differences between BigQuery and other SQL dialects. BigQuery indeed uses backticks (`) for identifiers, while many other databases use double quotes (").
Unfortunately, there isn't a specific "identifier quoting settings" option in AWS SCT that you can directly modify to change this behavior. The tool is designed to handle various database dialects, but it may not always perfectly translate the syntax between different systems.
To resolve this issue, you have a few options:
-
Manual Query Modification: If possible, you can manually edit the queries in AWS SCT to use backticks instead of double quotes for BigQuery identifiers. This would involve replacing instances of "database"."table" with
database.table. -
Custom SQL: In some cases, AWS SCT allows you to provide custom SQL for data extraction. If this option is available for your BigQuery source, you could write the queries using the correct BigQuery syntax.
-
Use Latest Version: Ensure you're using the latest version of AWS SCT. AWS regularly updates the tool to improve compatibility and fix issues. A newer version might handle BigQuery syntax more accurately.
-
Alternative Extraction Method: If the issue persists, you might need to consider alternative methods for extracting data from BigQuery, such as using BigQuery's export functionality to CSV files in Google Cloud Storage, then importing these into Amazon S3 for further processing.
-
Contact AWS Support: If you're unable to resolve the issue and it's critical for your migration process, consider reaching out to AWS Support. They might be able to provide specific guidance or workarounds for your use case.
Remember, AWS SCT has some limitations when using BigQuery as a source, including issues with certain types of functions and query structures. These limitations might be contributing to the syntax errors you're encountering.
Sources
Connecting to Google BigQuery with AWS Schema Conversion Tool - AWS Schema Conversion Tool
Schema Conversion Tool settings are missing from AWS DMS | AWS re:Post
answered 2 years ago
Relevant content
asked 2 years ago
asked 2 years ago
- AWS OFFICIALUpdated 9 months ago
The error log I am submitting here- CONTROLLER ERROR The following mandatory field(s) [agents] are missing com.amazon.exception.ActionValidationException: The following mandatory field(s) [agents] are missing at com.amazon.controller.action.TaskAction.validate(TaskAction.java:131) at com.amazon.controller.SCTController.validateAction(SCTController.java:497) at com.amazon.controller.SCTController.executeAction(SCTController.java:466) at com.amazon.controller.SCTController.executeOnTask(SCTController.java:239)