- Newest
- Most votes
- Most comments
While AWS Glue can connect to various databases using JDBC connections, there are some challenges when working with older versions of ClickHouse that require specific drivers.
For your scenario with an older ClickHouse version using clickhouse-jdbc-0.3.1.jar, you have a few options:
-
AWS Database Migration Service (DMS): You could use AWS DMS to migrate data from ClickHouse to Amazon Redshift. DMS supports various database sources and can handle the migration process, though you would need to verify compatibility with your specific ClickHouse version.
-
AWS Glue with custom connectors: While you mentioned that custom definition drivers cannot be tested in the AWS Glue connections section, you can still create a custom Glue job that uses your specific ClickHouse JDBC driver. You would need to:
- Upload the driver JAR to an S3 bucket
- Reference this JAR in your Glue job configuration
- Write a custom script that loads the driver and establishes the connection
-
Alternative data flow: Extract data from ClickHouse to Amazon S3 first (using a custom application or script), then use AWS Glue or Redshift's COPY command to load the data from S3 to Redshift.
-
Direct connection from Redshift: Use Redshift's federated query capability to query the ClickHouse database directly, though this would require compatible connectivity between the systems.
If you're specifically trying to use AWS Glue with the older ClickHouse driver, you might need to create a custom Glue ETL job with additional dependencies that include your specific driver version. This approach would give you more control over the connection process than using the pre-configured connection testing in the AWS Glue console.
Sources
Loading data from RDS to Redshift Direct | AWS re:Post
Using Amazon S3 as a target for AWS Database Migration Service - AWS Database Migration Service
Gaming Analytics: Leveraging AWS Glue and Amazon Redshift Spectrum for Player Insights | AWS for Games Blog
answered a year ago
Relevant content
asked 4 years ago
asked 3 years ago
