All Content tagged with Extract Transform & Load Data

I'm testing DataBrew as a no-code ETL option. Unfortunately, even a small toy-job run for over 10 minutes: * Read a 1000 rows dataset from RDS * Filter on a date column and restrict to 2 days -> 20 re...

I'm running a simple test from Redshift to Redshift using JDBC connector and Glue ETL. Objective is to test JOB BOOKMARK feature. I don't have a primary key defined in source, so using LoadTimestamp w...

I am trying to use the Glue table optimization feature that you find in the Glue console -> Go into any table -> "Enable optimization". The process asks you for an IAM role, for which I created a role...

put a csv file into SageMaker data lake and when I go to preview it I receive this error

We are utilizing AWS Glue as our ETL engine to transfer data from on-premises and cloud databases to an S3 bucket. Currently, we have a Lambda function that triggers a Glue job, but each time we invok...

Error Category: UNCLASSIFIED_ERROR; Failed Line Number: 245; An error occurred while calling o147.sql. Insufficient LF permission(s) on chorus (Service: Glue, Status Code: 400, Request ID: 0054e9fa-5d...

I've created a Glue ETL job with a Redshift table as the source node and a dropDuplicates transform over some specific keys (not the entire row, because they have created timestamps). I understand the...

I am using AWS Database Migration Service (DMS) to migrate data from MongoDB (source) to Amazon S3 (target). The migration includes both Full Load (FL) and Change Data Capture (CDC). However, after so...

I have a DMS job that writes records from postgres to s3 folder in parquet format. I want, for compliance reasons, to delete records from the parquet file when the records are deleted from source. I k...

Hi AWS Experts, I am working on an AWS Glue job to transfer data from an on-premise SQL Server to AWS RDS (SQL Server). My goal is to achieve a 20-second execution time, but despite multiple optimiza...

 I get error below when I try to use a custom regex pattern to detect sensitive data in Glue Studio Error Category: UNCLASS...

Can I enable optimization in AWS Glue when I create an iceberg table? Is there an option like a tableProperty? df.writeTo(f"{table_name}") \ .tableProperty("format-version", "...