All Content tagged with Extract Transform & Load Data

I am building a Lakehouse solution using aws glue visual etl. When writing the dataset using the target s3 node in visual editor, there is no option to specify writemode() to overwrite When i checked ...

I’m running into an issue with AWS Clean Rooms when associating an AWS Glue table. The table association succeeds, but when another member of the collaboration tries to query it, they get: "Status not...

AWS Glue Python Shell currently only supports Python 3.6 and 3.9, which are both end-of-life. Is there a roadmap or planned timeline for releasing a newer Python runtime (e.g., 3.10+ ) for Python Shel...

I have written a custom script to transform a column, data transformed successfully and I can verify that in data preview in aws glue studio, acft_tail_num -> 8911(earlier this was N8911 ) when I joi...

I am using KDS which pushes to Kinesis Data Firehose which loads data into my S3 bucket. My S3 bucket is currently structured as follows: s3://bucket_name/anyCompany/year/month/day/hour/jsonFiles Eac...

I am using Spark to read and write to a S3table bucket. I have configured it the following way: spark = SparkSession.builder.appName("SparkIcebergSQL") \ .config("spark.jars.packages", "org.apach...

Please find the two DQDL rules I am trying to formate. They are not working as desired. Please let me know if there is any issue with the syntax. - check_type: "DQDL" check: CustomSql "SELECT a,b...

I am writing a simple AWS Glue ETL job as a Python script which reads a record from a Snowflake table and updates the corresponding record in Salesforce. However, I am getting the following error "Err...

I'm using AWS Glue with the SAP OData Connector to extract data from an SAP OData service. The service has nullable fields properly configured on the SAP side (e.g., ESTIMATEDCOSTCOSTINGRUNDATE), and ...

hello, i am experiencing an error during one of the cloudquest Lab "document indexing and search" with the the Role "data analyticis" which consist of through AWS glue ingesting data to aws opensearch...

**Problem Summary:** I have successfully implemented AWS Glue Job Bookmarks for incremental processing from Snowflake to Iceberg using AWS Glue 4.0. However, the same code fails when running on AWS G...

I have an S3 bucket that is populated by csv files from the output of an API run. The problem is that before and after a particular date, my column order is different which is causing the load job to ...