All Content tagged with Extract Transform & Load Data

I have written a custom script to transform a column, data transformed successfully and I can verify that in data preview in aws glue studio, acft_tail_num -> 8911(earlier this was N8911 ) when I joi...

I am using KDS which pushes to Kinesis Data Firehose which loads data into my S3 bucket. My S3 bucket is currently structured as follows: s3://bucket_name/anyCompany/year/month/day/hour/jsonFiles Eac...

I am using Spark to read and write to a S3table bucket. I have configured it the following way: spark = SparkSession.builder.appName("SparkIcebergSQL") \ .config("spark.jars.packages", "org.apach...

Please find the two DQDL rules I am trying to formate. They are not working as desired. Please let me know if there is any issue with the syntax. - check_type: "DQDL" check: CustomSql "SELECT a,b...

I am writing a simple AWS Glue ETL job as a Python script which reads a record from a Snowflake table and updates the corresponding record in Salesforce. However, I am getting the following error "Err...

I'm using AWS Glue with the SAP OData Connector to extract data from an SAP OData service. The service has nullable fields properly configured on the SAP side (e.g., ESTIMATEDCOSTCOSTINGRUNDATE), and ...

hello, i am experiencing an error during one of the cloudquest Lab "document indexing and search" with the the Role "data analyticis" which consist of through AWS glue ingesting data to aws opensearch...

**Problem Summary:** I have successfully implemented AWS Glue Job Bookmarks for incremental processing from Snowflake to Iceberg using AWS Glue 4.0. However, the same code fails when running on AWS G...

I have an S3 bucket that is populated by csv files from the output of an API run. The problem is that before and after a particular date, my column order is different which is causing the load job to ...

In Glue, I am trying to join some tables together. When joining these tables it works fine until I have to rename the keys due to overlapping field names. After this, sometimes I will get a perfect jo...

I'm using the SAP OData Connector for AWS Glue to extract SAP data based on specific condition involving a single field (such as a fiscal period or profit center), the expected record count is relati...

I have a data repository in JSON that includes an attribute called "timestamp" (which is a reserved word). This attribute is part of a nested JSON/struct. The way I query the data is by creating a Gl...