Questions tagged with Extract Transform & Load Data
Content language: English
Select up to 5 tags to filter
Sort by most recent
Browse through the questions and answers listed below or filter and sort to narrow down your results.
AWS Glue Job Errorlg...
Im trying to convert CSV files in S3 to Parquet in another S3 bucket. So first I read the CSV files using a crawler, load the data into a Table, and then use a Job to convert from the Table to S3 in...
0
answers
0
votes
8
views
asked 32 minutes agolg...
Is there a way to use AWS SDK or some python script to get all the job names that failed on specific job queue on AWS Batch? After extracting them, I would like to upload this to AWS S3 or RDS so that...
1
answers
0
votes
72
views
asked a day agolg...
I'm having the same issue. Data is stored in below format in s3 as JSON array with partitions
S3 path - s3://fleet-fuelcard-data-import-dev/lambda/fuelsoft-morgan/660306/2024/Apr/03-Apr-2024.json....
1
answers
0
votes
32
views
asked 6 days agolg...
In AWS Glue jobs, within the Targets node, I am unable to see the data types such as struct, array or map while changing the schema. Does AWS Glue not support these data types?
1
answers
0
votes
146
views
asked 8 days agolg...
I've successfully set up AWS Glue with an RDS database serving as the data source and a Snowflake database as the data target. In this setup, I've configured AWS Glue crawlers to catalog the metadata...
0
answers
0
votes
170
views
asked 9 days agolg...
Hi,
When I try to export (expdp) a database schema in Oracle RDS using SQL Developer, I get the error below related to dump file size. This seems to be caused by default FILESIZE parameter in AWS...
1
answers
0
votes
253
views
asked 12 days agolg...
Hello, I am relatively new to Glue and encountering some challenges with Glue ETL.
Our setup involves a datalake that retrieves data from a backend database as its source. This datalake is...
1
answers
0
votes
209
views
asked 14 days agolg...
Hello,
I have parquets files in S3 that i parse using Glue Crawler and query in Athena. I found that some files have two columns "x" and "y" that have a type **int64** while other files have them as...
1
answers
0
votes
201
views
asked 15 days agolg...
In our ETL process we are building out a pipeline where someones job is to take input files (ex. csv) and map the columns to existing column names. After the mapping is complete a glue workflow will...
0
answers
0
votes
171
views
asked 16 days agolg...
I am reading multiple files from S3 and writing the output to Redshift DB. Below is my code to read all the files from a S3 location (s3://abc/oms/YFS_CATEGORY_ITEM/)
```
yfs_category_item_df =...
2
answers
0
votes
487
views
asked 22 days agolg...
We have a glue job that is writing large number of items to dynamo.
**If a write to dynamo fails, how can we have access to these individual failed records in order to attempt to resolve and...
1
answers
0
votes
274
views
asked 23 days agolg...
Hi I have created an external table on AWS Glue catalog db .
The table points to a lz4 compressed file on an s3.
the table definition looks like this
```
CREATE EXTERNAL TABLE `myapplogs`(
...
1
answers
0
votes
281
views
asked 23 days agolg...