Questions tagged with Extract Transform & Load Data
Content language: English
Select up to 5 tags to filter
Sort by most recent
Browse through the questions and answers listed below or filter and sort to narrow down your results.
Hello, so the context is:
We have a DMS service that sends data from Oracle to s3, after that a job is run to create the raw stage, and after that raw to trusted. The problem is that they changed...
0
answers
0
votes
8
views
asked 2 hours agolg...
I am trying to create two DynamicFrames based on a column that is a boolean. I have tried
`dyf.split_rows({'mybool': {'=': 'true'}}, 'is_true', 'is_not_true')`
`dyf.split_rows({'mybool': {'=':...
1
answers
0
votes
34
views
asked 21 hours agolg...
AWS Glue Job Errorlg...
Im trying to convert CSV files in S3 to Parquet in another S3 bucket. So first I read the CSV files using a crawler, load the data into a Table, and then use a Job to convert from the Table to S3 in...
0
answers
0
votes
37
views
asked a day agolg...
Is there a way to use AWS SDK or some python script to get all the job names that failed on specific job queue on AWS Batch? After extracting them, I would like to upload this to AWS S3 or RDS so that...
1
answers
0
votes
125
views
asked 2 days agolg...
I'm having the same issue. Data is stored in below format in s3 as JSON array with partitions
S3 path - s3://fleet-fuelcard-data-import-dev/lambda/fuelsoft-morgan/660306/2024/Apr/03-Apr-2024.json....
1
answers
0
votes
37
views
asked 7 days agolg...
In AWS Glue jobs, within the Targets node, I am unable to see the data types such as struct, array or map while changing the schema. Does AWS Glue not support these data types?
1
answers
0
votes
149
views
asked 9 days agolg...
I've successfully set up AWS Glue with an RDS database serving as the data source and a Snowflake database as the data target. In this setup, I've configured AWS Glue crawlers to catalog the metadata...
0
answers
0
votes
171
views
asked 10 days agolg...
Hi,
When I try to export (expdp) a database schema in Oracle RDS using SQL Developer, I get the error below related to dump file size. This seems to be caused by default FILESIZE parameter in AWS...
1
answers
0
votes
260
views
asked 12 days agolg...
Hello, I am relatively new to Glue and encountering some challenges with Glue ETL.
Our setup involves a datalake that retrieves data from a backend database as its source. This datalake is...
1
answers
0
votes
213
views
asked 15 days agolg...
Hello,
I have parquets files in S3 that i parse using Glue Crawler and query in Athena. I found that some files have two columns "x" and "y" that have a type **int64** while other files have them as...
1
answers
0
votes
207
views
asked 16 days agolg...
In our ETL process we are building out a pipeline where someones job is to take input files (ex. csv) and map the columns to existing column names. After the mapping is complete a glue workflow will...
0
answers
0
votes
172
views
asked 17 days agolg...
I am reading multiple files from S3 and writing the output to Redshift DB. Below is my code to read all the files from a S3 location (s3://abc/oms/YFS_CATEGORY_ITEM/)
```
yfs_category_item_df =...
2
answers
0
votes
488
views
asked 23 days agolg...