Questions tagged with Extract Transform & Load Data
Content language: English
Select up to 5 tags to filter
Sort by most recent
Browse through the questions and answers listed below or filter and sort to narrow down your results.
When i create a rule to compare arrays directly like here https://docs.aws.amazon.com/glue/latest/dg/data-quality-getting-started.html it works perfectly.
When i try use
```python
...
1
answers
0
votes
264
views
asked 6 months agolg...
In my study case, I have data coming from a relational database (which stores data directly from the product application) and it sends files into S3 by using AWS DMS with CDC logs;
Usually, I manage...
1
answers
0
votes
363
views
asked 6 months agolg...
Glue script job error spark_catalog requires a single-part namespace, but got [glue_catalog, foo]lg...
I am trying to query an iceberg table via the glue data catalog. This works fine in my visual etl job, but when I try to do it in a script, it's throwing an error. It's likely due to some sort of...
1
answers
0
votes
1338
views
asked 6 months agolg...
Hello,
I am upgrading Glue Version from 3.0 to 4.0. Using Hudi as datalake. I am getting below error -
py4j.protocol.Py4JJavaError: An error occurred while calling o973.pyWriteDynamicFrame.
:...
2
answers
0
votes
316
views
asked 6 months agolg...
We would like to know does AWS Glue encrypt data in Cache while processing the data through job.
1
answers
0
votes
167
views
asked 6 months agolg...
background:
Users upload files to an S3 bucket containing their predictions for events under certain circumstances. I want my query results to only show the most recently made prediction for the...
2
answers
0
votes
400
views
asked 6 months agolg...
Hi,
I have created a new AWS Glue visual ETL job with source- PostgreSQL, target- Snowflake.
Output schema option shows correct schema as per the source, but the python script shows all the datatype...
2
answers
1
votes
159
views
asked 6 months agolg...
I am trying to read a table from the same account that i used to create the table , the table is shared to other accounts through lake formation
in the glue job in the source account i get this...
1
answers
0
votes
242
views
asked 6 months agolg...
Hi! I have been searching and playing around with services and cannot seem to find what I need.
I am using the following architecture to guide me in building out my end-to-end solution:...
2
answers
0
votes
159
views
asked 6 months agolg...
When executing a task the last step is validating the data migrated with the source against target apparently using Athena, I have the following error:
2023-11-07T22:09:04 [VALIDATOR_TARGE ]E: Not...
1
answers
1
votes
567
views
asked 7 months agolg...
I'm looking for an open-source solution that can help us make our python API more accessible.
For simplicity's sake, the data is accessed using Athena and has three string fields A, B, C.
Every...
0
answers
0
votes
145
views
asked 7 months agolg...
When I´m about to start an ETL Job, usually I ask some main questions:
1. Where the original file/table is stored?
2. What should I do to delivery data to my end goal?
If I have already all the data...
1
answers
0
votes
417
views
asked 7 months agolg...