Questions tagged with Extract Transform & Load Data
Content language: English
Select up to 5 tags to filter
Sort by most recent
Browse through the questions and answers listed below or filter and sort to narrow down your results.
I am building a data pipeline to Load data into Redshift from an S3 data lake.
Data are stored in Parquet format on S3 and I would like to load them into the respective Redshift tables using an AWS...
1
answers
0
votes
474
views
asked 6 months agolg...
I need to create a aes encryption ECB UDF in the redshift. To achieve this, i have imported pycryptodome zip file in the S3 with the name Crypto.zip and create library in the redshift.
When i try to...
1
answers
0
votes
277
views
asked 6 months agolg...
Scenario:
Source table: Glue Data Catalog table **study** crawled from MySQL with columns:
* id (int),
* code (varchar),
* desc (varchar)
* and 2 other columns not used in the job.
Target table:...
0
answers
0
votes
102
views
asked 6 months agolg...
I want to add Confluent Cloud Apache Kafka as a Data source in AWS ETL job to read data stream from Kafka topic.
I created a cluster, topic, AWS SQS source connector and AWS S3 sink connector in...
1
answers
0
votes
364
views
asked 6 months agolg...
When i create a rule to compare arrays directly like here https://docs.aws.amazon.com/glue/latest/dg/data-quality-getting-started.html it works perfectly.
When i try use
```python
...
1
answers
0
votes
279
views
asked 6 months agolg...
In my study case, I have data coming from a relational database (which stores data directly from the product application) and it sends files into S3 by using AWS DMS with CDC logs;
Usually, I manage...
1
answers
0
votes
386
views
asked 7 months agolg...
Glue script job error spark_catalog requires a single-part namespace, but got [glue_catalog, foo]lg...
I am trying to query an iceberg table via the glue data catalog. This works fine in my visual etl job, but when I try to do it in a script, it's throwing an error. It's likely due to some sort of...
1
answers
0
votes
1540
views
asked 7 months agolg...
Hello,
I am upgrading Glue Version from 3.0 to 4.0. Using Hudi as datalake. I am getting below error -
py4j.protocol.Py4JJavaError: An error occurred while calling o973.pyWriteDynamicFrame.
:...
2
answers
0
votes
343
views
asked 7 months agolg...
We would like to know does AWS Glue encrypt data in Cache while processing the data through job.
1
answers
0
votes
184
views
asked 7 months agolg...
background:
Users upload files to an S3 bucket containing their predictions for events under certain circumstances. I want my query results to only show the most recently made prediction for the...
2
answers
0
votes
435
views
asked 7 months agolg...
Hi,
I have created a new AWS Glue visual ETL job with source- PostgreSQL, target- Snowflake.
Output schema option shows correct schema as per the source, but the python script shows all the datatype...
2
answers
1
votes
172
views
asked 7 months agolg...
I am trying to read a table from the same account that i used to create the table , the table is shared to other accounts through lake formation
in the glue job in the source account i get this...
1
answers
0
votes
263
views
asked 7 months agolg...