Questions tagged with Extract Transform & Load Data
Content language: English
Select up to 5 tags to filter
Sort by most recent
Browse through the questions and answers listed below or filter and sort to narrow down your results.
I'm trying to build an ETL pipeline with AWS Glue, and the first step is to copy raw data from the original source to a staging bucket. The job is rather simple: source is a data catalog table (from...
1
answers
0
votes
301
views
asked 5 months agolg...
Hello,
In a Glue ETL made of nodes: Amazon S3, Change Schema, AWS Glue Data Catalog with the table "us_spending" backed by S3, I have the following error:
> Error Category: PERMISSION_ERROR;...
1
answers
0
votes
238
views
asked 5 months agolg...
I am looking for the best way to pass a parameter from one glue job to another within a step function.
Each day, I will receive a file. In the file there will be data for certain dates. The first...
1
answers
0
votes
968
views
asked 5 months agolg...
Hi. I am trying to run an AWS Glue job where I transfer data from S3 to Amazon Redshift. However, I am receiving the following error:
```
Error Category: UNCLASSIFIED_ERROR; An error occurred while...
2
answers
0
votes
1323
views
asked 5 months agolg...
```
df = spark.read.parquet("s3://folder/")
df = df.withColumn('filename', input_file_name())
AmazonS3_node1697616892615 = DynamicFrame.fromDF(df, glueContext, "s3sparkread")
```
if this is the code...
1
answers
0
votes
416
views
asked 5 months agolg...
I'm trying to achieve data change capture using AWS Glue and don't want to use DMS. I'm trying to transfer data between two Oracle RDS instances which are in different AWS Account. Here I am trying to...
1
answers
0
votes
547
views
asked 5 months agolg...
I'm trying to achieve data change capture using AWS Glue and don't want to use DMS. I'm trying to transfer data between two Oracle RDS instances which are in different AWS Account. Here I am trying to...
1
answers
0
votes
510
views
asked 5 months agolg...
is it possible to conveniently mask all columns in a Redshift table? The example in the [docs](https://docs.aws.amazon.com/redshift/latest/dg/t_ddm.html#ddm-example) only masks one column but is there...
1
answers
0
votes
543
views
asked 5 months agolg...
For a DeltaTarget, glue.create_crawler is not recognizing the parameter "CreateNativeDeltaTable"lg...
Unknown parameter in Targets.DeltaTargets[0]: "CreateNativeDeltaTable", must be one of: DeltaTables, ConnectionName, WriteManifest.
However, in the documentation it shows the parameter (see...
1
answers
0
votes
184
views
asked 5 months agolg...
I have a Delta Lake table saved in s3. I am running the following command:
spark.sql("""
CREATE EXTERNAL TABLE db.my_table
USING DELTA
LOCATION 's3://path/to/delta/table
""")
Everything...
2
answers
0
votes
459
views
asked 5 months agolg...
Hello.
I am currently completing the AWS Industry Quest: Financial Industry but I'm unable to do so as 2 modules produce errors that prevent me from continuing. The modules "Securing a Banking Data...
1
answers
0
votes
476
views
asked 5 months agolg...
I have a strange problem.
I created my first Visual ETL job; it runs fine, but there are a few mistakes in the output schema.
The problem is now when I try to open the job, it will not load in...
2
answers
0
votes
372
views
asked 5 months agolg...