Questions tagged with AWS Data Pipeline
Content language: English
Select up to 5 tags to filter
Sort by most recent
Browse through the questions and answers listed below or filter and sort to narrow down your results.
Hi Team,
I am trying to run SQL statements in Redshift by triggering using EventBridge on S3 file arrival. I am able to run SQLs using Data API but how ever I want to pass the event details to...
1
answers
0
votes
565
views
asked 10 days agolg...
In AWS Glue jobs, within the Targets node, I am unable to see the data types such as struct, array or map while changing the schema. Does AWS Glue not support these data types?
1
answers
0
votes
188
views
asked a month agolg...
I have the the data of the access logs of the users in opensearch index, this data track the information of the users like at what time users access the door and got the arrival, but one user can...
0
answers
0
votes
362
views
asked a month agolg...
Hi team,
Our organization is embarking on a project to design a shared data model where we can efficiently store and manage a large payload.
Multiple lines of business will be both **updating this...
1
answers
0
votes
341
views
asked 2 months agolg...
I have a MySQL database in AWS RDS. I want to import bulk data in to table in database. which contains thousands of rows.
I need to do it from my website with also deployed in AWS. Web is developed...
1
answers
0
votes
250
views
asked 6 months agolg...
I am building a data pipeline to Load data into Redshift from an S3 data lake.
Data are stored in Parquet format on S3 and I would like to load them into the respective Redshift tables using an AWS...
1
answers
0
votes
474
views
asked 6 months agolg...
I want to add Confluent Cloud Apache Kafka as a Data source in AWS ETL job to read data stream from Kafka topic.
I created a cluster, topic, AWS SQS source connector and AWS S3 sink connector in...
1
answers
0
votes
364
views
asked 6 months agolg...
Don't see any data when Migrating existing and changes data from MYSQL to S3 using DMS and Kinesis.lg...
I am using DMS and Kinesis DataStream and Delivery stream to migrate existing and changes from MYSQL to S3 bucket. But I don't see the data is coming in S3. I specified the schema name and table name...
0
answers
0
votes
190
views
asked 7 months agolg...
Everyday a new emr cluster span up and terminated after completing the step job. Checking the cloudtrail, seems a Data Pipeline created it. I am not sure how to get more details like who created, what...
2
answers
1
votes
272
views
asked 7 months agolg...
I HAVE MULTIPLE CSVS ABOUT A SINGLE PATIENT AND I WOULD LIKE TO KNOW HOW DO I COMBINE ALL THE CSVS BECAUSE ALL THE COLUMNS INSIDE THE CSVS MAKE UP AN ALL THE INFORMATION FOR ONE PATIENT. THE CSV'S ARE...
1
answers
0
votes
427
views
asked 8 months agolg...
I am getting following error in my data pipeline
`We currently do not have sufficient t1.micro capacity in the Availability Zone you requested (ap-southeast-2a). Our system will be working on...
2
answers
0
votes
287
views
asked 9 months agolg...
Im trying to delete all the data pipeline that shows up in
```
aws datapipeline list-pipelines
```
in one go. How do I do that using aws-cli?
1
answers
0
votes
329
views
asked 9 months agolg...