Questions tagged with AWS Data Pipeline

I am getting following error in my data pipeline `We currently do not have sufficient t1.micro capacity in the Availability Zone you requested (ap-southeast-2a). Our system will be working on...

Im trying to delete all the data pipeline that shows up in ``` aws datapipeline list-pipelines ``` in one go. How do I do that using aws-cli?

I want to get the notification on Failed DataPipelines through event bridge only when they fail. I used the following event pattern but this didn't work. It can be done through the data pipeline SNS...

We have one API gateway that receive data 24x7 of gzipped meter data and the data come in concurrently(some times 5000 posts per second, sometimes not much), we are sure the compressed data won't...

I'm trying to set up an AWS Data Pipeline so I can clone large huggingface repo's to S3. I'm encountering issue's when creating the permissions policy to use with a role for my data pipeline. [I'm...

Hello. I'm running SageMaker training jobs through a library called ZenML. The library is just there as an abstraction layer, so that when I return the artifacts gets automatically saved to S3. The...

Hi, I'm running a data pipeline from legacy DB(oracle) to Redshift using AWS Glue. I want to test the connection to the legacy DB before executing ETL without test query in working python...

Hi Team, I have set up an **AWS DataPipeline** to run my EMR jobs on `On-Demand` instances. However, I now want to switch to using `Spot` Instances to reduce costs. I have configured the...

Hello team, As per our client requirement we need to migrate data from quickbook to aws rds, for that we need to use appflow service and use connector for quickbook. Is anyone able to connect via...

I have to convert a pipeline to step function and I have this step which loads a dynamo db table with data from S3. I am not sure how to convert this to step function. Below is the step. Any idea how...

Good afternoon. I would like to know SE when we send a snapshoot of mysql 5.7 rds allocated storage size of 8tb. When I restore from S3 to a new RDS instance, it can only consume the instance that has...

Hi there, We are migrating from datapipeline to step function, and on datapipeline I am able to forward the log messages for individual steps to a log file, but this doesn't work on step function....