Questions tagged with AWS Data Pipeline
Content language: English
Select up to 5 tags to filter
Sort by most recent
Browse through the questions and answers listed below or filter and sort to narrow down your results.
We currently run into the Problem, that some data analytic workloads exceed the 15 minutes timeout of a lambda.
It is a multistep process with steps that are parallelizable and some that are not....
3
answers
0
votes
2757
views
asked 2 years agolg...
What are the key points to choose one of the following:
* Data pipeline,
* Step function
* Amazon Managed Workflows for Apache Airflow
1
answers
0
votes
2554
views
I have a Data Pipeline which reads CSV files from an S3 bucket and copies the data into an RDS database.
I specify the bucket/folder name and it goes through each CSV file in the bucket/folder and...
2
answers
0
votes
344
views
asked 3 years agolg...
I am using Data Pipelines to replicate data from RDS to Redshift with one pipeline per table they are replicating.
Is it possible to consolidate the pipelines into a single pipeline for multiple...
1
answers
0
votes
558
views
asked 3 years agolg...
A customer has a MySQL database and want to offload historical data to another platform for reporting and analysis. Currently they are facing performance issues during normal operations.
What do I...
2
answers
0
votes
1375
views
asked 4 years agolg...
Hi AWS Team,
There is a data pipeline which is stuck in the state DELETING for some time now, which has never happened before.
The pipeline ID is df-02620721YVOGSCYDLKDT, in the us-west-2...
1
answers
0
votes
261
views
asked 5 years agolg...
Hi,
I am facing a weird issue while trying to set up a **DataPipeline** via **Cloudformation**.
The Cloudformation yaml file is used to create the two needed Roles ( **DataPipelineDefaultRole**...
2
answers
0
votes
872
views
asked 5 years agolg...
Hi,
all my AWS Data Pipelines i create to export data from a dynamodb table to a .csv in an s3 bucket are stuck in the "WAITING FOR RUNNER" state. I am following the tutorial here:...
4
answers
0
votes
896
views
asked 5 years agolg...
Hi team,
I was wondering if there are any resources that someone may have already worked on or have access to which outlines some of detail information on data ingestion methods that are suitable...
1
answers
0
votes
467
views
asked 6 years agolg...
A customer has a question about data sources
> “most of our data is stored in SQL databases, while the SageMaker docs
> say that I have to put it all in S3. It’s not obvious what the best
> way to...
3
answers
0
votes
832
views
asked 6 years agolg...
Hi All, Are there best practices and recommended tools for moving a copy of production data (in RDS/Dynamo/S3) to dev, with the ability to detect and anonymize personally identifiable information...
1
answers
1
votes
905
views
asked 6 years agolg...
What would be the best AWS ETL tool that a customer would use to set up ongoing loads of data into Redshift, that would provide some sort of transform functionality, similar to Microsoft SSIS? E.g....
2
answers
0
votes
352
views
asked 7 years agolg...