All Content tagged with AWS Data Pipeline

AWS Data Pipeline is a web service that helps you reliably process and move data between different AWS compute and storage services, as well as on-premises data sources, at specified intervals.

Content language: English

Select up to 5 tags to filter
Sort by most recent
92 results
I was trying to build a Data pipeline with No-Code ETL using AWS Glue Studio, and I kept getting the error message "Node is Misconfigured", when I add the join node using Visual ETL (see attached). Un...
1
answers
0
votes
27
views
asked 18 days ago
Hi! I have some heavy endpoints in my backend right now to procesate and generate that. The flow is the next one. I get a request from the frontend side in my main microservice, I take those paramet...
2
answers
0
votes
157
views
asked 2 months ago
We have been incurring cost for aws data pipeline in **us-east-1** region but there is no data pipeline in that region in our account. We used this method to check datapipeline. https://docs.aws.amazo...
2
answers
0
votes
107
views
asked 4 months ago
We are planning to migrate our databases to Aurora, and we would like to know what the impact in our pipeline would be. We are currently running a dms task with Full load, ongoing replication that mig...
1
answers
0
votes
311
views
asked 5 months ago
Hi, Am new to aws services. Unfortunately my project manager gave me a task to do the below, can any on please give me a steps and clear understandings. **Deploy a three tier web application using C...
3
answers
0
votes
278
views
profile picture
asked 5 months ago
Hi Team, I am trying to run SQL statements in Redshift by triggering using EventBridge on S3 file arrival. I am able to run SQLs using Data API but how ever I want to pass the event details to redshi...
1
answers
0
votes
931
views
asked 6 months ago
In AWS Glue jobs, within the Targets node, I am unable to see the data types such as struct, array or map while changing the schema. Does AWS Glue not support these data types?
1
answers
0
votes
498
views
asked 7 months ago
I have the the data of the access logs of the users in opensearch index, this data track the information of the users like at what time users access the door and got the arrival, but one user can swip...
0
answers
0
votes
377
views
asked 7 months ago
Hi team, Our organization is embarking on a project to design a shared data model where we can efficiently store and manage a large payload. Multiple lines of business will be both **updating this ...
1
answers
0
votes
443
views
asked 8 months ago
I have a MySQL database in AWS RDS. I want to import bulk data in to table in database. which contains thousands of rows. I need to do it from my website with also deployed in AWS. Web is developed us...
1
answers
0
votes
426
views
asked a year ago
I am building a data pipeline to Load data into Redshift from an S3 data lake. Data are stored in Parquet format on S3 and I would like to load them into the respective Redshift tables using an AWS ...
1
answers
0
votes
929
views
asked a year ago
I want to add Confluent Cloud Apache Kafka as a Data source in AWS ETL job to read data stream from Kafka topic. I created a cluster, topic, AWS SQS source connector and AWS S3 sink connector in Con...
1
answers
0
votes
643
views
asked a year ago