Questions tagged with AWS Data Pipeline
Content language: English
Select up to 5 tags to filter
Sort by most recent
Browse through the questions and answers listed below or filter and sort to narrow down your results.
Hi,
I have a use case where I am fetching data on certain items (unique itemID) multiple times a day (identified by day_BatchTime) and storing them on a dyanmoDB. My composite primary key consists...
3
answers
0
votes
560
views
asked 2 years agolg...
Hello
I am working on serverless application, and i was looking for something handle the frontend part and I found Honeycode since its native and its codeless.
So is it possible to my Honeycode app...
1
answers
0
votes
519
views
asked 2 years agolg...
Hi,
I have a problem in that I make heavy use of EMRs, and I orchestrate their use with Data Pipeline - multiple daily runs are automated and EMRs are launched and terminated on conclusion.
However,...
1
answers
0
votes
372
views
asked 2 years agolg...
I'm trying to create a data pipeline to export Dynamodb data to S3, but after following the online guide to the letter, the DataPipelineDefaultResourceRole isn't in the dropdown referred to above, the...
1
answers
1
votes
574
views
asked 2 years agolg...
## Main problem
I understand that is no need to add Auto Scaling to an EMR cluster launched by Data Pipeline. Instead, we can specify the **capacity up-front** and it will be used for the duration of...
0
answers
0
votes
152
views
asked 2 years agolg...
Currently I use the [CreateExportTask](https://docs.aws.amazon.com/ko_kr/AmazonCloudWatchLogs/latest/APIReference/API_CreateExportTask.html) API to backup my log data.
The problem is, exported data...
1
answers
1
votes
1621
views
asked 2 years agolg...
I'm confused about how staging of an S3Datanode is billed when done as part of a ShellCommandActivity with the 'stage' property set to true (i.e. I do not have CSV data and am not using a...
0
answers
0
votes
220
views
asked 2 years agolg...
I have a few questions regarding data preparation for Forecast.
I have a dataset with about 3,000 item_id's, the data is recorded on weekdays only (no row for weekends/holidays), and the forecast...
0
answers
0
votes
113
views
asked 2 years agolg...
What's the best way to filter out duplicated records in a Glue ETL Job with bookmarking enabled?lg...
I have an etl pipeline that loads json data from a source bucket, runs an etl job with bookmarking enabled, and writes as parquet to a target bucket.
I'd like to ensure that the target bucket never...
1
answers
0
votes
5953
views
asked 2 years agolg...
I'm working on a step function state machine and can create lambdas in python and node to update an existing item in ddb. However, I can't seem to find any examples with service integrations AND...
1
answers
0
votes
671
views
asked 2 years agolg...
On the AWS EMR console, we are seeing AWS EMR 6.5.0 version being available.
However, EMR Documentation doesn't have any specific information on 6.5.0.
When will the documentation be updated based on...
1
answers
0
votes
438
views
asked 2 years agolg...
Hello,
Where can I find more details on AWS' approach around data models? This would include industry-specific data models AWS is fully invested in.
1
answers
0
votes
264
views
asked 2 years agolg...