AWS Appflow dynamic filter

0

Hi Community, I am new to AWS Appflow. I am trying to extract SAP table data using AppFlow. I am able to create a connection. And then data was successfully extracted. However, this table has a watermark column using which we need to extract incremental records. Hence it is important that we set the filter dynamically i.e. on a daily basis it would fetch last 3 days data. Is it possible to make the filter dynamic? Or Should I call the AppFlow from a Glue job with passing the dates in the filter option as parameters?

bigbang
質問済み 10ヶ月前850ビュー
1回答
0

Hi Bigbang

AWS AppFlow has Incremental transfers, which seem to be what you are looking for: see https://docs.aws.amazon.com/appflow/latest/userguide/flow-triggers.html

When you select incremental transfer, Amazon AppFlow transfers only the records that 
have been added or changed since the last successful flow run. You can also select a 
source timestamp field to specify how Amazon AppFlow identifies new or changed 
records. For example, if you have a Created Date timestamp field, choose this to instruct 
Amazon AppFlow to transfer only newly-created records (and not changed records) since the
 last successful flow run. The first schedule-triggered flow will pull 30 days of past records 
at the time of the first flow run.

Same page details the Offset option

Optionally, you can add a time offset (t) to the time range for the incremental transfer. 
The flow run will import records that were created or changed between the previous flow 
run and the specified offset prior to the current flow run. This feature can be used to accommodate 
any latencies in the source systems in timestamping changes to records. By choosing a 
sufficiently large offset, you can avoid missing records that changed in the source application close 
to the run time of the scheduled flow.

If a schedule-triggered flow runs at time instances T0, T1, T2, and so on, then records that are new 
or have changed between T0 minus t and T1 minus t will be imported from the source at T1, and 
those that have changed between T1 minus t and T2 minus t will be imported from the source at T2.

So, by combining both, you should be able to achieve what you want.

Best, Didier

profile pictureAWS
エキスパート
回答済み 10ヶ月前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ