AWS Step Functions: how to deal with AWS API endpoints which support pagination

0

Hello,

I am using AWS Step Functions with the new AWS SDK integrations. I am wondering how to deal with API endpoints which return only a limit number of items like for instance "rds:describePendingMaintenanceActions" so that following requests using the "NextToken" value are required to get all data. Is there any construct/suggestion available how to model this. I am aware that I can use a Choice state in order to check if "NextToken" is empty or not and to act accordingly (go back to the API request step and repeat the whole process with the NextToken value) but this feels somehow clumsy. Thank you very much for any information.

質問済み 3年前979ビュー
3回答
0
承認された回答

What you suggest is the right way to go at the moment, i.e., run the loop yourself and use a Choice state to check if the NextToken is empty as an exit criteria from the loop. If you need to do it in different places, maybe it is possible to do it with a nested workflow to run the loop.

profile pictureAWS
エキスパート
Uri
回答済み 3年前
profile picture
エキスパート
レビュー済み 3ヶ月前
  • Looping by using the NextToken is currently not working within Step Functions and SDK Integration, since the NextToken only works within the same IAM Session, but Step Functions assumes a new IAM Session/Role at each SDK Integration step/task. So the workaround is to call a Lambda which calls the API and the handles possible NextTokens.

0

Lambda supports up to 15 min execution time as of October 18, 2018. You could process the next tokens in the function itself.

profile pictureAWS
回答済み 3年前
0

We use the same technique: We want to load > 1 million data sets from a database and process it with different lambda functions and do some other stuff with it. Therefore, we use a choice state with a loop within the step functions and have an "iterator" step that is loading new data right after the choice.

Seems like I can not add an image here, I'm sorry :)

I would NOT go the way Rodney Lester suggests to do the looping in Lambda as you do not have retries over certain batches for example.

回答済み 3年前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ