AppFlow to backup SharePointOnline to S3: Incremental load "429 Too Many Requests" restarts sync | How to initial load huge sites

Question

Hi,

I'd like to use AWS AppFlow to backup my Microsoft SharePoint Online files to AWS S3. My SharePoint has a total size of 4TB across ~200 Sharepoint Sites.
It seems like that for each "Microsoft SharePoint Online site", I need to create an own AppFlow flow. I automated this with a Lambda function and now end up with ~200 AWS AppFlow flows, each running scheduled "incremental transfer" once a day.

1) During the first initial load, I observed all Flows in error state:
```
The request failed because the service Source Customconnector, ConnectorLabel: Microsoft SharePoint Online returned the following error: Details: Too Many Requests, StatusCode: 429.
```
**It seems like AppFlow is not considering this error, the next incremental transfer is starting from scratch instead of continuing where it left off.** I moved the scheduled start time to different wall-clock times, but the sync takes very long and cancels each other after some hours again due to the same error.
Did someone else also face this issue, and how to deal with it?

2) I have a few SharePoint Sites each having around 500GB. AppFlow is syncing them since 40 hours now. I read about the AppFlow limit of 48h (https://docs.aws.amazon.com/appflow/latest/userguide/service-quotas.html). I suspect it will fail and then start from scratch instead of continuing (like above).

Does someone have similar requirements? How did you manage it?

Thank you in advance for your valuable guidance!
Max

re:Post Agent · Answer

When using Amazon AppFlow to backup Microsoft SharePoint Online to Amazon S3, you're encountering some common challenges with large-scale data transfers.

For the 429 "Too Many Requests" errors you're experiencing, this is happening because SharePoint Online has API rate limits that are being exceeded when multiple flows run simultaneously. Unfortunately, Amazon AppFlow doesn't currently have built-in retry logic that would allow it to continue from where it left off after encountering these rate limit errors. Instead, as you've observed, it restarts the transfer from the beginning.

To address this issue, you might want to:

1. Stagger your flow schedules more aggressively to reduce concurrent API calls to SharePoint
2. Reduce the number of flows running per day by prioritizing sites that change frequently
3. Consider grouping smaller SharePoint sites into fewer flows to reduce the total number of concurrent connections

Regarding your 500GB SharePoint sites and the 48-hour execution limit - this is indeed a service limitation. For flows that run longer than 48 hours, AppFlow will terminate them, and as you suspect, subsequent runs will start from the beginning rather than continuing where they left off.

For very large SharePoint sites, you might need to:

1. Break down the backup by selecting specific document libraries or folders within each site rather than the entire site
2. Consider using a more specialized backup solution for the largest sites if AppFlow cannot complete the transfers within the time limit

Amazon AppFlow is designed to transfer documents from SharePoint Online to S3, but it does have these limitations when dealing with very large data volumes and API rate limits from the source system.  
**Sources**  
[Amazon AppFlow | SaaS Integrations List | Amazon Web Services](https://aws.amazon.com/appflow/integrations/)  
[No partitioning option possible when connecting AppFlow to Sharepoint? | AWS re:Post](https://repost.aws/questions/QUS8j9TTnVQRmte7A0v67MoQ/no-partitioning-option-possible-when-connecting-appflow-to-sharepoint)  
[Microsoft SharePoint Online connector for Amazon AppFlow - Amazon AppFlow](https://docs.aws.amazon.com/appflow/latest/userguide/connectors-microsoft-sharepoint-online.html)

AppFlow to backup SharePointOnline to S3: Incremental load "429 Too Many Requests" restarts sync | How to initial load huge sites

Add your answer

Relevant content