Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

SAP OData Connector for AWS Glue keeps returning "Next Token to be serialized" — pagination won't stop despite small dataset

0

I'm using the SAP OData Connector for AWS Glue to extract SAP data based on specific condition involving a single field (such as a fiscal period or profit center), the expected record count is relatively small (around 75,000), but the job keeps paginating indefinitely.

Observed behavior: The connector is configured with PAGE_SIZE = 100000. Despite this, the logs show repeated pagination with 50156 records fetched per page. The log repeatedly outputs Next Token to be serialized and Total records fetched not reached given limit. Eventually, records fetched exceeds 350,000, far beyond the expected volume.

Questions:

  1. Why does pagination continue even when the expected record count is small and PAGE_SIZE is set to 100000?
  2. Why is the actual page size in the logs consistently 50156 instead of the configured 100000?
  3. Has anyone encountered persistent pagination when applying filters on a single field (e.g., fiscal period or profit center), where the expected record count is small but pagination continues?
  4. Is there any supported way to control or disable pagination?

Any insights or suggestions would be greatly appreciated.

Topics
AnalyticsDatabase
Tags
AnalyticsExtract Transform & Load DataAWS Glue
Language
English

asked a year ago325 views

1 Answer
1

Based on your description, there are a few key issues to address with your SAP OData connector pagination behavior:

Regarding the page size discrepancy: The AWS Glue SAP OData connector has a default page size of 50,000 records. While you've configured PAGE_SIZE=100000, there's a maximum allowed page size of 500,000. However, it appears the SAP system itself might be enforcing its own limit of approximately 50,156 records per page, which would explain why you're consistently seeing that number in the logs rather than your configured 100,000.

As for the continuous pagination issue: The connector is designed to continue paginating until it has retrieved all records that match your query criteria. The behavior you're seeing suggests that either:

  1. Your filter criteria might not be limiting the dataset as expected
  2. There could be an issue with how the SAP OData service is handling the pagination tokens
  3. The total record count reported by the SAP system might be incorrect

For controlling pagination: Unfortunately, there isn't a direct way to disable pagination in the SAP OData connector as it's a fundamental mechanism for handling large datasets. However, you could try these approaches:

  1. Verify your filter predicates are correctly applied and working as expected
  2. Consider using the DELTA_TOKEN option if you're implementing incremental data loads
  3. For CDC-enabled entities, use the ENABLE_CDC option set to "True" along with DELTA_TOKEN for more efficient data extraction
  4. Examine if your SAP OData service supports the ODP framework, which provides better handling of data extraction through Operational Delta Queues

If you're consistently retrieving more records than expected, I'd recommend checking the filter predicate syntax and confirming with your SAP team that the OData service is correctly implementing the filtering capabilities.
Sources
SAP OData connection options - AWS Glue
Scaling RISE with SAP data and AWS Glue | AWS Big Data Blog

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content