- Newest
- Most votes
- Most comments
When working with the Monday.com connection in AWS Glue, you're encountering some common challenges related to data extraction and filtering. Let's address both of your issues:
-
For the Board entity: The default limit of 25 items is a common pagination constraint. To retrieve all items from a board, you'll need to implement pagination in your Glue job. This typically involves making multiple requests, each time using the 'after' parameter to get the next set of items until you've retrieved all of them. You may need to write a custom script to handle this pagination logic, as the default Glue connector might not automatically handle it for you.
-
For the Item entity: The error message you're receiving indicates that the 'ids' filter is mandatory when querying items. This is likely a security measure to prevent overly broad queries. To resolve this, you need to provide the specific item IDs you want to retrieve in your filter. The format for the filter should be something like:
{"ids": ["item_id_1", "item_id_2", "item_id_3"]}
If you don't know the specific item IDs beforehand, you might need to first query the Board entity to get a list of item IDs, and then use those IDs to query the Item entity.
As an alternative approach, you could consider using the Monday.com API directly within a custom Glue job. This would give you more control over the queries and allow you to implement pagination and filtering as needed. You'd need to use the appropriate API endpoints and handle the authentication and data processing yourself, but it would provide more flexibility in retrieving the data you need.
Remember to ensure that your Glue job has the necessary permissions to access both the Monday.com connection and the target S3 bucket where you plan to store the extracted data.
Sources
error o100.pyWriteDynamicFrame using glue job | AWS re:Post
AWS Glue job takes very long time to write a spark dataframe on S3 | AWS re:Post
Access Denied on S3 from AWS Glue | AWS re:Post
