Cognito Access Tokens and SAML Attributes

We are using OKTA SAML as an iDP along with Cognito as a SP.
We have groups that are assigned to the users, and these attributes are mapped are part of the Okta SAML config.
The issue is the following - on the Cognito side, we get 2 tokens - id_token and access_token.
These groups appear as part of the decoded id_token as "custom:groups": "[Group1, Group2, Everyone, Group3]", - which is what we want.
Is it possible instead of these groups to appear in the id_token, to be on the access_token?
If that is not possible is there a workaround with some other kind of attributes to appear as part of the claims in access_token?
I am asking this because as per best practices - it is not good to have custom logic for Authorization and use the id_token to call API's.
Best Regards