How to limit access to specific Identity Center groups?

I have multiple groups in my default Identity Center directory. I want members of a specific group manage users of other groups except a couple of administrator-managed groups. Simple use case is to "Allow team leaders manage limited number of teams by adding/removing users" sso-directory does not support (have) any ARNs and does not have any specific Conditions. Global conditions does not allow to filter by resource ARN or, in my case, group_id. Tags cannot be added to Identity Center groups to allow for aws:ResourceTag/... filtering.

Are there any feasible way to resolve my use-case?