Only allow specific Roles to invalidate API Gateway Cache

Is it possible to have an API with Auth: NONE and still prevent unauthorized requests from invalidating the API Gateway Cache? I only want to regulate who can invalidate the cache of the API not the normal GET requests without the Cache-control header.

I have tried different approaches. For example i have assigned a policy to some roles that allows them to invalidate the cache.
"Effect": "Allow", "Action": "execute-api:InvalidateCache", ...
But how do I prevent everything else from invalidating it?

I also tried different resource policies. Here one example:

        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "execute-api:Invoke",
            "Resource": "arn:aws:execute-api:region:account-id:api-id/*"
        },
        {
            "Effect": "Deny",
            "Principal": "*",
            "Action": "execute-api:InvalidateCache",
            "Resource": "arn:aws:execute-api:region:account-id:api-id/stage/GET/path",
            "Condition": {
                "StringNotEquals": {
                    "aws:SourceArn": "arn:aws:lambda:region:account-id:function:functionName"
                }
            }
        }

region, account-id, api-id, stage and so on is only a placeholder here for the real values.

If I do not allow execute-api:Invoke and for example only deny InvalidateCache nothing is able to Invoke the API anymore and if I allow Invoke the InvalidateCache policy seems to be ignored.

Is there any way to achieve my desired result?