How to apply the policyDocument returned from a HTTP request lambda authorizer

I have a request lambda authorizer. So far, to get it to work, I have had to apply permissions (Effect: Allow) to all services I need, to the authorizer lambda, however, I really only want to apply the minimum perms to the authorizer and apply specific perms for the user in the policyDocument returned from the authorizer. ATM the only statement which has any effect is one to allow the lambda to use the api gateway (Execute-api: invoke)

eg: in serverless.yml I apply access to dynamoDB (put, get etc) as an identity policy, and in the returned policyDocument I Deny access (also an identity policy), but I still get my data back from the calling lambda.

This is for a public API via API Gateway

Any ideas?