Api Gateway with AWS_IAM authorizer | 403 Forbidden

I have an OCR service hosted in the below setup
API Gateway -> Lambda

I have enabled AWS_IAM Authorization on API Gateway, and for input data like JSON it's working as expected
However, when I choose to enable binary media types and try to upload Image to the same setup

The request is denied with status code 403 and error message as,
{"message": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details ...[TRUNCATED]"}

I do know that within API Gateway, there is the conversion of Headers and request body for binary inputs, which I believe is the main cause for this issue.
But this must be a standard setup and I must be missing something here.
Appreciate your help.

Update: this is not working only from POSTMAN, It works seamlessly via python requests and AWS4Auth