Cloudfront adding extra latency for API Gateway

When calling the regional API Gateway Endpoints, response time is ~300ms, (same with custom domain name).

~300ms for https://xxxxxx.execute-api.us-west-2.amazonaws.com/stage/api/users ~300ms with custom domain name https://api.myexample.com/api/users Our webapp gets served from Cloudfront + S3 at https://myexample.com

Now to avoid CORS, I added another origin (API Gateway) in Cloudfront with behavior of /api/* to point to API Gateway

~800ms for https://myexample.com/api/users -> https://xxxxxx.execute-api.us-west-2.amazonaws.com/stage/api/users Basically Cloudfront is adding those extra 500ms, is there a way to optimize this?

I found similar question, but that was unanswered: https://stackoverflow.com/questions/52999581/api-gateway-edge-optimized-added-latency