Increasing performance of fetching secrets

I'm trying to see if there's anything I can do to improve performance when fetching secrets from Secrets Manager. At the moment, we're seeing it taking somewhere between 100-400ms every time (probably averaging ~200ms)... which is a pretty big overhead considering our lambdas themselves typically take less than that.

We were previously caching the secret in memory (for warm/provisioned lambdas), however, that brings us unstuck when those secrets get changed and the lambdas have out-of-date values.

So, my question is two-fold:

• Is this just "the way it is"? ie. I understand that there's no performance guarantees, but is this just the ballpark that I should expect for these types of fetches?
• Would using something like a VPC Endpoint help? All the doc seems to suggest that the benefits are more about security, rather than performance, but I'm happy to explore.

Thanks.