I have written a lambda function in python. The lambda function calls an AWS OpenSearch cluster using the OpenSearch endpoint and using python request library and awsauth library.
I have the function uses a configured role with the necessary fine grain access control configure. I know this is all working as when I run a test on the lambda function it works perfectly fine and return the results quickly enough for my use case ( Duration: 197.47 ms Billed Duration: 198 ms Memory Size: 128 MB Max Memory Used: 69 MB Init Duration: 460.95 ms). I then have connected this up to a wider workflow which triggers the lambda and this works great get results etc as needed.
However here is the kicker, the code randomly stops working. There has been no change to OpenSearch and no change to lambda or the roles. However it will run regularly over a long period of time sometimes up to a day or so (sometimes shorter). But then the function returns no results and says it has timed out after 3 seconds.
I log into the aws console, open the lambda and run test, This fails with the same timeout. I go into the code add a space or something just to allow me to redeploy. As soon as it redeploys, I run test again and works absolutely fine again. And the whole cycle happens again.
Note: Just to confirm the connectivity and access all works but it intermittently stops, a redeploy fixes it.
What can be causing this? seems like some authorisation is timing out out or something and by redeploying it forces a new validation or something. Any ideas would be appreciated. thanks.
