Iot Core HttpAction failed with timeout

Question

Hello everyone.

I am looking for advices and maybe assistance. here is my problem .

We have multiple IoT Core message routing rules.  Two of them send http request to our staging en prod env on two different AWS account. Behind those endpoint there is an api gateway with integration rule to kinesis.

Recap :

iot provider -> iot Core -> http call to endpoint -> api gateway -> kinesis.

The problem is that we have some payload that got the error 
```
HttpAction failed to make a request to the specified endpoint. Request timeout to ENDPOINT after 4000 ms after 2 retries.
```
Sometimes on 1 endpoint sometimes on both. 
On the api gateway side I checked the logs and metrics and I have nothing, neither Timeout event nor errors in the integration. Just nothing, no event at all.

It happens maybe one or two times a day over thousands of events.

Is it something that is normal because you know, internet, things happens ? 
Maybe I could make my pipeline more resilient to this kind of event ? change the http destination ? or add replay on error.

Thanks for your help and experience.

Have a nice day.

Answer

Hello,
 
Yes, you are correct. Due to the transitive nature of the Internet network, there could be chances that, at times,  some of the message send to HTTP endpoints get dropped. Although this is very rare, It could occur.

In this case, as shared in the error message, IoT Rule service was unable to send the message to API Gateway as the request got timeout even after 2 retries. Since the request hasn't reached API Gateway, you will not be able to see anything on API gateway metrics or logs.

On such cases, the recommended approach is to use the IoT Rules error action [1] to capture such messages and retry with exponential backoff [2]. Since such errors are transitive, the messages would get through when retried after some time.

[1] https://docs.aws.amazon.com/iot/latest/developerguide/rule-error-handling.html
[2] https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/

Answer

hello thank your for your answer.

I set up a SNS notification to start. 
But in your opinion which will be the best way to re-execute those actions ? I republish to another topic with the same action, or send to a queue or S3 ?

Thank you and have a nice day.

Iot Core HttpAction failed with timeout

Relevant content