404 error after adding load balancer

Based on your description, there are several potential causes for the 404 errors your IoT devices are experiencing between 3AM-6AM after adding a load balancer.

Since you're seeing traffic on the load balancer but nothing reaching your EC2 instance (no logs at all), this suggests the load balancer might not be properly routing traffic to your instance during those specific hours. Here are some possible causes:

1. Availability Zone configuration: If your EC2 instance is in an Availability Zone that is not enabled on the load balancer, it won't receive traffic. Check that the Availability Zone where your EC2 instance resides is properly enabled on the load balancer.

2. Health check failures: Your instance might be failing health checks during those specific hours, causing the load balancer to consider it unhealthy and not route traffic to it. Review your health check configuration and check if there are any scheduled maintenance tasks or high CPU usage during those hours.

3. Security groups or network ACLs: Verify that the security groups associated with both your load balancer and EC2 instance allow traffic on the necessary ports during all hours.

4. Target group configuration: Ensure your instance is properly registered with the target group and that the target group is correctly associated with the load balancer.

5. Service unavailability: The 404 errors could be related to an HTTP 503 Service Unavailable condition if there are no healthy instances available to the load balancer during those hours.

To troubleshoot this issue:

• Check CloudWatch metrics for your load balancer during the problematic hours
• Verify the health status of your target in the target group during those hours
• Review any scheduled tasks or maintenance that might be occurring during that time window
• Consider enabling access logs on your load balancer to get more detailed information about the requests and responses
  Sources
  Troubleshoot a Classic Load Balancer: Response code metrics - Elastic Load Balancing
  Troubleshoot your Network Load Balancer - Elastic Load Balancing

First, Load balancers do not generate HTTP 404 errors directly. These 404 responses are originating from either the backend targets registered with the load balancer or an intermediate proxy server between the client and load balancer. To troubleshoot this issue, review the ELB access logs (stored in S3 if logging is enabled) which will show the complete request details, response codes, and identify which specific backend server processed each request. Please follow this guide to enable access log: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/enable-access-logging.html