Firehose Splunk HEC - Server error

When encountering a server error from the Splunk HEC node while using Amazon Data Firehose, there are several steps you can take to troubleshoot and resolve the issue:

1. First, verify that your Firehose can access the Splunk platform in your VPC. Ensure that the necessary network configurations and security groups are set up correctly to allow communication between Firehose and your Splunk endpoint.

2. If you're using a load balancer in front of your Splunk HEC nodes, make sure you're using either a Classic Load Balancer or an Application Load Balancer with duration-based sticky sessions enabled. This helps maintain consistent connections between Firehose and the HEC nodes.

3. Check that your Splunk platform meets the required version (6.6.X or later) to ensure compatibility with Firehose.

4. Verify that the HEC token you're using is valid and enabled in your Splunk configuration. Double-check that you've entered the correct token in your Firehose settings.

5. Ensure that the data being sent to Splunk is formatted correctly and that the HEC token and input event are configured with a valid index in Splunk.

6. Review the Firehose configuration and increase the value of HECAcknowledgmentTimeoutInSeconds in the Splunk destination configuration of your Firehose stream. This gives Splunk more time to process and acknowledge the incoming data.

7. Similarly, increase the value of DurationInSeconds under RetryOptions in the Splunk destination configuration. This allows Firehose to retry failed deliveries for a longer period.

8. Check the health of your HEC endpoint. Ensure that the Splunk cluster is running properly and that there are no issues with the HEC configuration on the Splunk side.

9. If the issue persists, check if your data appears in the Amazon S3 backup bucket. When uploads to Splunk fail due to server errors, Firehose automatically retries and then backs up the data to S3 if all retries fail.

10. If you're using data transformation with a Lambda function, ensure that your function never returns responses with a payload size exceeding 6 MB, as this can cause issues with data delivery.

By systematically going through these steps, you should be able to identify and resolve the underlying cause of the server error from the HEC node. If the problem continues, you may need to work with your Splunk administrator to further investigate any potential issues on the Splunk side.
Sources
Troubleshooting Splunk - Amazon Data Firehose
Handle data delivery failures - Amazon Data Firehose

I had this issue, same error message. I solved it by having my Splunk admin to tick the "Acknowledgment" checkbox in Splunk for that token. Firehose was keeping waiting for the ack from Splunk when trying to send data, and ended up timing out... (180 s in my case). Hopefully this will solve your case.