How to resolve 400 Bad Request when sending Grafana alert webhooks to AWS DevOps Agent

Problem

You have configured a webhook contact point in Grafana to trigger AWS DevOps Agent investigations when alerts fire. Testing the webhook endpoint with curl from the Grafana server's CLI works successfully, but when Grafana sends the webhook from the UI (either via the "Test" button or when an alert fires), you receive a 400 Bad Request response.

Root Cause

Grafana's webhook contact point sends its own native alert payload structure by default. This payload includes fields like receiver, status, orgId, alerts[], commonLabels, groupKey, etc. — which does not match the schema that the AWS DevOps Agent webhook endpoint expects (eventType, incidentId, action, priority, title, description, timestamp, etc.).

When you use curl, you are manually constructing the correct payload format with proper authentication headers, which is why it succeeds. The Grafana UI sends the default format, causing the 400 error.

Solution

Create a custom notification template in Grafana that transforms the native alert payload into the format DevOps Agent expects, then reference it in your webhook contact point.

Step 1: Create a Custom Notification Template

1. In your Grafana instance, navigate to Alerting → Contact points → Notification templates.
2. Click Add template.
3. Give it a name (e.g., devops-agent-payload).
4. Paste the following template content:

{{ define "devops-agent-payload" }}
{
  "eventType": "incident",
  "incidentId": "{{ (index .Alerts 0).Labels.alertname }}-{{ (index .Alerts 0).Fingerprint }}",
  "action": "{{ if eq .Status "resolved" }}resolved{{ else }}created{{ end }}",
  "priority": "{{ if eq .Status "resolved" }}MEDIUM{{ else }}HIGH{{ end }}",
  "title": "{{ (index .Alerts 0).Labels.alertname }}",
  "description": "{{ (index .Alerts 0).Annotations.summary }}",
  "service": "{{ if (index .Alerts 0).Labels.job }}{{ (index .Alerts 0).Labels.job }}{{ else }}grafana{{ end }}",
  "timestamp": "{{ (index .Alerts 0).StartsAt }}",
  "data": {
    "metadata": {
      {{ range $k, $v := (index .Alerts 0).Labels }}
      "{{ $k }}": "{{ $v }}",
      {{ end }}
      "_source": "grafana"
    }
  }
}
{{ end }}

This template maps Grafana's alert labels, annotations, and status into the payload structure DevOps Agent expects.

Step 2: Configure the Webhook Contact Point

1. Navigate to Alerting → Contact points.

2. Edit your existing webhook contact point or create a new one.

3. Select Webhook as the integration type.

4. Set the URL to your AWS DevOps Agent webhook endpoint (e.g., https://event-ai.<region>.api.aws/webhook/generic/<YOUR_WEBHOOK_ID>).

5. Under Optional Webhook settings, configure authentication headers based on your webhook type:

   For HMAC authentication (generic webhooks):

   • Add custom header: x-amzn-event-timestamp with a dynamic timestamp value
   • Add custom header: x-amzn-event-signature with the HMAC-SHA256 signature

   For Bearer Token authentication (integration-specific webhooks):

   • Add custom header: Authorization with value Bearer <your-token>

6. Set the Custom Payload field to:

   {{ template "devops-agent-payload" . }}
   

7. Click Save contact point.

Step 3: Assign the Contact Point to a Notification Policy

1. Navigate to Alerting → Notification policies.
2. Edit an existing policy or create a new one.
3. Set the contact point to the webhook contact point you configured above.
4. Click Save policy.

Verification

After completing the setup:

1. Use the Test button on the contact point to verify you receive a 200 response.
2. If you receive a 200 but an investigation does not start, check that:
   • Both timestamp and incidentId fields are unique (duplicate messages are deduplicated).
   • The payload is valid JSON.
3. If you still receive a 4xx error, verify your authentication headers are configured correctly by testing with curl first, then matching that configuration in Grafana.

Additional Notes

• Grafana version: This approach requires Grafana 9.0 or later, which supports custom notification templates and the custom payload field on webhook contact points.
• Multiple alerts: The template above uses (index .Alerts 0) which takes only the first alert in a group. If you need to handle grouped alerts individually, consider configuring your notification policy to group by alertname so each alert fires its own webhook.

References

• Connecting Grafana — AWS DevOps Agent User Guide
• Invoking DevOps Agent through Webhook — AWS DevOps Agent User Guide
• Grafana Webhook Notifier Documentation