How to drop specific endpoints from CloudWatch Application Signals?

To effectively drop specific endpoints and reduce costs in CloudWatch Application Signals, you need to use the specific Sampling Rules syntax designed for the AWS Distro for OpenTelemetry (ADOT) collector. The generic "drop" action is often not recognized in the application_signals block; instead, the service relies on sampling logic to filter telemetry.

1. Updated CloudWatch Agent Configuration

Update your sidecar configuration to use sampling_rules. This tells the ADOT collector to ignore these paths before they are processed by the Application Signals managed service:

{
  "traces": {
    "traces_collected": {
      "application_signals": {
        "sampling_rules": [
          {
            "rule_name": "ExcludeHealthChecks",
            "selectors": [
              {
                "dimension": "Operation",
                "match": "GET /actuator/health/*"
              }
            ],
            "should_sample": false
          }
        ]
      }
    }
  }
}

• Wildcards: The * wildcard is fully supported for the Operation dimension.
• Cost Impact: By setting should_sample: false, these requests are not counted towards your Application Signals managed service charges because the telemetry is discarded at the agent level.

2. Application-Level Exclusion (Java Agent)

For optimal performance on ECS Fargate, you should also prevent the Java instrumentation from generating these spans in the first place. This reduces CPU overhead in your application container. Add this environment variable to your ECS Task Definition:

• Key: OTEL_JAVAAGENT_EXCLUDED_URLS
• Value: /actuator/health/*

Answers to your specific questions:

• Is this the correct way? Yes, using sampling_rules within the application_signals block is the standard approach for ADOT-based monitoring.
• Traces vs. Logs? Focus on Traces. Application Signals dashboards and metrics are primarily derived from trace data. Filtering traces is the key to cost control.
• How long does it take? Changes take effect immediately upon restarting your ECS Tasks.
• Best Practice: Always use the OTEL_JAVAAGENT_EXCLUDED_URLS variable alongside the Agent's sampling rules for a "defense-in-depth" approach to telemetry filtering.

Based on the available documentation for CloudWatch Application Signals, I can provide guidance on filtering telemetry data, though the specific drop rules configuration you're attempting may not be the standard approach.

Recommended Approaches for Filtering Endpoints

1. Disable Metrics Exporters Selectively

You can control what telemetry is sent by configuring environment variables in your application container. For your Java application on ECS Fargate, you're already using these key variables:

• OTEL_AWS_APPLICATION_SIGNALS_ENABLED - Controls whether Application Signals telemetry is sent
• OTEL_METRICS_EXPORTER - Set to none to disable other metrics exporters
• OTEL_LOGS_EXPORTER - Set to none to disable other logs exporters

2. Application-Level Filtering

The most effective approach for filtering health check endpoints is typically at the application or OpenTelemetry instrumentation level, rather than at the CloudWatch agent level. You can:

• Configure the OpenTelemetry Java agent to exclude specific endpoints from instrumentation
• Use OpenTelemetry's sampling configuration to drop traces for specific patterns
• Implement custom span processors that filter out health check endpoints before they're exported

3. Endpoint Configuration

Ensure your metrics and traces are being sent to the correct endpoints:

• OTEL_AWS_APPLICATION_SIGNALS_EXPORTER_ENDPOINT should be set to http://localhost:4316/v1/metrics for the sidecar configuration
• OTEL_EXPORTER_OTLP_TRACES_ENDPOINT should be set to http://localhost:4316/v1/traces

Important Considerations

Protocol Setting: Make sure you have OTEL_EXPORTER_OTLP_PROTOCOL set to http/protobuf to properly send metrics and traces to Application Signals.

Version Requirements: Ensure you're using version 1.32.2 or later of the AWS Distro for OpenTelemetry auto-instrumentation agent for Java, as this is required for proper Application Signals functionality.

Alternative Approach

Since the drop rules in the CloudWatch agent configuration may not be the standard method for Application Signals, consider using OpenTelemetry's native filtering capabilities through environment variables or configuration files specific to the Java auto-instrumentation agent. This would allow you to filter traces before they even reach the CloudWatch agent, reducing both network overhead and costs more effectively.

Unfortunately, the specific syntax for drop rules within the Application Signals configuration isn't clearly documented in the standard setup guides, which suggests this may not be the intended filtering mechanism for this service.
Sources
Deploy using the daemon strategy - Amazon CloudWatch
Deploy using the sidecar strategy - Amazon CloudWatch