How do I trace an Application Load Balancer request?

Short description

When an Application Load Balancer manages a request, you can find the trace information in the X-Amzn-Trace-Id header:

X-Amzn-Trace-Id: Root=1-67891233-abcdef012345678912345678

To log the ID, use Apache, Tomcat, NGINX, or Internet Information Service (IIS).

Resolution

Apache, Tomcat, and NGINX

Complete the following steps:

1. Open the configuration file in a text editor.
   Note: The configuration file is /etc/httpd/conf/httpd.conf for Apache, /etc/tomcat9/server.xml for Tomcat, and /etc/nginx/nginx.conf for NGINX.

2. Update the file based on your web server configuration. To apply changes to all existing and new sites, configure server-level changes. To use a different log format for a specific virtual host or application, configure site-level changes.
   Server-level changes:
   For Apache, add the %{X-Amzn-Trace-Id}i option to the LogFormat section:

   LogFormat "%{X-Forwarded-For}i %h %l %u %t \"%r\" %>s %b %D \"%{Referer}i\" \"%{User-Agent}i\" \"%{X-Amzn-Trace-Id}i\"" combined

   For Tomcat, add the %{X-Amzn-Trace-Id}i option to the org.apache.catalina.valves.AccessLogValve class:

   <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
         prefix="localhost_access_log." suffix=".txt"
   
         pattern="%h %l %u %t &quot;%r&quot; %s %b %{X-Amzn-Trace-Id}i" />

   For NGINX, add the $http_x_amzn_trace_id to the log_format section:

   log_format  main  '"$http_x_forwarded_for" $remote_addr - $remote_user [$time_local] "$request" '
       '$status $body_bytes_sent "$http_referer" '
   
       '"$http_user_agent" "$http_x_amzn_trace_id"';

   Site-level changes:
   For Apache, update a specific block:

   <VirtualHost *:80>
       ServerName mysite.com
       LogFormat "%{X-Forwarded-For}i %h %l %u %t \"%r\" %>s %b %D \"%{Referer}i\" \"%{User-Agent}i\" \"%{X-Amzn-Trace-Id}i\"" combined
       CustomLog /var/log/httpd/mysite-access.log combined
   </VirtualHost>

   For Tomcat, update a specific block:

   <Context path="/myapp">
       <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
             prefix="myapp_access_log." suffix=".txt"
             pattern="%h %l %u %t &quot;%r&quot; %s %b %{X-Amzn-Trace-Id}i" />
   </Context>

   For NGINX, update a specific server block:

   server {
       server_name mysite.com;
       log_format site_format '"$http_x_forwarded_for" $remote_addr - $remote_user [$time_local] "$request" '
           '$status $body_bytes_sent "$http_referer" '
           '"$http_user_agent" "$http_x_amzn_trace_id"';
       access_log /var/log/nginx/mysite.log site_format;
   }

3. To apply your changes, run the following command:

   sudo systemctl restart servicename

   Note: Based on your web server configuration, replace servicename with apache, tomcat9, or nginx.

IIS

Complete the following steps:

1. Open the IIS Manager.
2. In the Connections navigation pane, expand the server, and then choose your site name.
3. Double-click Logging.
4. To make updates to new sites, proceed to the next step. To make updates to existing sites, for Format, select W3C.
5. Choose Select Fields, and then choose Add Field.
6. In the dialog box, configure the following settings:
   For Field Name, enter X-Amzn-Trace-Id.
   For Source Type, enter Request Header.
   For Source, enter X-Amzn-Trace-Id.
7. Choose OK, and then choose Apply.

Or, run the following command as a PowerShell administrator:

Add-WebConfigurationProperty -PSPath "MACHINE/WEBROOT/APPHOST" -Filter "system.applicationHost/sites/site[@name='Default Web Site']/logFile/customFields" -Name "." -Value @{logFieldName="X-Amzn-Trace-Id"; sourceName="X-Amzn-Trace-Id"; sourceType="RequestHeader"}

Note: Replace Default Web Site with your site name. To update a new site, replace [@name='Default Web Site'] with siteDefaults. The siteDefaults is a template only for newly created sites. Existing sites that already have a section in applicationHost.config don't inherit changes.

IIS applies the change immediately after you make updates. You don't need to restart your system.

To confirm that your changes worked as expected, run the following command:

Select-String -Path "$env:windir\System32\inetsrv\config\applicationHost.config" -Pattern "X-Amzn-Trace-Id"

You can use the logged ID to troubleshoot issues with your load balancer. For example, identify when the same client sends similar requests within a short time. If you have many layers in your stack, then you can also use the X-Amzn-Trace-Id header to track a request across all the layers.

Related information

Request tracing for your Application Load Balancer

Access logs for your Application Load Balancer