Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

SES click tracking with custom domain suddenly stopped working

0

We set up SES click tracking with a custom domain as follows:

  • We set up a subdomain click.chordify.net whose DNS points to our production nginx at chordify.net
  • Our production nginix has a proxy_pass directive to reverse-proxy all requests from users to click.chordify.net to https://r.us-east-1.awstrack.me
    • including the setting proxy_set_header Host $http_host;
  • This worked for a couple of days, but since last night every request that nginx performs on behalf of a user to the above mentioned awstrack.me results in a 403 and a response body is plain text "User is not authorized to perform this action"

The weird thing is that it used to work for a couple of days. Also, when I manually rewrite a click tracking URL in the same way that nginx does, I can query it from my machine.

The following does not work. It is a link from a test email sent with a configuration set that uses click.chordify.net as custom domain for open and click tracking.

$ curl -v 'https://click.chordify.net/CL0/https:%2F%2Fchordify.net%2Fpremium/1/010REDACTED6c3a-000000/tREDACTEDY=366'

< HTTP/2 403
< server: nginx/1.14.0 (Ubuntu)
< date: Mon, 19 Aug 2024 15:16:17 GMT
< content-type: application/json
< content-length: 45
< x-amzn-requestid: a06REDACTED091975
<
* Connection #0 to host click.chordify.net left intact
User is not authorized to perform this action

But the same link, with just the domain rewritten and an added Host header works just fine. awstrack.me responds with a 302 as it should.

$ curl -v 'https://r.us-east-1.awstrack.me/CL0/https:%2F%2Fchordify.net%2Fpremium/1/0REDACTED000/tREDACTEDY=366' -H 'Host: click.chordify.net'

< HTTP/1.1 302 Found
< Date: Mon, 19 Aug 2024 15:17:43 GMT
< Location: https://chordify.net/premium
< Content-Length: 0
< Connection: keep-alive
<
* Connection #0 to host r.us-east-1.awstrack.me left intact

I installed nginx on my local machine and set up a similar proxy pass, but I can't reproduce the error. My local nginx reverse-proxies requests to awstrack.me just fine, with this configuration:

        location / {
            proxy_pass https://r.us-east-1.awstrack.me;
            proxy_set_header Host click.chordify.net;
        }

It just works, and I see the click event appear on our production system.

$ curl -v 'http://localhost/CL0/https:%2F%2Fchordify.net%2Fpremium/1/01REDACTEDc3a-000000/tREDACTEDbzoY=366'

< HTTP/1.1 302 Found
< Server: nginx/1.26.2
< Date: Mon, 19 Aug 2024 15:27:31 GMT
< Content-Length: 0
< Connection: keep-alive
< Location: https://chordify.net/premium
<
* Connection #0 to host localhost left intact

What could be the reason that when I query the click tracking URL manually it works, but when nginx does it, AWS responds with 403? How come it worked before and suddenly stopped working? I cannot reproduce the error on my local machine. What can I do to debug it?

Topics
Front-End Web & MobileBusiness Applications
Tags
Amazon Simple Email Service
Language
English
asked a year ago201 views
3 Answers
0
Accepted Answer

I figured out what was wrong.

  • nginx only resolves proxy_pass DNS hostnames on startup
  • awstrack.me IP addresses rotate on a regular basis, apparently every couple of days

This means when awstrack IPs change, nginx proxies requests to old IP addresses. Every time we restarted our nginx it would work for a couple of days and then stop working again. The solution is to force nginx to re-resolve the awstrack host regularly. This has to be done in an awkward manner by storing the awstrack.me hostname in a variable, and adding a resolver clause to the nginx config. Here are some posts describing this

answered a year ago
EXPERT
reviewed 10 months ago
0

Hello.

Are there any logs output in the Nginx access log or error log that could lead to troubleshooting?
Also, I'm concerned about the fact that it was functioning for several days.
By the way, does it improve if I restart Nginx?

EXPERT
answered a year ago
0

Yes there are logs. We are using a custom log format:

log_format proxy_log
    '[$time_local] a $remote_addr u $remote_user h "$host$request_uri" r $request '
    's $status b $body_bytes_sent ref "$http_referer" h "$http_host" '
    'ua "$http_user_agent" xfor "$http_x_forwarded_for"'
    ' Proxy: $proxy_host $proxy_port $upstream_addr $upstream_http_host us $upstream_status';

Here is an excerpt of some working requests, to show how the logs should look like. The line from 20:03:53 is the last 200 we got in the night of Sunday, after that it's only 403s. I just noticed that there are lots and lots of 403s even before that. Which means it wasn't even properly working before. Anyway, the 403s are the upstream_status, which means they come from AWS.

[18/Aug/2024:20:03:00 +0200] a ZZZ.ZZZ.ZZZ.ZZZ u - h "click.chordify.net/CI0/REDACTED" r GET /CI0/REDACTED HTTP/2.0 s 200 b 43 ref "-" h "click.chordify.net" ua "Mozilla/5.0 (Linux; Android 14; SM-S901W Build/UP1A.231005.007; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/127.0.6533.66 Mobile Safari/537.36" xfor "-" Proxy: r.us-east-1.awstrack.me 443 34.238.100.36:443 - us 200
[18/Aug/2024:20:03:00 +0200] a ZZZ.ZZZ.ZZZ.ZZZ u - h "click.chordify.net/CI0/REDACTED" r GET /CI0/REDACTED HTTP/2.0 s 499 b 0 ref "-" h "click.chordify.net" ua "Mozilla/5.0" xfor "-" Proxy: r.us-east-1.awstrack.me 443 52.72.15.49:443 - us -
[18/Aug/2024:20:03:05 +0200] a ZZZ.ZZZ.ZZZ.ZZZ u - h "click.chordify.net/CI0/REDACTED" r GET /CI0/REDACTED HTTP/2.0 s 403 b 45 ref "-" h "click.chordify.net" ua "Mozilla/5.0 (Linux; Android 8.0.0; SM-A320FL Build/R16NW; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/125.0.6422.165 Mobile Safari/537.36" xfor "-" Proxy: r.us-east-1.awstrack.me 443 52.71.132.198:443 - us 403
[18/Aug/2024:20:03:18 +0200] a ZZZ.ZZZ.ZZZ.ZZZ u - h "click.chordify.net/CI0/REDACTED" r GET /CI0/REDACTED HTTP/2.0 s 200 b 43 ref "-" h "click.chordify.net" ua "Mozilla/5.0 (Linux; Android 8.0.0; SM-A320FL Build/R16NW; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/125.0.6422.165 Mobile Safari/537.36" xfor "-" Proxy: r.us-east-1.awstrack.me 443 34.238.100.36:443 - us 200
[18/Aug/2024:20:03:19 +0200] a ZZZ.ZZZ.ZZZ.ZZZ u - h "click.chordify.net/CI0/REDACTED" r GET /CI0/REDACTED HTTP/2.0 s 403 b 45 ref "-" h "click.chordify.net" ua "Mozilla/5.0" xfor "-" Proxy: r.us-east-1.awstrack.me 443 52.71.132.198:443 - us 403
[18/Aug/2024:20:03:22 +0200] a ZZZ.ZZZ.ZZZ.ZZZ u - h "click.chordify.net/CI0/REDACTED" r GET /CI0/REDACTED HTTP/1.1 s 200 b 43 ref "-" h "click.chordify.net" ua "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)" xfor "-" Proxy: r.us-east-1.awstrack.me 443 34.238.100.36:443 - us 200
[18/Aug/2024:20:03:26 +0200] a ZZZ.ZZZ.ZZZ.ZZZ u - h "click.chordify.net/CI0/REDACTED" r GET /CI0/REDACTED HTTP/2.0 s 403 b 45 ref "-" h "click.chordify.net" ua "Mozilla/5.0" xfor "-" Proxy: r.us-east-1.awstrack.me 443 52.71.132.198:443 - us 403
[18/Aug/2024:20:03:29 +0200] a ZZZ.ZZZ.ZZZ.ZZZ u - h "click.chordify.net/CL0/REDACTED" r GET /CL0/REDACTED HTTP/2.0 s 302 b 0 ref "-" h "click.chordify.net" ua "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Mobile Safari/537.36" xfor "-" Proxy: r.us-east-1.awstrack.me 443 52.72.15.49:443, 34.238.100.36:443 - us 504, 302
[18/Aug/2024:20:03:43 +0200] a ZZZ.ZZZ.ZZZ.ZZZ u - h "click.chordify.net/CI0/REDACTED" r GET /CI0/REDACTED HTTP/2.0 s 403 b 45 ref "-" h "click.chordify.net" ua "Mozilla/5.0" xfor "-" Proxy: r.us-east-1.awstrack.me 443 52.71.132.198:443 - us 403
[18/Aug/2024:20:03:45 +0200] a ZZZ.ZZZ.ZZZ.ZZZ u - h "click.chordify.net/CI0/REDACTED" r GET /CI0/REDACTED HTTP/2.0 s 499 b 0 ref "-" h "click.chordify.net" ua "Mozilla/5.0" xfor "-" Proxy: r.us-east-1.awstrack.me 443 52.72.15.49:443 - us -
[18/Aug/2024:20:03:53 +0200] a ZZZ.ZZZ.ZZZ.ZZZ u - h "click.chordify.net/CI0/REDACTED" r GET /CI0/REDACTED HTTP/1.1 s 200 b 43 ref "-" h "click.chordify.net" ua "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)" xfor "-" Proxy: r.us-east-1.awstrack.me 443 34.238.100.36:443 - us 200

After 20:03:53 it's only 403s. There are some 499s, not sure where they are coming from, but they don't seem to come from AWS since there is no upstream_status.

[18/Aug/2024:20:06:20 +0200] a ZZZ.ZZZ.ZZZ.ZZZ u - h "click.chordify.net/CL0/redacted" r GET /CL0/redacted HTTP/2.0 s 403 b 45 ref "-" h "click.chordify.net" ua "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 Edg/127.0.0.0" xfor "-" Proxy: r.us-east-1.awstrack.me 443 52.71.132.198:443 - us 403
[18/Aug/2024:20:06:23 +0200] a ZZZ.ZZZ.ZZZ.ZZZ u - h "click.chordify.net/CI0/redacted" r GET /CI0/redacted HTTP/1.1 s 403 b 45 ref "-" h "click.chordify.net" ua "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)" xfor "-" Proxy: r.us-east-1.awstrack.me 443 52.72.15.49:443, 52.71.132.198:443 - us 504, 403
[18/Aug/2024:20:06:27 +0200] a ZZZ.ZZZ.ZZZ.ZZZ u - h "click.chordify.net/CI0/redacted" r GET /CI0/redacted HTTP/2.0 s 499 b 0 ref "-" h "click.chordify.net" ua "Mozilla/5.0" xfor "-" Proxy: r.us-east-1.awstrack.me 443 52.72.15.49:443 - us -
[18/Aug/2024:20:06:31 +0200] a ZZZ.ZZZ.ZZZ.ZZZ u - h "click.chordify.net/CI0/redacted" r GET /CI0/redacted HTTP/1.1 s 499 b 0 ref "-" h "click.chordify.net" ua "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0)" xfor "-" Proxy: r.us-east-1.awstrack.me 443 52.72.15.49:443 - us -
[18/Aug/2024:20:06:34 +0200] a ZZZ.ZZZ.ZZZ.ZZZ u - h "click.chordify.net/CI0/redacted" r GET /CI0/redacted HTTP/2.0 s 499 b 0 ref "-" h "click.chordify.net" ua "Mozilla/5.0" xfor "-" Proxy: r.us-east-1.awstrack.me 443 34.238.100.36:443 - us -
[18/Aug/2024:20:06:42 +0200] a ZZZ.ZZZ.ZZZ.ZZZ u - h "click.chordify.net/CI0/redacted" r GET /CI0/redacted HTTP/1.1 s 403 b 45 ref "-" h "click.chordify.net" ua "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)" xfor "-" Proxy: r.us-east-1.awstrack.me 443 52.71.132.198:443 - us 403
[18/Aug/2024:20:06:45 +0200] a ZZZ.ZZZ.ZZZ.ZZZ u - h "click.chordify.net/CI0/redacted" r GET /CI0/redacted HTTP/1.1 s 403 b 45 ref "-" h "click.chordify.net" ua "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)" xfor "-" Proxy: r.us-east-1.awstrack.me 443 52.71.132.198:443 - us 403
[18/Aug/2024:20:06:46 +0200] a ZZZ.ZZZ.ZZZ.ZZZ u - h "click.chordify.net/CI0/redacted" r GET /CI0/redacted HTTP/2.0 s 499 b 0 ref "-" h "click.chordify.net" ua "Mozilla/5.0" xfor "-" Proxy: r.us-east-1.awstrack.me 443 52.72.15.49:443 - us -
[18/Aug/2024:20:06:51 +0200] a ZZZ.ZZZ.ZZZ.ZZZ u - h "click.chordify.net/CI0/redacted" r GET /CI0/redacted HTTP/1.1 s 403 b 45 ref "-" h "click.chordify.net" ua "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)" xfor "-" Proxy: r.us-east-1.awstrack.me 443 52.71.132.198:443 - us 403
[18/Aug/2024:20:06:51 +0200] a ZZZ.ZZZ.ZZZ.ZZZ u - h "click.chordify.net/CL0/redacted" r GET /CL0/redacted HTTP/2.0 s 403 b 45 ref "-" h "click.chordify.net" ua "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" xfor "-" Proxy: r.us-east-1.awstrack.me 443 52.71.132.198:443 - us 403
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content