How can I contact AWS regarding observed packet loss to ap-southeast-2 over the internet?

Question

I've observed and am currently observing packet loss to public-facing AWS services, in the region ap-southeast-2, from hosts in both London and Amsterdam. Both IPv4 and IPv6 ping/traceroute reports (using `mtr`) have similar concerning output, which coincides with the reason I was investigating - greatly degraded functionality of services I use, e.g. Lambda, Private ECR.

As far as I have been able to observe, the issue seems to originate within AWS infrastructure.

As of writing:
```
$ mtr --report --report-cycles 10 -4bw s3.dualstack.ap-southeast-2.amazonaws.com
Start: 2023-11-17T01:35:12+0000
HOST:                                                        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ???                                                   100.0    10    0.0   0.0   0.0   0.0   0.0
  2.|-- vl199-ds1-b5-r3b12.lon3.constant.com (45.63.102.193)   0.0%    10    1.6   2.1   1.4   7.5   1.9
  3.|-- 10.69.1.221                                            0.0%    10    1.1   2.3   0.9   9.1   2.5
  4.|-- 10.69.5.21                                             0.0%    10    0.4   3.6   0.4  26.1   7.9
  5.|-- ldn-b13-link.ip.twelve99.net (62.115.58.25)           80.0%    10    0.3   0.3   0.3   0.3   0.0
  6.|-- ldn-bb1-link.ip.twelve99.net (62.115.127.100)          0.0%    10    1.2   1.0   0.8   1.2   0.2
  7.|-- nyk-bb2-link.ip.twelve99.net (62.115.113.20)           0.0%    10   75.2  75.3  75.1  75.5   0.2
  8.|-- sjo-b23-link.ip.twelve99.net (62.115.137.171)          0.0%    10  144.5 144.7 144.5 144.8   0.1
  9.|-- amazon-ic-366611.ip.twelve99-cust.net (80.239.195.53)  0.0%    10  141.9 143.2 141.9 148.1   2.3
 10.|-- 15.230.28.61                                           0.0%    10  141.8 143.5 141.2 150.6   3.0
 11.|-- 54.240.242.59                                          0.0%    10  140.1 142.1 140.1 149.3   3.0
 12.|-- ???                                                   100.0    10    0.0   0.0   0.0   0.0   0.0
 13.|-- 15.230.212.79                                         20.0%    10  275.2 272.0 267.3 275.2   2.4
 14.|-- 15.230.210.138                                        20.0%    10  270.0 270.9 268.4 278.6   3.1
 15.|-- 15.230.211.22                                         10.0%    10  277.8 275.8 271.9 280.5   3.3
 16.|-- ???                                                   100.0    10    0.0   0.0   0.0   0.0   0.0
 17.|-- ???                                                   100.0    10    0.0   0.0   0.0   0.0   0.0
 18.|-- ???                                                   100.0    10    0.0   0.0   0.0   0.0   0.0
 19.|-- ???                                                   100.0    10    0.0   0.0   0.0   0.0   0.0
 20.|-- ???                                                   100.0    10    0.0   0.0   0.0   0.0   0.0
 21.|-- ???                                                   100.0    10    0.0   0.0   0.0   0.0   0.0
 22.|-- s3-ap-southeast-2.amazonaws.com (52.95.135.0)         30.0%    10  269.7 269.9 269.6 270.3   0.2
```

From yesterday, IPv6:
```
Start: 2023-11-16T15:58:33+0000
HOST:                                                                        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ???                                                                    100.0    10    0.0   0.0   0.0   0.0   0.0
  2.|-- vl198-ds1-j2-c35r106-b.sea3.constant.com (2001:19f0:fc01:b::6464:c801)  0.0%    10    0.6   0.6   0.5   0.6   0.0
  3.|-- ethernetet-2-0-6-sr2.ams1.constant.com (2001:19f0:fc00::a48:f1)         0.0%    10    0.9   1.0   0.9   1.3   0.1
  4.|-- ethernetae2-er1.ams1.constant.com (2001:19f0:fc00::a48:409)             0.0%    10    0.7   6.1   0.3  35.1  11.9
  5.|-- xe-1-4-1-0.a00.amstnl08.nl.bb.gin.ntt.net (2001:728:0:5000::b11)        0.0%    10    3.5   1.2   0.4   3.5   1.0
  6.|-- ae-15.r20.amstnl07.nl.bb.gin.ntt.net (2001:728:0:2000::2b1)             0.0%    10    6.7   1.9   1.0   6.7   1.7
  7.|-- ae-15.r20.londen12.uk.bb.gin.ntt.net (2001:728:0:2000::171)             0.0%    10    6.1   6.6   6.0   8.4   0.7
  8.|-- ae-7.r20.nwrknj03.us.bb.gin.ntt.net (2001:418:0:2000::252)              0.0%    10   83.6  84.1  83.1  86.8   1.2
  9.|-- ae-4.r24.sttlwa01.us.bb.gin.ntt.net (2001:418:0:2000::332)             90.0%    10  139.2 139.2 139.2 139.2   0.0
 10.|-- ae-0.a02.sttlwa01.us.bb.gin.ntt.net (2001:418:0:2000::216)              0.0%    10  144.2 148.3 143.9 179.2  10.9
 11.|-- 2001:418:0:5000::dfb                                                    0.0%    10  143.0 145.5 142.9 156.3   5.0
 12.|-- 2620:107:4000:4::52                                                     0.0%    10  144.6 145.4 144.6 149.5   1.5
 13.|-- 2620:107:4000:4::19                                                     0.0%    10  140.9 143.5 140.7 153.8   4.2
 14.|-- 2400:6500:0:2::3                                                       20.0%    10  272.5 273.2 270.3 276.5   1.7
 15.|-- 2400:6500:0:2::10                                                       0.0%    10  274.9 272.6 268.6 274.9   2.0
 16.|-- ???                                                                    100.0    10    0.0   0.0   0.0   0.0   0.0
 17.|-- 2400:6500:0:2::29                                                       0.0%    10  280.2 278.0 273.2 285.8   4.5
 18.|-- 2620:107:4000:b390::f001:b80d                                          20.0%    10  274.2 273.5 272.4 274.2   0.8
 19.|-- 2620:107:4000:b390::f001:b814                                          50.0%    10  274.3 273.2 272.0 274.3   1.0
 20.|-- 2620:107:4000:cfff::f204:6200                                          20.0%    10  274.4 273.6 271.4 274.5   1.3
 21.|-- ???                                                                    100.0    10    0.0   0.0   0.0   0.0   0.0
 22.|-- ???                                                                    100.0    10    0.0   0.0   0.0   0.0   0.0
 23.|-- ???                                                                    100.0    10    0.0   0.0   0.0   0.0   0.0
 24.|-- 2406:daa0:c0ac:509:345f:8396::                                         30.0%    10  274.1 273.8 272.3 274.4   0.7
```

Another test I did yesterday:
```
Start: 2023-11-16T15:57:12+0000
HOST:                                                                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ???                                                                  100.0    10    0.0   0.0   0.0   0.0   0.0
  2.|-- 100.100.100.1                                                         0.0%    10    0.4   0.4   0.4   0.5   0.0
  3.|-- 10.72.1.193                                                           0.0%    10    1.1   1.2   0.9   2.4   0.4
  4.|-- 10.72.4.5                                                             0.0%    10    7.1   1.7   0.5   7.1   1.9
  5.|-- cr2-ams13.ip4.gtt.net (213.200.83.175)                                0.0%    10    0.5   1.1   0.4   5.3   1.5
  6.|-- ae19.cr5-sea2.ip4.gtt.net (89.149.139.222)                            0.0%    10  139.4 139.5 138.9 139.9   0.4
  7.|-- ip4.gtt.net (173.205.51.102)                                          0.0%    10  137.0 137.1 136.6 139.0   0.8
  8.|-- ???                                                                  100.0    10    0.0   0.0   0.0   0.0   0.0
  9.|-- 150.222.214.198                                                       0.0%    10  135.9 135.8 135.8 135.9   0.0
 10.|-- 52.95.52.125                                                          0.0%    10  138.1 140.2 137.8 153.1   5.0
 11.|-- ???                                                                  100.0    10    0.0   0.0   0.0   0.0   0.0
 12.|-- 52.95.36.182                                                         20.0%    10  271.5 272.8 271.1 278.4   2.5
 13.|-- 150.222.112.157                                                      10.0%    10  271.1 271.4 270.5 274.4   1.2
 14.|-- 150.222.112.150                                                      20.0%    10  271.7 271.5 271.1 271.8   0.3
 15.|-- 52.95.36.73                                                          20.0%    10  271.3 271.3 271.1 271.6   0.2
 16.|-- 52.95.38.224                                                         10.0%    10  272.5 274.0 271.3 277.3   2.0
 17.|-- ???                                                                  100.0    10    0.0   0.0   0.0   0.0   0.0
 18.|-- ???                                                                  100.0    10    0.0   0.0   0.0   0.0   0.0
 19.|-- ???                                                                  100.0    10    0.0   0.0   0.0   0.0   0.0
 20.|-- ec2-13-237-29-32.ap-southeast-2.compute.amazonaws.com (13.237.29.32) 20.0%    10  271.1 270.2 265.1 271.4   2.1
```

Accepted Answer

Hello, please open a support case with them, and this will be forwarded to their Network team for a more in-depth investigation. When you open the ticket, you can opt for a live chat, allowing both you and the engineer to address the issue in real-time. They may also request you to run additional commands, which they will provide.

In your support ticket, you could inquire about any changes made on their end, specifying the timestamp when you encountered the packet loss issue. Some months ago, I faced a similar situation, and they confirmed that they had made a change which impacted our production system. As a result, our system did not recover, and a restart was required

Answer

Hi,

There is one way available to everybody: open a ticket via the AWS console for the corresponding service.

Then, if you work for a large AWS account having its dedicated TAM (Technical Account Manager), you can ask this person to leverage her / his contacts inside AWS to get experts on it.

Best,

Didier

How can I contact AWS regarding observed packet loss to ap-southeast-2 over the internet?

Relevant content