By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Connection issues from AWS instances to an https download website

0

Hello,

I deleted my previous question and re-posted here because it is only a connection problem and has nothing to do with the client certificate required by the web site.

I download files from an Internet NON AWS web site: https://download.finratraqs.org with a python program. This program was working fine for years on AWS EC2 until August 31st, 2023. After that date, it started to fail, giving time out intermittently: it works once and then it hangs many times sending time out.

I have tested the connection from 3 different AWS Linux Instances (CentOS 6, CentOS 7 and RHEL 8.5) using: *curl, wget, telnet and "openssl s_client" *with the same intermittent behavior (works once and then the command hangs many times).

The problem only happens in AWS instances. Running the same commands and python program in my physical machine and/or VBox instances (CentOS 6, CentOS 8 and RHEL 8.5) they all work just fine without a single time out.

To replicate the problem you can execute any or all of the following commands:

To solve the problem I already tried:

  • Change the Instance type from t3 to t2
  • Add an Inbound rule on the security group opening port 443
  • Changing the DNS Servers
  • Testing the connection from different us-east-1 availability zones (1a, 1b and 1c)

Unfortunately, none of the previous solutions worked, and the connection keeps sending intermittently the time out.

Any suggestion to solve this will be appreciated because AWS is the production environment.

Regards,

Isaac

Topics
ComputeNetworking & Content Delivery
Tags
Amazon EC2Networking & Content DeliveryComputeNetwork SecurityLinux
Language
English
Isaac
asked 8 months ago394 views
3 Answers
0
Accepted Answer

The AWS Networking team have reviewed the issue and have the following findings:

  • download.finratraqs.org maps in DNS to two IPs: 198.55.199.4 and 198.55.215.4
  • AWS sees no issues with reaching of the 198.55.199.4 address from any AWS region
  • However, we see an issue with reaching the 198.55.215.4 address.
  • 198.55.215.4 address is not reachable from the following AWS regions: us-east-1, us-east-2, us-west-1, us-west-2, ca-central-1 but it is reachable from all other AWS regions globally.
  • In the case of the tcptraceroute to 198.55.199.4 port 443 the destination responds
  • In the the case of the tcptraceroute to 198.55.215.4 port 443 a hop on the way to the destination, 206.200.252.251, responds but the final destination does not respond.

traceroute -A -T -p443 198.55.215.4 traceroute to 198.55.215.4 (198.55.215.4), 30 hops max, 60 byte packets 1 * * * 2 * * * 3 * * * 4 241.0.12.27 (241.0.12.27) [] 0.218 ms 241.0.12.9 (241.0.12.9) [] 0.192 ms 241.0.12.14 (241.0.12.14) [] 0.196 ms 5 242.2.74.69 (242.2.74.69) [] 6.474 ms 6.456 ms 242.2.75.195 (242.2.75.195) [] 1.187 ms 6 100.91.218.19 (100.91.218.19) [] 6.748 ms 100.91.218.81 (100.91.218.81) [] 6.412 ms 100.91.218.83 (100.91.218.83) [] 6.583 ms 7 100.91.218.71 (100.91.218.71) [] 5.904 ms 100.100.6.99 (100.100.6.99) [] 6.400 ms 100.100.6.11 (100.100.6.11) [] 6.820 ms 8 100.100.76.198 (100.100.76.198) [] 6.524 ms 100.100.84.6 (100.100.84.6) [] 6.806 ms 100.100.80.134 (100.100.80.134) [] 6.725 ms 9 100.100.80.70 (100.100.80.70) [] 38.774 ms 100.100.84.6 (100.100.84.6) [] 6.436 ms 100.100.72.6 (100.100.72.6) [] 6.967 ms 10 100.100.4.46 (100.100.4.46) [] 6.990 ms 100.100.76.69 (100.100.76.69) [] 6.960 ms 100.100.4.58 (100.100.4.58) [] 6.389 ms 11 100.100.4.46 (100.100.4.46) [] 6.944 ms 100.100.4.58 (100.100.4.58) [] 6.357 ms 100.100.4.34 (100.100.4.34) [*] 7.004 ms 12 eqix-ny1.imperva.com (198.32.118.20) [AS10026] 7.001 ms * * 13 * * * 14 206.200.252.251 (206.200.252.251) [AS14495] 9.985 ms 10.010 ms 10.330 ms 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * *

The above clearly indicates that the issue is in the AS14495, Nasdaq network. The Nasdaq team can verify the issue exists by trying to connect to port 443 any of the IP addresses listed under "US EAST" on this page http://ec2-reachability.amazonaws.com/

As a potential workaround for you whilst the Finra/Nasdaq teams are looking into the issue is that you could try modifying the OS HOSTS file on the EC2 instance to only directl requests for download.finratraqs.org to 198.55.199.4

Isaac
answered 7 months ago
0

Hello.

In the Tokyo region, no timeout occurred even when executed.
The region I'm using is ap-northeast-1.
I have confirmed that the timeout occurs on us-east-1.
It's possible that the site you're downloading from rejects your region's IP range.
We recommend using some kind of proxy and routing through the Tokyo Region or another region.
It's also possible that the site you're downloading from has rejected your downloads because you've accessed them too many times, in which case you may need to contact the site's administrator.
By the way, am I correct in understanding that you can access other websites from EC2 without any problems?

curl -k https://download.finratraqs.org
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"/>
<title>403 - Forbidden: Access is denied.</title>
<style type="text/css">
<!--
body{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica, sans-serif;background:#EEEEEE;}
fieldset{padding:0 15px 10px 15px;}
h1{font-size:2.4em;margin:0;color:#FFF;}
h2{font-size:1.7em;margin:0;color:#CC0000;}
h3{font-size:1.2em;margin:10px 0 0 0;color:#000000;}
#header{width:96%;margin:0 0 0 0;padding:6px 2% 6px 2%;font-family:"trebuchet MS", Verdana, sans-serif;color:#FFF;
background-color:#555555;}
#content{margin:0 0 0 2%;position:relative;}
.content-container{background:#FFF;width:96%;margin-top:8px;padding:10px;position:relative;}
-->
</style>
</head>
<body>
<div id="header"><h1>Server Error</h1></div>
<div id="content">
 <div class="content-container"><fieldset>
  <h2>403 - Forbidden: Access is denied.</h2>
  <h3>You do not have permission to view this directory or page using the credentials that you supplied.</h3>
 </fieldset></div>
</div>
</body>
</html>
profile picture
EXPERT
Riku_Kobayashi
answered 8 months ago
profile pictureAWS
EXPERT
Didier_Durand
reviewed 8 months ago
  • Isaac
    8 months ago

    Thanks for your prompt answer, Riku.

    You're right, I can access other websites with no hassle, however the same problem happens accessing the testing environment download-ntf.finratraqs.org

    Did you try more than once connecting to download.finratraqs.org?

    The issue here is that it works intermittently, it can run one, two, even 5 times ok and then failing 10 times (just hanging).

    Regards,

    Isaac

  • Isaac
    8 months ago

    By the way, I forgot to mention that I already contacted tech. support from the website and I was told that when it fails, they do not see the connection attempt.

  • Riku_Kobayashi EXPERT
    8 months ago

    Thank you for your reply. I've executed the curl command about 20 times in the Tokyo region, and it never timed out. It may be a good idea to check whether the site you are downloading from has implemented WAF or IP restrictions.

  • Isaac
    8 months ago

    Hello Riku.

    I think the limits come from AWS. Do you know how to check it?

0

Hi,

I did test it from the us-east-1 region, about 20 times, and there were no issues. Could you try curl -kI https://download.finratraqs.org to see if there are no extra headers added which can explain the timeout? It could be some rate limit/other WAF at the destination side.

profile picture
EXPERT
Dmytro Sirant
answered 8 months ago
  • Isaac
    8 months ago

    Hi Dmytro.

    I ran curl adding the I option and it simply timed out:

    [centos@ip-172-xx-xx-xx-CentOS7 ~]$ curl -kI https://download.finratraqs.org curl: (7) Failed connect to download.finratraqs.org:443; Connection timed out

    It could be a limit, but in AWS side, because I ran curl from my physical machine within a loop 100 times without a single timeout.

    Is there a way to check the connection limits from AWS?

    Regards

  • Isaac
    8 months ago

    Hello again.

    Finally I was able to get an answer from my Instance running curl -kI:

    [centos@ip-172-xx-xx-xx-CentOS7 ~]$ curl -kI https://download.finratraqs.org HTTP/1.1 403 Forbidden Content-Length: 1233 Content-Type: text/html Server: Microsoft-IIS/8.5 X-Powered-By: ASP.NET Date: Tue, 12 Sep 2023 04:50:10 GMT

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content