Unable to get access to OpenSearch Dashboards through vpn

1

I have an Open Search service set up with a VPC and I trying to access opensearch dashboards through vpn but i get time out every time. I don't have anymore ideas as to why it is not connecting. I tested my vpc domain through sshing into the server through proxy with this guide: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/vpc.html#vpc-security and that is the only way i have access to dashboards, but proxy doesn't fit my use case so i don't want to use it. i do have a full config in my cloudformation temaplate that i can provide if needed.

my domain config:

 "OpenSearchDomain": {
            "Type": "AWS::OpenSearchService::Domain",
            "Properties": {
                "DomainName": "stage",
                "EngineVersion": "OpenSearch_1.3",
                "ClusterConfig": {
                    "InstanceCount": 3,
                    "ZoneAwarenessEnabled": true,
                    "ZoneAwarenessConfig": {
                        "AvailabilityZoneCount": 3
                    },
                    "InstanceType": "t3.small.search"
                },
                "EBSOptions": {
                    "EBSEnabled": true,
                    "VolumeSize": "20",
                    "VolumeType": "gp2"
                },
                "EncryptionAtRestOptions": {
                    "Enabled": true
                },
                "OffPeakWindowOptions": {
                    "Enabled": true,
                    "OffPeakWindow": {
                        "WindowStartTime": {
                            "Hours": 0,
                            "Minutes": 0
                        }
                    }
                },
                "SoftwareUpdateOptions": {
                    "AutoSoftwareUpdateEnabled": true
                },
                "AccessPolicies": {
                    "Version": "2012-10-17",
                    "Statement": [
                        {
                            "Effect": "Allow",
                            "Principal": "*",
                            "Action": "es:ESHttp*",
                            "Resource": [
                                {
                                    "Fn::Sub": "arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/stage/*"
                                }
                            ]
                        }
                    ]
                },
                "AdvancedOptions": {
                    "rest.action.multi.allow_explicit_index": false
                },
                "VPCOptions": {
                    "SubnetIds": [
                        {
                            "Ref": "PrivateSubnetA"
                        },
                        {
                            "Ref": "PrivateSubnetB"
                        },
                        {
                            "Ref": "PrivateSubnetC"
                        }
                    ],
                    "SecurityGroupIds": [
                        {
                            "Ref": "SecurityGroup"
                        }
                    ]
                }
            }
        }

VPN conifg:

"SecurityGroupVPNAccess": {
            "Type": "AWS::EC2::SecurityGroupIngress",
            "Properties": {
                "Description": "VPN Access",
                "CidrIp": "10.12.0.0/16",
                "FromPort": 0,
                "GroupId": {
                    "Ref": "SecurityGroup"
                },
                "IpProtocol": "tcp",
                "ToPort": 65535
            },
            "DependsOn": [
                "SecurityGroup"
            ]
        },
2 Answers
1

Can you name resolve OpenSearch Service endpoints via VPN?

If you cannot confirm the IP address by executing the following command, you need to configure DNS settings
"dig OpenSearch Service endpoints"

profile picture
EXPERT
answered a year ago
  • thanks for your answer, so I do have DNS settings enabled for my OSS- DNS hostnames -enabled and DNS resolution-enabled. but i haven't done any custom config for the dns. wondering if site-to-site vpn connection would resolve my issue.

  • Can you run "dig OpenSearch Service endpoints" from your local environment to check IP addresses?

  • this is the result from that command: ; <<>> DiG 9.10.6 <<>> OpenSearch Service endpoints ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5594 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

    ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;OpenSearch. IN A

    ;; Query time: 45 msec ;; SERVER: fe80::3e17:10ff:fe48:e180%6#53(fe80::3e17:10ff:fe48:e180%6) ;; WHEN: Thu Apr 06 08:16:47 BST 2023 ;; MSG SIZE rcvd: 39

    ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43132 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

    ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;Service. IN A

    ;; Query time: 3 msec ;; SERVER: fe80::3e17:10ff:fe48:e180%6#53(fe80::3e17:10ff:fe48:e180%6) ;; WHEN: Thu Apr 06 08:16:47 BST 2023 ;; MSG SIZE rcvd: 36

    ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46244 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

    ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;endpoints. IN A

    ;; Query time: 3 msec ;; SERVER: fe80::3e17:10ff:fe48:e180%6#53(fe80::3e17:10ff:fe48:e180%6) ;; WHEN: Thu Apr 06 08:16:47 BST 2023 ;; MSG SIZE rcvd: 38

  • Sorry, replace "OpenSearch Service endpoints" with "https://vpc-domain-name-identifier.region.es.amazonaws.com"
    The endpoints should be available from the OpenSearch Service screen

  • omg, sorry for that :face_palm_emoji: here are the results for my OSS endpoint: ; <<>> DiG 9.10.6 <<>> https://vpc-stage-ardj6fcbx6owjstjlhajqtsbqa.eu-west-1.es.amazonaws.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 15924 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

    ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;https://vpc-stage-ardj6fcbx6owjstjlhajqtsbqa.eu-west-1.es.amazonaws.com. IN A

    ;; AUTHORITY SECTION: eu-west-1.es.amazonaws.com. 60 IN SOA ns-1550.awsdns-01.co.uk. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 60

    ;; Query time: 69 msec ;; SERVER: fe80::3e17:10ff:fe48:e180%6#53(fe80::3e17:10ff:fe48:e180%6) ;; WHEN: Thu Apr 06 10:04:57 BST 2023 ;; MSG SIZE rcvd: 184

0

I experienced a very similar problem. We have a VPN connection from office to AWS VPC and the browser could not connect to the Opensearch Dashboard via VPN. DNS was not the issue so after doing some packet capturing I noticed that Opensearch was sending close to maximum size TCP packets to our end and the VPN endpoint on AWS side (we are using a separate EC2 instance as the VPN endpoint just for additional info) was telling Opensearch that fragmentation was needed. Opensearch server did not how ever seem to acknowledge the "Fragmentation Needed" packets but kept on trying to send too large packets when doing the TLS handshake.

What fixed the issue for us, was setting MSS clamping on our end of the VPN tunnel to 1400 (or in our case, the default value used by Netgate/pfSense).

For a more detailed explanation for why that works: When your browser initiates connection to the OpenSearch server, it likely sets the maximum segment size (MSS) to 1460 (+20 bytes for IP headers + 20 bytes of TCP headers equals 1500 bytes, the Maximum Transfer Unit (MTU) size of a ethernet link), telling OpenSearch server that it can send payloads as large as 1460. However when going over the VPN connection, additional TCP headers are added to the packet causing the maximum size to exceed the MTU on the VPN interface (which is often also 1500). Setting MSS clamping to 1400 on the VPN router/endpoint at our end causes the initial TCP packet passing through to be modified and the MSS setting in the packet is changed from 1460 to 1400. Now the OpenSearch server will not send payloads over 1400 bytes, leaving room for the additional VPN related TCP headers and avoiding the packets being over the MTU of the VPN link.

macn
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions