AWS Grafana Bad Gateway error to AWS Opensearch



We have AWS Grafana set up in our Organization root account with access to AWS Opensearch services. I have created a new aws opensearch in a different account I have added it as a data source to aws grafana (shows up in adding new data sources) But when i go to save and test in the data source details it shows as Opensearch error: Bad Gateway

I have tried adding a index name to the opensearch details of grafana , using basic auth, but always get the same error.

any ideas how to resolve?

asked 2 months ago118 views
1 Answer

The error reported indicates that Amazon Managed Grafana couldn't reach the OpenSearch domain. I would like to mention that in order for AMG workspaces to access datasources in a VPC: > The datasource must be publicly accessible via the internet. > AMG IP addresses must be allow-listed for inbound traffic into their VPC.

However, a feature request is currently open to provide access to private sources in AMG, since product road maps are not shared with Premium Support, I will not be able to confirm if/when this new feature will be available. Nevertheless, to keep abreast of all the latest developments in AMG, you can follow

As a workaround, you can configure a proxy with a publicly accessible endpoint for your OpenSearch domain, and connect to it from AMG. To do so, you can follow the steps below:

Amazon OpenSearch Service / Nginx configuration [1]

1. Launch an EC2 instance in the public subnet of the same VPC as your remote OS domain.

2. Register a DNS name with the public IP address of the EC2 instance as an "A Record".

3. Install and configure your NGNIX proxy server. 
a. If you launched an Amazon Linux AMI:
$ sudo yum install nginx

b. For instances launched with an Amazon Linux 2 AMI:
$ sudo amazon-linux-extras install nginx1

c. Configure Nginx:
Note: The following example is meant to be used as a starting point, and adjustments might be necessary to reflect your use case and requirements.

Under the http section in the file /etc/nginx/nginx.conf, add the following:

	resolver ipv6=off;

This parameter represents your VPC DNS resolver and might change according to your VPC settings.

Create a configuration file (for example, /etc/nginx/conf.d/os-proxy.conf) with the following content:

server {
    listen 443 ssl;
    server_name <AmazonOS_External_DNS>;		# Replace by the DNS name created on step 2.
    rewrite ^/$ https://$server_name/_plugin/dashboards redirect;

    ssl_certificate /etc/nginx/cert.crt;		# Replace by your certificate
    ssl_certificate_key /etc/nginx/cert.key;	# Replace by your certificate private key

    ssl_session_cache builtin:1000 shared:SSL:10m;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
    ssl_prefer_server_ciphers on;

    set $os_endpoint <AmazonOpenSearchService_Endpoint>;
    set $cognito_endpoint <Cognito_Host>;

    location /_plugin/dashboards {
        # Forward requests to OpenSearch Dashboards
        proxy_pass https://$os_endpoint;

        # Handle redirects to Amazon Cognito - comment/remove if you are not using Cognito authentication
        proxy_redirect https://$cognito_endpoint https://$server_name;

        # Update cookie domain and path
        proxy_cookie_domain $os_endpoint $server_name;

        # Response buffer settings
        proxy_buffer_size 128k;
        proxy_buffers 4 256k;
        proxy_busy_buffers_size 256k;

    # In the case you are not using Cognito authentication, comment/remove the following section
    location ~ \/(log|sign|error|fav|forgot|change|confirm) {
        # Forward requests to Cognito
        proxy_pass https://$cognito_endpoint;

        # Handle redirects to OpenSearch Dashboards
        proxy_redirect https://$os_endpoint https://$server_name;

        # Handle redirects to Amazon Cognito
        proxy_redirect https://$cognito_endpoint https://$server_name;

        # Update cookie domain
        proxy_cookie_domain $cognito_endpoint $server_name;

d. Enable and start Nginx service:

$ sudo systemctl enable --now nginx

4. After starting the NGINX service, your domain can be accessed via OpenSearch Dashboards through https://<AmazonOS_External_DNS>.

Amazon Managed Grafana configuration [2]

1. Login to your Amazon Managed Grafana workspace and, in the left navigation bar, select the AWS icon.

2. In the 'AWS services' tab, select 'Amazon OpenSearch Service'.

3. Under 'Regions', select the corresponding region and choose the AOS domain you want to add as a data source. Click on 'Add data source'.

4. Under 'Provisioned data sources', click on 'Go to settings'.

5. Update the datasource with the following information:

	* HTTP:
		- For 'URL', add the public DNS name created before, in the format https://<AmazonOS_External_DNS>.
		- For 'Access', select 'Server (default)'.
	* Auth:
		- Select 'Basic auth', and 'Skip TLS verify'.
		- Unselect 'SigV4 auth'.
	* Basic Auth Details:
		- Add the credentials for an OpenSearch user.
	* OpenSearch details:
		- Add the index name on 'Index name', and its corresponding 'Time field name'.

Click on 'Save & test'.


[1] How do I use an NGINX proxy to access OpenSearch Dashboards from outside a VPC that's using Amazon Cognito authentication?

[2] Use AWS data source configuration to add Amazon OpenSearch Service as a data source

answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions