How do I establish a connection between Amazon Neptune and Amazon EKS?

Thanks for posting this query here. Did you have a check on the AWS Blog shared at: https://aws.amazon.com/blogs/architecture/financial-crime-discovery-using-amazon-eks-and-graph-databases/ which talks about the same?

Accessing Amazon Neptune from Amazon EKS: Setup and Configuration Guide

In this tutorial, you will establish a connection between Amazon Neptune and Amazon EKS, it's crucial to understand that Amazon Neptune DB clusters are confined within an Amazon Virtual Private Cloud (Amazon VPC). Consequently, accessing Neptune requires careful configuration of the VPC to enable connectivity.

Unlike RDS PostgreSQL, Neptune doesn't rely on typical credentials for database access. Instead, it utilizes IAM roles for authentication. Therefore, connecting to Neptune from EKS involves setting up an IAM role with the necessary permissions to access Neptune.

Furthermore, Neptune endpoints are only accessible within the VPC where the cluster resides. This necessitates configuring network settings appropriately to facilitate communication between EKS and Neptune.

Depending on your specific requirements and networking preferences, there are various approaches to configuring the VPCto enable seamless connectivity between Amazon Neptune and Amazon EKS. Each method offers distinct advantages and considerations, ensuring flexibility in designing your database architecture to suit your application's needs.

Prerequisites

Before you begin this tutorial, you need to:

• Install the latest version of kubectl. To check your version, run: kubectl version --short.
• Install the latest version of eksctl. To check your version, run: eksctl info
• Install the latest version of the AWS CLI (v2). To check your version, run: aws --version
• Create a Neptune DB cluster and make sure it is in “available” status and make sure security group pf Neptune DB cluster has allow inbound rule on port 8182 from security group source.
• Get IAM OIDC provider configured on an existing EKS cluster.

Step 1: Set Environment Variables

Before interacting with your Amazon EKS cluster using Helm or other command-line tools, it's essential to define specific environment variables that encapsulate your cluster's details. These variables will be used in subsequent commands, ensuring that they target the correct cluster and resources.

1. First, confirm that you are operating within the correct cluster context. This ensures that any subsequent commands are sent to the intended Kubernetes cluster. You can verify the current context by executing the following command:

kubectl config current-context

1. Define the CLUSTER_NAME environment variable for your EKS cluster. Replace the sample value for cluster region. If you are using your own existing EKS cluster, replace the sample value for name.

export CLUSTER_NAME=$(aws eks describe-cluster --region us-west-2 --name eks-workshop --query "cluster.name" --output text)

To validate the variables have been set properly, run the following commands. Verify the output matches your specific inputs.

echo $CLUSTER_NAME

Step 2: Create IAM Role and Associate With Kubernetes Service Account

In this section, you'll use IAM Roles for service accounts (IRSA) to map your Kubernetes service accounts to AWS IAM roles, thereby enabling fine-grained permission management for your applications running on EKS. Using eksctl, you'll create and associate an AWS IAM Role with a specific Kubernetes service account within your EKS cluster. We will use the AWS managed policy named “NeptuneFullAccess” which allow write and read for all your specified Neptune cluster.. Note that you must have an OpenID Connect (OIDC) endpoint associated with your cluster before you run these commands.

Step 1: Create a service account which associate with AWS managed policy named “NeptuneFullAccess”

eksctl create iamserviceaccount --name eks-neptune-sa --namespace default --cluster $CLUSTER_NAME --attach-policy-arn arn:aws:iam::aws:policy/NeptuneFullAccess --approve --override-existing-serviceaccounts

Upon completion, you should see the following response output:

2024-02-07 01:12:39 [ℹ] created serviceaccount "default/eks-neptune-sa"

Step 2: Ensure the "eksdemo-secretmanager-sa" service account is correctly set up in the "default" namespace in your cluster.

kubectl get sa eks-neptune-sa -o yaml

The expected output should look like this:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::XXXXXXXXXXXXX:role/eksctl-eks-workshop-addon-iamserviceaccount-d-Role1-343432AAD3434
  creationTimestamp: "2024-02-07T01:12:39Z"
  labels:
    app.kubernetes.io/managed-by: eksctl
  name: eks-neptune-sa
  namespace: default
  resourceVersion: "517475323"
  uid: cd6ba2f7-a0f5-40e1-sd343-4081e0042316

Step 3: Deploy a sample pod and check connectivity with Amazon Neptune

apiVersion: v1
kind: Pod
metadata:
  name: pod-util
  namespace: default
spec:
  serviceAccountName: eks-neptune-sa
  containers:
  - name: pod-util
    image: public.ecr.aws/patrickc/troubleshoot-util
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent

kubectl apply -f pod-util.yaml

kubectl exec --stdin --tty pod-util -- /bin/bash
bash-5.1# curl -X POST -d '{"gremlin":"g.V().limit(1)"}' https://db-neptune-1.cluster-XXXXXXXX.us-west-2.neptune.amazonaws.com:8182/gremlin
{"requestId":"","status":{"message":"","code":200,"attributes":{"@type":"g:Map","@value":[]}},"result":{"data":{"@type":"g:List","@value":[]},"meta":{"@type":"g:Map","@value":[]}}}
bash-5.1# exit
exit

Step 3: Validate EKS connectivity after enabling IAM database authentication in Neptune.

By default, IAM database authentication is disabled when you create an Amazon Neptune DB cluster. You can enable IAM database authentication (or disable it again) using the AWS Management Console. Follow the steps in the AWS documentation to enable IAM database authentication in Neptune.

Step 1: Shell into pod-util container which is already in running status.

kubectl get pods

NAME READY STATUS RESTARTS AGE pod-util 1/1 Running 0 50m

kubectl exec --stdin --tty pod-util -- /bin/bash
bash-5.1#pip3 install awscurl
Installing collected packages: idna, configparser, configargparse, charset-normalizer, certifi, requests, awscurl
Successfully installed awscurl-0.32 certifi-2024.2.2 charset-normalizer-3.3.2 configargparse-1.7 configparser-6.0.0 idna-3.6 requests-2.31.0

bash-5.1# awscurl https://db-neptune-1.cluster-XXXXXXXX.us-west-2.neptune.amazonaws.com:8182/status --region us-west-2 --service neptune-db
{"status":"healthy","startTime":"Thu Feb 08 01:22:14 UTC 2024","dbEngineVersion":"1.3.0.0.R1","role":"writer","dfeQueryEngine":"viaQueryHint","gremlin":{"version":"tinkerpop-3.6.4"},"sparql":{"version":"sparql-1.1"},"opencypher":{"version":"Neptune-9.0.20190305-1.0"},"labMode":{"ObjectIndex":"disabled","ReadWriteConflictDetection":"enabled"},"features":{"SlowQueryLogs":"disabled","ResultCache":{"status":"disabled"},"IAMAuthentication":"enabled","Streams":"disabled","AuditLog":"disabled"},"settings":{"clusterQueryTimeoutInMs":"120000","SlowQueryLogsThreshold":"5000"}}