Questions tagged with Amazon Neptune

I am syncing my AWS Neptune nodes in an AWS OpenSearch cluster as per the documentation https://docs.aws.amazon.com/neptune/latest/userguide/full-text-search.html. The name of the OpenSearch index is amazon_neptune. The OpenSearch index type is _doc. Following is the index configuration ``` { "settings": { "number_of_shards": 1, "number_of_replicas": 1, "analysis": { "normalizer": { "useLowercase": { "type": "custom", "filter": "lowercase" } } } }, "mappings": { "properties": { "document_type" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "entity_id" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "entity_type" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "predicates": { "properties": { "content": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above" : 1000, "normalizer": "useLowercase" } } }, "visibilityType": { "type": "keyword" }, "status": { "type": "keyword" }, "type": { "type": "keyword" }, "firstName": { "type": "text", "fields": { "keyword": { "type": "keyword", "normalizer": "useLowercase" } } }, "lastName": { "type": "text", "fields": { "keyword": { "type": "keyword", "normalizer": "useLowercase", "ignore_above" : 1000 } } } } } } } } ``` Using the npm gremlin package, I'm trying to query my documents. Following is the code: ``` 'use strict'; const gremlin = require('gremlin'); exports.handler = async (event, context) => { try { const DriverRemoteConnection = gremlin.driver.DriverRemoteConnection; const Graph = gremlin.structure.Graph; const dc = new DriverRemoteConnection(<neptune_endpoint>,{}); const graph = new Graph(); const dbClient = graph.traversal().withRemote(dc); const res = await dbClient .withSideEffect("Neptune#fts.endpoint",<https_opensearch_endpoint>) .withSideEffect('Neptune#fts.queryType', 'term') .V().has("visibilityType","Neptune#fts PUBLIC") .toList(); console.log('res:', res); } catch(err) { console.error('Failed to query', err); } } ``` But I'm getting the following error ``` Failed to query ResponseError: Server error: {"detailedMessage":"method [POST], host [<https_opensearch_endpoint>], URI [/amazon_neptune/_search?typed_keys=true&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 403 Forbidden]\n{\"Message\":\"User: anonymous is not authorized to perform: es:ESHttpPost\"}","requestId":"23a9e7d7-7dde-465b-bf29-9c59cff12e86","code":"BadRequestException"} (500) ``` I have given the following permission to my lambda ``` Type: AWS::IAM::Policy Properties: PolicyName: <Policy_Name> Roles: - 'Ref': <lambda_role> PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Action: - es:ESHttpGet - es:ESHttpPost - es:ESHttpPut - es:ESHttpDelete Resource: <opensearch_cluster_arn> ``` My OpenSearch cluster as well as Neptune cluster are located inside the same VPC. My lambda is hosted inside the same VPC as well. Please help me in understanding why I'm getting the 403 error when I've given the proper reading permissions to my lambda. Any help would be highly appreciated.

I'm evaluating Neptune after working with Neo4j. AWS Notebooks are analagous to Neo4j Browser. The Graph tab shows a visualization of results. Nodes can be rearranged, colored etc. In Neo4j browser, a node can be selected and all of it neigbors can be unhidden and added to the display. Nodes can be selectively hidden again. Does the Notebook's Graph tab provide for this node neighborhood expansion? If so I have not found it. Thanks.

Neptune can use openCypher.. Neo4j GraphQL? Is this Library in use? https://neo4j.com/product/graphql-library/

I need to implement a custom propagation algorithm that starts from a given node and visits its neighbors and update their properties accordingly based on the previous node properties. Example use case is, a quality flag that changed in a given node from good to bad and I need to make all the subsequent nodes become bad. What is the recommend way of implement this efficiently in Neptune? (I'm trying to avoid to have client code to have this graph traversal one hop at a time)

I'm using [this cloudformation template](https://s3.amazonaws.com/aws-neptune-customer-samples/v2/cloudformation-templates/neptune-base-stack.json) as part of a deployment of an amazon neptune db. However the NAT gateways that it creates are quite expensive. One suggestion that seems appealing is to use NAT instances instead of NAT gateways. My understanding so far is that the NAT gateways allow my db and its associated resources to make outbound network requests while remaining unavailable from the public internet. I would like to preserve that functionality while switching to NAT instances, but I'm not sure how to go about modifying the template to achieve that. Can someone please offer direction on how I can do this?

Hi, **TL;DR: Can anyone give a recommend approach to storing customisable n-level hierarchy trees for grouping and navigating results via a frontend Amplify-powered web app (using DynamoDB or any other database solution that can be mapped to AppSync ideally)?** **Some background** I'm building an multi-tenant IoT analytics solution that takes data from some sensors out in the field, uploads to AWS, processes this data and stores in a DynamoDB table (i.e. a very "standard" setup). I'm planning on adding a web frontend (built using Amplify and an AppSync GraphQL layer) that will allow users to navigate a **customisable, n-level** hierarchy tree of assets, in order to view the sensor data we've collected. Examples of valid hierarchies include: Country -> Site -> Building -> Floor -> Room -> Sensor (6-level) or Site -> Building -> Room -> Sensor (4-level) etc. The important thing here, is that this hierarchy tree can differ per customer, and needs to be customisable on a tenant-by-tenant basis, but we don't need to do any complex analysis or navigation of relationships between hierarchy levels (so, to me, something like Amazon Neptune or another graph database feels a bit overkill, but perhaps I’m wrong). My first thought was to try and build a hierarchical relationship inside of a DynamoDB table, possibly making use of a GSI to provide this, but in all of the examples I’ve seen online, the focus is very much on quick retrieval, but not so quick updating of hierarchy trees – now, whilst it’s unlikely that these tree structures would be updated on a regular basis, it is something we need to be able to support, so the idea of possibly updating ‘000s of rows in DynamoDB every time we want to make a change to the hierarchy tree for a given control area doesn’t seem quite right to me. Hence, my question above. I'm ideally looking for guidance on how to structure a DDB table to best support BOTH optimal retrieval of, and updates to, hierarchy trees in our application, but if DDB isn't the right answer here, then suggestions of alternatives would also be greatly appreciated. Many thanks in advance.

Hi All, While trying to use federated query for FTS, i am getting ``` { "code": "BadRequestException", "detailedMessage": "An IOException happened while fetching data from ES", "requestId": "xxx-xxxx-xxxx-xxxx" } ``` The Query looks very simple- ``` PREFIX neptune-fts: <http://aws.amazon.com/neptune/vocab/v01/services/fts#> SELECT * WHERE { SERVICE neptune-fts:search { neptune-fts:config neptune-fts:endpoint 'Elasticsearch Endpint' . neptune-fts:config neptune-fts:queryType 'match' . neptune-fts:config neptune-fts:field '*' . neptune-fts:config neptune-fts:query 'naruto' . neptune-fts:config neptune-fts:return ?res. } } ```

According to the following documentation for creating a target endpoint using AWS DMS https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Neptune.html The setup looks like as follows: 1. A Neptune DB cluster(without Notebook) 2. A DMS replication instance 3. A source endpoint (S3) 4. A role that has full access to S3, Neptune and RDS 5. A target endpoint (AWS Neptune) <-- This is the point of failure After configuring the target endpoint when I click on "Run test" I see following error: `arn:aws:iam::12xxxxxx2:role/xxxx-dms-role role should be associated with xxx-cluster-xxx-neptune cluster.` Is there something missing? The role is configured with policies that provide full access to all resources(especially Neptune and S3) ```JSON { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:*" ], "Resource": "*" }, { "Effect": "Allow", "Action": "neptune-db:*", "Resource": "*" }, { "Effect": "Allow", "Action": "rds:*", "Resource": "*" } ] } ```

I have data in mongodb and I want to use some of the properties saved in that database to start and maintain a graph database using amazon neptune. I can use aws dms to get the initial batch of data, but how about maintaining and updating the data as it changes in mongodb? Essentially I need to keep the neptune database synced up with the mongodb database because mongodb remains my primary database. As a simple example, consider that mongodb contains, among other things, users' lists of subscribers. I want this info available for graph queries in neptune, but changes to that info are saved in mongodb primiarily so they need to be forwarded to neptune to keep neptune current. Is there a best-practice for doing something like this? The bulk loader and dms options implicitly assume that I'd be inserting large amounts of data at a time, but these may be simple small changes like one user subscribing to another user. What's the best/correct/recommended way to go about this?

Can you provide examples of how %%graph_notebook_vis_options can be used to color different nodes using rdfs:subClassOf or rdf:type or any other custom attribute. Thanks.

Hi I have created a Neptune cluster and when I inspect the certificate associated with SPARQL endpoint it is expired. Below is what I see ``` * Server certificate: * subject: CN=*.ckhctmtgdvww.eu-west-2.neptune.amazonaws.com * start date: Nov 12 00:00:00 2020 GMT * expire date: Dec 11 23:59:59 2021 GMT * issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon * SSL certificate verify result: certificate has expired (10), continuing anyway. > HEAD /sparql HTTP/1.1 > Host: eg-neptune-db-cluster-trial-egdata.cluster-ckhctmtgdvww.eu-west-2.neptune.amazonaws.com:3030 > User-Agent: curl/7.79.1 ``` Anyone any idea?

I am working through the getting started with Neptune ML [notebook](https://aws-neptune-neptunenotebook.notebook.eu-west-2.sagemaker.aws/notebooks/Neptune/04-Machine-Learning/Neptune-ML-00-Getting-Started-with-Neptune-ML-Gremlin.ipynb) and I get an error in running the first cell in "configuring the endpoints": `ERROR:root:An error occurred (403) when calling the HeadObject operation: Forbidden`. I assume that this is some sort of permissioning error but I unable to get to the bottom of it. Has anybody experience something similar?

I see that Neptune is available in the GovCloud but I do not see it in the list of services in scope for FedRAMP. Can you clarify whether Neptune will be in scope for FedRAMP compliance soon?

We use Neo4j AuraDB for our graph database but there we have issues with data upload. So, we decided to move to AWS Neptune using the migration tool. We have 3.7M nodes and 11.2M relations in our database. The DB instance is db.r5.large with 2 CPUs and 16GiB RAM. The same AWS Neptune OpenCypher queries are much slower than AuraDB Cypher queries (about 7-10 times slower). Also, we tried to rewrite the queries to Gremlin and test performance but it is still very slow. We have node and lookup indexes on AuraDB but we can't create them on AWS Neptune as it handles them automatically. Is there any way to reach better performance on AWS Neptune?

I am trying to use Neptune for my SaaS application. My application includes data from multiple users. I want to restrict the user's access and allow a user to access only to vertices/ edges that contain his data. I read AWS Neptune documentation and I couldn't find anything that relates to tenant isolation/access restriction. Is it possible to limit access within the graph vertices/edges?

AWS Neptune has maintenance windows, during which the service could go down, especially the write operations since just one of the nodes handle this. Is this a correct statement? If we have a requirement for high availability, how do we workaround this?

I am trying to update my Neptune cluster from engine version 1.0.5.1.R2 to the latest 1.1.0.0 version, but am unable to do that either via the console (only 1.0.5.1.R2 is in drop down), or using the AWS CLI as suggested in the [release notes](https://docs.aws.amazon.com/neptune/latest/userguide/engine-releases-1.1.0.0.html). The CLI command I am using is like follows: ``` aws neptune modify-db-cluster \ --db-cluster-identifier my-neptune-cluster \ --engine neptune \ --engine-version 1.1.0.0 \ --allow-major-version-upgrade \ --apply-immediately \ --db-cluster-parameter-group-name my-neptune-pg \ --db-instance-parameter-group-name my-neptune-pg ``` This fails with the error: ``` An error occurred (InvalidParameterCombination) when calling the ModifyDBCluster operation: Cannot upgrade neptune from 1.0.5.1 to 1.1.0.0 ``` Here are the parameters that are set on both the cluster and database: ``` $ aws neptune describe-db-cluster-parameters --db-cluster-parameter-group-name my-neptune-pg | jq -r '.Parameters[] | [.ParameterName, .ParameterValue] | @csv' "neptune_autoscaling_config","" "neptune_enable_audit_log","0" "neptune_enforce_ssl","0" "neptune_lab_mode", "neptune_lookup_cache","1" "neptune_ml_endpoint", "neptune_ml_iam_role", "neptune_query_timeout","120000" "neptune_streams","0" $ aws neptune describe-db-parameters --db-parameter-group-name my-neptune-pg | jq -r '.Parameters[] | [.ParameterName, .ParameterValue] | @csv' "neptune_query_timeout","120000" "neptune_result_cache","0" ```

Hi, we are sometimes experiencing issues with big insert queries. We are running Neptune Engine 1.1.0.0 on a db.r5.large instance. Our initial approach is to insert new data which includes masterdata also (possibly new). To ensure the masterdata will not be created twice, we use appropriate coalesce-steps. A resulting query includes ~ 8000 gremlin steps for example. This query takes more than 20 seconds to execute on an empty database. Our second approach was to assume that all master data already existed in the database. Therefore no more coalesce statement is needed to insert our new data. The corresponding query of our first attempt reduces then to ~ 5000 gremlin steps. This query fails with an InternalFailureException. A descriptive reason is not given. To analyze why the query of our second approach fails, we cut it in half which first also fails (InternalFailureException). Next we tried the quarter of the query, which then succeeded. We retried to run again half of the query which than also succeeded. After that we even were able to run the full query. Please notice, that after each try we reset the database to only include our masterdata. We are wondering why our insert statement runs over 20 seconds on an empty database. Are we doing something wrong here? On the other hand we are wondering why our second approach sometimes work and sometimes fails with the InternalFailureException. It seems that this behavior is undeterministic. What can be the reason for this? How can we fix the query, so that it works everytime we retry on our reset database, including our masterdata? You can access the queries here: 1. Initial, full query with COALESCE steps: https://pastebin.com/d4u6edTH 2. Query to drop all data except master data (to execute after execution of query 1): https://pastebin.com/z8iLeumM 3. Our second approach query without COALESCE: https://pastebin.com/71WCC1iM

I have a Python 3 Neptune Notebook. When I try to run a command such as result = g.V().has('code','test').valueMap().toList() print('result is ' + str(result)) I get: **~/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/yarl/_url.py in __new__(cls, val, encoded, strict)** **156 val = urlsplit(str(val))** **157 else:** **--> 158 raise TypeError("Constructor parameter should be str")** **159** **160 if not encoded:** **TypeError: Constructor parameter should be str** This is a new error, these commands worked for the past year or so. This is a similar error I get in python when I use Python 3.8 instead of 3.7 to call Neptune... is it possible for me to change the notebook Python to 3.7? Edited by: ColinWithMothrazzz on Nov 18, 2021 6:09 AM Over the past week or so, in order to run a notebook I have been downgrading my notebook using this command: %%bash aws s3 cp s3://aws-neptune-notebook/graph_notebook.tar.gz /tmp/graph_notebook.tar.gz rm -rf /tmp/graph_notebook tar -zxvf /tmp/graph_notebook.tar.gz -C /tmp /tmp/graph_notebook/install.sh -v="3.0.6" to run the code. Now I am unable to downgrade the notebook, when I try I get this: **WARNING: Skipping NeptuneGraphNotebook as it is not installed.** **Config option `kernel_spec_manager_class` not recognized by `InstallNBExtensionApp`.** **Installing /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/gremlin_syntax/static -> gremlin_syntax** **Removing: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/gremlin_syntax** **Making directory: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/gremlin_syntax/** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/gremlin_syntax/static/__init__.py -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/gremlin_syntax/__init__.py** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/gremlin_syntax/static/main.js -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/gremlin_syntax/main.js** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/gremlin_syntax/static/description.yaml -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/gremlin_syntax/description.yaml** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/gremlin_syntax/static/README.md -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/gremlin_syntax/README.md** **Making directory: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/gremlin_syntax/__pycache__** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/gremlin_syntax/static/__pycache__/__init__.cpython-36.pyc -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/gremlin_syntax/__pycache__/__init__.cpython-36.pyc** **- Validating: OK** **Installing /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/sparql_syntax/static -> sparql_syntax** **Removing: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/sparql_syntax** **Making directory: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/sparql_syntax/** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/sparql_syntax/static/__init__.py -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/sparql_syntax/__init__.py** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/sparql_syntax/static/main.js -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/sparql_syntax/main.js** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/sparql_syntax/static/description.yaml -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/sparql_syntax/description.yaml** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/sparql_syntax/static/README.md -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/sparql_syntax/README.md** **Making directory: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/sparql_syntax/__pycache__** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/sparql_syntax/static/__pycache__/__init__.cpython-36.pyc -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/sparql_syntax/__pycache__/__init__.cpython-36.pyc** **- Validating: OK** **Installing /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/opencypher_syntax/static -> opencypher_syntax** **Removing: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/opencypher_syntax** **Making directory: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/opencypher_syntax/** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/opencypher_syntax/static/__init__.py -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/opencypher_syntax/__init__.py** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/opencypher_syntax/static/main.js -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/opencypher_syntax/main.js** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/opencypher_syntax/static/description.yaml -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/opencypher_syntax/description.yaml** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/opencypher_syntax/static/README.md -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/opencypher_syntax/README.md** **Making directory: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/opencypher_syntax/__pycache__** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/opencypher_syntax/static/__pycache__/__init__.cpython-36.pyc -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/opencypher_syntax/__pycache__/__init__.cpython-36.pyc** **- Validating: OK** **Installing /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/neptune_menu/static -> neptune_menu** **Removing: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/neptune_menu** **Making directory: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/neptune_menu/** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/neptune_menu/static/__init__.py -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/neptune_menu/__init__.py** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/neptune_menu/static/main.js -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/neptune_menu/main.js** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/neptune_menu/static/description.yaml -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/neptune_menu/description.yaml** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/neptune_menu/static/README.md -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/neptune_menu/README.md** **Making directory: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/neptune_menu/__pycache__** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/neptune_menu/static/__pycache__/__init__.cpython-36.pyc -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/neptune_menu/__pycache__/__init__.cpython-36.pyc** **- Validating: OK** **Installing /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/playable_cells/static -> playable_cells** **Removing: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/playable_cells** **Making directory: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/playable_cells/** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/playable_cells/static/playable_cells.css -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/playable_cells/playable_cells.css** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/playable_cells/static/__init__.py -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/playable_cells/__init__.py** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/playable_cells/static/main.js -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/playable_cells/main.js** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/playable_cells/static/description.yaml -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/playable_cells/description.yaml** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/playable_cells/static/README.md -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/playable_cells/README.md** **Making directory: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/playable_cells/__pycache__** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/nbextensions/playable_cells/static/__pycache__/__init__.cpython-36.pyc -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/playable_cells/__pycache__/__init__.cpython-36.pyc** **- Validating: OK** **To initialize this nbextension in the browser every time the notebook (or other app) loads:** **jupyter nbextension enable graph_notebook.nbextensions --py --sys-prefix** **Config option `kernel_spec_manager_class` not recognized by `EnableNBExtensionApp`.** **Enabling notebook extension gremlin_syntax/main...** **- Validating: OK** **Enabling notebook extension sparql_syntax/main...** **- Validating: OK** **Enabling notebook extension opencypher_syntax/main...** **- Validating: OK** **Enabling notebook extension neptune_menu/main...** **- Validating: OK** **Enabling notebook extension playable_cells/main...** **- Validating: OK** **Config option `kernel_spec_manager_class` not recognized by `InstallNBExtensionApp`.** **Installing /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/widgets/nbextension/static -> graph_notebook_widgets** **Out of date: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/graph_notebook_widgets/__init__.py** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/widgets/nbextension/static/__init__.py -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/graph_notebook_widgets/__init__.py** **Up to date: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/graph_notebook_widgets/extension.js** **Up to date: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/graph_notebook_widgets/index.js** **Out of date: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/graph_notebook_widgets/__pycache__/__init__.cpython-36.pyc** **Copying: /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/graph_notebook/widgets/nbextension/static/__pycache__/__init__.cpython-36.pyc -> /home/ec2-user/anaconda3/envs/JupyterSystemEnv/share/jupyter/nbextensions/graph_notebook_widgets/__pycache__/__init__.cpython-36.pyc** **- Validating: OK** **To initialize this nbextension in the browser every time the notebook (or other app) loads:** **jupyter nbextension enable graph_notebook.widgets --py --sys-prefix** **Config option `kernel_spec_manager_class` not recognized by `EnableNBExtensionApp`.** **Enabling notebook extension graph_notebook_widgets/extension...** **- Validating: OK** **/home/ec2-user/anaconda3/envs/JupyterSystemEnv/bin/python: No module named graph_notebook.start_notebook** **DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.** Edited by: ColinWithMothrazzz on Nov 18, 2021 6:31 AM

How to effectively check the existence of an edge? I tried using a query of this type: g.V(srcId).out(edgeName).has(dstId).hasNext() But after checking the performance, I realized that this is not an efficient query (it takes about 0.1-0.3 seconds) while extracting edges takes ~0.02

Hi, We are doing POC with AWS Neptune and based on the gremlin profile execution time our query takes around 10-15 ms to execute. We are using db.r5.8xlarge instance which has 32 vCPUs i.e 64 neptune threads. So our expectation when we run this query concurrently using multiple Go client we should able to get around 4200 queries/sec (1000 ms / 15 qps * 64 threads) but as per the db instance Gremlin request/second metric the number of queries processed is at 950/sec. This value is way too low from the expected values based on profile output of the query. As per the gremlin status curl output during the concurrent run of the query, I see the acceptedQueryCount is always around 150 and runningQueryCount is always 64. By which I assume there is enough job present in the FIFO queue waiting for the threads to process. But in spite of that db metric Gremlin request/sec is consistently pointing at 950/sec. Need Clarification: 1) What are the things accounted for in the gremlin profile execution time calculation? 2) What are the other factors which will affect the Gremlin request/sec metric which is not accounted for in profile execution time calculation? Data Model: R1----R1R2Edge ---> R2 ----R2R3Edge--->R3 (properties: x1, x2, x3) Query: g.withSideEffect('Neptune#typePromotion',false).V("R1:1:99").as('r1').out('R1R2Edge').out('R2R3Edge').not(and(has('x1',within('-1','tcp')),and(has('x2',lte(5432)),has('x3',gte(5432))))).select('r1').id() Thanks

I noticed 1.0.4.1.R2.1 was released recently with a vague description of fixes... "Disabled an optimization for Gremlin conditional insert patterns which can add or append to existing labels and properties" I'm trying to determine if our team has been impacted and whether we need to cleanup any bad data. Could example queries be provided that were affected? Are there ways to detect if Nodes/Edges have been affected by this bug?

Hi all, I have removed all neptune databases and when I was trying to delete 2 parameter groups, I got this error message: Failed to delete default.neptune1: Default DBParameterGroup cannot be deleted: default.neptune1 (Service: AmazonRDS; Status Code: 400; Error Code: InvalidDBParameterGroupState; Request ID: 9c2df930-c3c7-4e93-b2b9-9762cd1ab690; Proxy: null). I can also see in the AWS billing, the neptune is still charging me even though there is 0 neptune database. Can someone tell me how to avoid getting charged by neptune? And how to remove those 2 parameter groups?

getting random error on querying from nodejs lambda: ERROR getaddrinfo EMFILE neptune-server-instance Any advice welcome

Does the AWS Neptune team plan to implement a Geospatial API for Neptune that would allow "nearby" queries using geolocation coordinates latitude and longitude? If not, can you give guidance or an example query? It doesn't need to be perfectly spherical, but at least something that shows two users are next to each other given their latitude and longitude coords, and sorted by ascending distance. Edited by: JasonD1 on Jan 4, 2021 9:41 PM

not a particularly large data set but I receive the following error: \[code]ERROR org.apache.tinkerpop.gremlin.driver.Handler$GremlinResponseHandler - Could not process the response io.netty.handler.codec.http.websocketx.CorruptedWebSocketFrameException: Max frame length of 65536 has been exceeded. at io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder.protocolViolation(WebSocket08FrameDecoder.java:426) at io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder.decode(WebSocket08FrameDecoder.java:286) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518) at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.lang.Thread.run(Thread.java:748) Max frame length of 65536 has been exceeded.\[/code] I can query up to around 20 of them but more of them I get this error (only 31 total) The vertices are all of one type/label - query is g.V().hasLabel('DATALABEL').valueMap(true).toList() the vertices do have a good few properties including set property types of json strings

I want to create a knowledge graph based on a variety of raw documents (essentially technical documentation). Are there any typical patterns or solutions?

The turtle below fails to load with a PARSING_ERROR -- the error is caused by the blank array "\[ ]". This is valid turtle according to the W3C validator -- https://www.w3.org/2015/03/ShExValidata/ @prefix fa: <http://example.com/entities/> . @prefix fb: <http://example.com/ontology/#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . fa:1234567890 a fb:Something ; rdfs:label "some label" ; fb:someArray \[ ] .

Hello, let me share a very simple query which I am getting poor performance on. I'll share the profile and hopefully someone can explain why it is taking over 10 seconds. Additional filters after the .HasLabel() are significantly hurting performance even further (2 to 3 times worse). "gremlin":"g.V().has('__graph','123').hasLabel('MyType').count()" ******************************************************* Neptune Gremlin Profile ******************************************************* Query String ================== g.V().has('__graph','123').has(label,'MyType').count() Original Traversal ================== \[GraphStep(vertex,\[]), HasStep(\[__graph.eq(123), ~label.eq(MyType)]), CountGlobalStep] Optimized Traversal =================== Neptune steps: \[ NeptuneCountGlobalStep { JoinGroupNode { PatternNode\[(?1, <~label>, ?2=<MyType>, <~>) . project ?1 .], {estimatedCardinality=1273722, expectedTotalOutput=1273722, indexTime=0, joinTime=1131, numSearches=1, actualTotalOutput=830889} PatternNode\[(?1, <__graph>, "123", ?) . project ask .], {estimatedCardinality=2854796, indexTime=887, joinTime=8371, numSearches=830889} }, annotations={path=\[Vertex(?1):GraphStep], joinStats=true, optimizationTime=0, maxVarId=3, executionTime=10614} } ] Physical Pipeline ================= NeptuneCountGlobalStep |-- StartOp |-- JoinGroupOp |-- SpoolerOp(1000) |-- DynamicJoinOp(PatternNode\[(?1, <~label>, ?2=<MyType>, <~>) . project ?1 .], {estimatedCardinality=1273722, expectedTotalOutput=1273722}) |-- SpoolerOp(1000) |-- DynamicJoinOp(PatternNode\[(?1, <__graph>, "123", ?) . project ask .], {estimatedCardinality=2854796}) Runtime (ms) ============ Query Execution: 10614.135 Traversal Metrics ================= Step Count Traversers Time (ms) % Dur ------------------------------------------------------------------------------------------------------------- NeptuneCountGlobalStep 1 1 10614.027 100.00 >TOTAL - - 10614.027 - Predicates ========== # of predicates: 189 Results ======= Count: 1 Output: 414752 Index Operations ================ Query execution: # of statement index ops: 830890 # of unique statement index ops: 830890 Duplication ratio: 1.0 # of terms materialized: 0 Edited by: austinmalpede on Jul 7, 2020 2:02 PM Edited by: austinmalpede on Jul 7, 2020 2:03 PM Edited by: austinmalpede on Jul 8, 2020 7:55 AM

Hello, I'm stuck with a problem I try to upload public turtle data in Neptune and I constantly get the same error : {"requestId":"xxxxx","code":"InternalFailureException","detailedMessage":"Failed to start load from the source s3://pubchem-rdf-data/data/gene/pc_gene.ttl"} My data are in a S3 bucket s3://pubchem-rdf-data/data/gene/pc_gene.ttl My Neptune cluster is up and running I've setup a role for Neptune named "PubChem-NeptuneLoaderRole", first, with S3readOnly and now with S3FullAcess (in a desperate move). This role is attached to the Neptune cluster I have setup the S3 endpoint on my VPC, "Full access policy" too Here is the curl I execute curl -X POST \ -H 'Content-Type: application/json' \ https://chimie-rdf-store.cluster-ck0rf6wslxzi.eu-west-3.neptune.amazonaws.com:8182/loader -d { "source" : "s3://pubchem-rdf-data/data/gene/pc_gene.ttl", "format" : "turtle", "iamRoleArn" : "arn:aws:iam::xxxxxxx:role/PubChem-NeptuneLoaderRole", "region" : "eu-west-3", "failOnError" : "TRUE", "mode" : "RESUME", "parallelism" : "HIGH" }' Really, I do not see what I miss / I've done wrong It seems that the loader cannot access the S3 file (error is almost immediate) but i do not get the standard S3 errors Any idea ? Is there any way to get extra information on the error ? Thanks guys

I've just started experimenting with Neptune for the past month, and my usage has been pretty minimal. I've probably only got about a dozen nodes in the database, and I've been doing some queries through the Gremlin Console and through AWS Lambda functions just to familiarize myself with how things work. I'm the only person who has access to this account. I just got my first month's bill, and it says that I've had 797,681 I/O's, which seems impossible! I'm not worried about the $0.16 charge for that this month, but I'm pretty concerned about how this might scale. I've only had one Neptune cluster running for a total of 124.266 hours this month. I'd estimate that as much as 25% of that time I wasn't interacting with the cluster at all. When I was interacting with it, however, I'd estimate I was only submitting about 100 queries in an hour at the very most, and often much less than that. Yet, at 797,681 I/O's in my billing, that works out to an average of 6,419 queries per hour! Am I missing something here? How does Amazon calculate I/O's? I had assumed that a single, multi-step query would count as a single I/O, but might it actually count as 100's of I/O's, assuming that there are multiple steps, and each of those steps may need to touch multiple quads in order to find the desired information? If that's the case, it seems like the cost has the potential to quickly get out of hand. If someone could clarify how I/O's are calculated, I'd appreciate it!

Hello, I have 1 million + vertices in Neptune with the following property: ('__graph','db8db2af-8b0c-49fb-bcf1-c44cae2d2e95') I want to purge all 1 million vertices in the database. I know i'll have to drop the vertices in small chunks, otherwise the query will time out. Each chunk will be responsible for deleting 1000 vertices, which takes about 2 seconds normally. This is what the query looks like for a chunk: g.V().has('__graph','db8db2af-8b0c-49fb-bcf1-c44cae2d2e95').limit(1000).drop() I'm going to show you what the profile looks like after performing the operation twice: gremlin> g.V().has('__graph','db8db2af-8b0c-49fb-bcf1-c44cae2d2e95').limit(1000).drop().profile() ==>Traversal Metrics Step Count Traversers Time (ms) % Dur ============================================================================================================= NeptuneGraphQueryStep(Vertex) 1 1 2020.007 99.99 NeptuneDropStep 0.101 0.01 >TOTAL - - 2020.109 - ----- gremlin> g.V().has('__graph','db8db2af-8b0c-49fb-bcf1-c44cae2d2e95').limit(1000).drop().profile() ==>Traversal Metrics Step Count Traversers Time (ms) % Dur ============================================================================================================= NeptuneGraphQueryStep(Vertex) 1 1 57883.392 100.00 NeptuneDropStep 0.130 0.00 >TOTAL - - 57883.522 - Why is dropping so inconsistent? It typically should take 2000ms, however we are seeing large spikes in time like the second example. A lot of the time the query completely times out after 2 minutes. Please help, Thanks, Austin Edited by: austinmalpede on Apr 3, 2020 10:59 AM

Hi, If we include multiple SPARQL update queries separated by a semi-colon in a single POST message, are those executed as part of one transaction? Or is each query a transaction of it&#39;s own? Thanks, Rakesh

When inserting a triple with a blank node into an Amazon Neptune instance: ``` INSERT DATA { <http://example.com/s> <http://example.com/p> _:something } ``` …the blank node&#39;s value gets replaced with b<SOME NUMBER>. Running a query like this: ``` SELECT ?o WHERE { <http://example.com/s> <http://example.com/p> ?o } ``` …will return something like this: ``` { "results": { "bindings": [ { "o": { "type": "bnode", "value": "b24508943" } } ] } } ``` Is there a way to retain original blank node names when loading data into Neptune?

If I have jobs that require being processed in a certain order, can I rely on Neptune to process them serially in the order I create the jobs? Or does Neptune try to run the jobs in parallel at the same time? Thank you, Mike Edited by: mveilleux on Jan 28, 2020 9:03 AM

>I upgraded a lambda (node.js 10.x) from gremlin 3.2.9 to 3.4.4 and `valueMap()` no longer returns data. values() does as do most other >functions (though cap() with multiple columns does not). The return often appears to have data, e.g. `JSON.stringify(data)` will show a >structure but no data -- the array is blank. I have tried dozens of independent queries (none now work though they do in 3.2.9) as well as >those same queries from the command line (all work!). Any help is greatly appreciated!

Hello, I understand that increasing a writer instance tier will cause downtime, however i'm exploring ways to avoid that. One way that comes to mind is to add a read replica within the cluster which is a higher tier, then to promote it to the writer instance. Can this be achieved without causing downtime? If so, what Neptune API's would be invoked to do it? Thanks

I'm new in Amazon Neptune service and trying to load a simple RDF file from my S3 bucket using Neptune Loader command but it Failed with this error : **( "errorMessage" : "Invalid data type encountered for header <rdf:RDF>",)** any help please ?