Gremlin Query to count Edges - MemoryLimitExceededException AWS Neptune

Question

Gremlin Query on Neptune to count no of edges for vertexes with certain label results in MemoryLimitExceededException. Error only occurs a few times i.e. if i run query 10 times, succeeds twice.  
  
Neptune writer is setup with db.r5.xlarge.  
  
Query  
  
g.V().hasLabel('MyVertex').has('myLabel',1234).outE().hasLabel('MY_EDGE_LABEL').count().next()  
There are over 3 million vertices with label MyVertex and 1 million edges matching MY_EDGE_LABEL edge label.  
  
Error:  
  
{  
    "code": "MemoryLimitExceededException",  
    "requestId": "123456-abcd-1234-1234-11aa221122ab",  
    "detailedMessage": "Query cannot be completed due to memory limitations."  
}  
  
My Questions:  
Is there any setting like memory limit for a query on Neptune that can be increased?  
Is there anything in the query that can be adjusted to get same result but consumes less memory.  
  
Edited by: BaluVV on Mar 7, 2021 8:04 PM  
  
Edited by: BaluVV on Mar 7, 2021 8:07 PM  
  
Edited by: BaluVV on Mar 7, 2021 8:07 PM

Answer

Hi BaluVV,  
  
Neptune engine for Gremlin processes solutions in chunks and specifying a limit(1) has the behavior of hinting the engine to process one solution at a time at each query step (node) instead of processing a chunk at a time. Setting a lower chunk size has a proportional effect on latency. You can observe the same indicated using "chunkSize" attribute in the query profile or explain output. If you need more assistance, please reach out through a support ticket.  
  
Thanks.

Answer

Hi,  
is it possible to reach us via a ticket (TT). That way we have access to your instance and we can investigate deeper. For example it would be interesting to know what would be the size of the result approximately.   
  
As an alternative for now I wonder if you can split the query in multiple steps, each of them retrieving a subset of the data.   
  
Gremlin also has support for query profiling but this is only useful when query completes: https://docs.aws.amazon.com/neptune/latest/userguide/gremlin-profile-api.html  
  
--Gabriel

Answer

Adding limit(1) fixed it, not sure how though.   
  
g.V().hasLabel('MyVertex').has('myLabel',1234).outE().hasLabel('MY_EDGE_LABEL').count().limit(1).next()

Gremlin Query to count Edges - MemoryLimitExceededException AWS Neptune

Relevant content