Skip to content

Excluding vertices from neptune stream poller(Connecting Neptune with OpenSearch)

0

When configuring the neptune stream poller, only PropertiesToExclude and DatatypesToExclude can be set. Is it impossible to exclude by vertex label (entity_type)?

asked 3 years ago425 views
1 Answer
1

Filtering by label is not directly exposed in the CloudFormation stack, but you could add this into the Stream Poller Lambda function code. If you look at the NeptuneStreamPollerLambda that the stack creates, there is a file at the path of ./neptune_to_es/neptune_es_gremlin_handler.py where the logic for filtering takes place. Within that file, there is a filter_records() method. You could add the logic there for additional label filters. I'll add this to our backlog to support this natively, but do not have a timeline as to when we'll be able to address this.

AWS
answered 3 years ago
  • Thanks for your answer. I'll check filter records(). As a user, I hope the filtering by label will be added soon :)

  • If I specify in PropertiesToExclude something called "name", then all name properties are excluded from indexing? Is there a way of controlling from which vertex types those names should be excluded? If I have two vertex types (person and book) and both have a name property and I want just person name to be searchable I need to modity that neptune_es_gremlin_handler filter_records method?

  • Yes, you would need to customize the filter_records() function to address this. I would also suggest changing your data model, too. Having a property called name across multiple labeled vertices could have adverse effects to query performance. Though Neptune doesn't provide the ability to define a schema or constraints, it's a good idea to implement property keys such that they are unique across different vertex/edge labels. This reduces the cardinality of each property type and would potentially reduce query latency if you have a query that needs to do some form of range/full scan.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.