Best way to re-stream data to Neptune Stream

Question

We have a Neptune with Stream enabled and we listen to that to add data to other type of databases. We had issue in one of our listeners that we have missed some data and it already expired in the stream.

So what is the best way to add data again to that stream ( we only need specific vertex properties ), Only think I can think of is updating property in gremlin and then set it back to what it was like this :

`g.V().hasLabel('user').property(single, "username", "testUserName_randomCode").iterate();`

`g.V().hasLabel('user').property(single, "username", "testUserName").iterate();`

I don't like this method but I am struggling to find a better solution to re add properties values to Neptune stream.

Accepted Answer

There's currently no other method to add data to the stream - other than updating or deleting/re-inserting the data into the database.  The stream log is modified as part of a transaction.

If you need to make these modifications on a regular basis, you may want to look at pushing the stream to an intermediary Kinesis Stream that would allow you to replay the stream from a particular point-in-time.

As of Neptune engine version 1.2.0.0, we now support changing the stream expiry [1] from 1 to 90 days (whereas before it was statically set to 7 days).  There's likely some tradeoffs to be determined on your end on whether it makes sense to have a longer stream expiry and maintain the stream in Neptune (consuming more capacity within Neptune's cluster volume), or pushing the stream to a Kinesis Stream and maintaining the logs there.

[1] https://docs.aws.amazon.com/neptune/latest/userguide/parameters.html#parameters-db-cluster-parameters-neptune_streams_expiry_days

Best way to re-stream data to Neptune Stream

相关内容