dynamo zero etl pipeline configuration examples for templating/extracting nested values



I am setting up my pipeline configuration for our dynamodb table's zero-etl integration pipeline with our opensearch domain, but am struggling to find any examples for how to solve the following issue:

Our table has a ListAttribute, which is a list of objects. Each object (sometimes) has a id : value pair. objects = [ object 1: {id: 123, other_id: 123455, blah blah other key values we don’t need} , object 2: {id: 456, other_id: 1355, blah blah other key values we don’t need} , ]

Whats the best way to configure our dynamo ingestion pipeline configuration so that we not only index on every other attribute in the row, but also create an additional index on a list of all of the concatenations of “id#other_id”, extracted from each of the objects nested in the list? Am i supposed to add this inside a "processor" section or inside a template_content section? There doesn't seem to be thorough documentation for processing this kind of data, so I appreciate the help i can get! Thanks.

asked 3 months ago207 views
1 Answer

template_type and template_content in the pipeline configuration are used to define explicit mapping rules whereas Processors perform an action on your data, such as filtering, transforming, or enriching.

Having said that, if you could you provide how a sample row in your Dynamo DB table looks like at your end and what is your expectation on how the OpenSearch Document should be indexed for that particular row in the index.

Also, you can also reach out AWS Support in case if you need further assistance.

Navya T
answered 3 months ago
profile picture
reviewed 3 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions