dynamo zero etl pipeline configuration examples for templating/extracting nested values

Question

Hi,

I am setting up my pipeline configuration for our dynamodb table's zero-etl integration pipeline with our opensearch domain, but am struggling to find any examples for how to solve the following issue:

Our table has a ListAttribute, which is a list of objects. Each object (sometimes) has a id : value pair.
objects = [
object 1: {id: 123, other_id: 123455, blah blah other key values we don’t need} ,
object 2: {id: 456, other_id: 1355, blah blah other key values we don’t need} ,
]

Whats the best way to configure our dynamo ingestion pipeline configuration so that we not only index on every other attribute in the row, but also create an additional index on a list of all of the concatenations of “id#other_id”, extracted from each of the objects nested in the list?
Am i supposed to add this inside a "processor" section or inside a template_content section? 
There doesn't seem to be thorough documentation for processing this kind of data, so I appreciate the help i can get! Thanks.

Navya T · Answer

template_type and template_content in the pipeline configuration are used to define explicit mapping rules whereas Processors perform an action on your data, such as filtering, transforming, or enriching.

Having said that, if you could you provide how a sample row in your Dynamo DB table looks like at your end and what is your expectation on how the OpenSearch Document  should be indexed for that particular row in the index.

Also, you can also reach out AWS Support in case if you need further assistance.

dynamo zero etl pipeline configuration examples for templating/extracting nested values

関連するコンテンツ