By using AWS re:Post, you agree to the Terms of Use
/IoT Core OpenSearch Action Rule & "Mapper parsing exception"/

IoT Core OpenSearch Action Rule & "Mapper parsing exception"

0

Hi

I was following this AWS tutorial but with my own sensor data. I've created the following IoT Core Rule OpenSearch Action:

OpenSearchTopicRule:
    Type: AWS::IoT::TopicRule
    Properties:
      TopicRulePayload:
        Actions:
          - OpenSearch:
              Endpoint: !Join ['', ['https://', !GetAtt OpenSearchServiceDomain.DomainEndpoint]]
              Id: '${newuuid()}'
              Index: sensors
              RoleArn: !GetAtt IoTOSActionRole.Arn
              Type: sensor_data
        Sql: SELECT *, timestamp() as ts FROM 'Greenhouse/+/Sensor/Status'

The IoTOSActionRole has propper es:ESHttpPut permission. But when I try to create an index with following command send from Postman that would match the Type: sensor_data attribute:

curl --location --request PUT 'https://search-iot***-avt***i.eu-west-1.es.amazonaws.com/sensors' \
--header 'Content-Type: application/json' \
--data-raw '{
  "mappings": {
    "sensor_data": {
      "properties": {
        "ts": { "type": "long",
          "copy_to": "datetime"},
        "datetime": {"type": "date",
          "store": true},
        "deviceID": {"type": "text",
          "store": true},
        "humidity": {"type": "integer",
          "store": true},
        "temperature": {"type": "integer",
          "store": true},
        "lux": {"type": "integer",
          "store": true},
        "soil": {"type": "integer",
          "store": true}
}}}'

I receive an error:

{
    "error": {
        "root_cause": [
            {
                "type": "mapper_parsing_exception",
                "reason": "Root mapping definition has unsupported parameters:  [sensor_data : {properties={datetime={store=true, type=date}, temperature={store=true, type=integer}, humidity={store=true, type=integer}, soil={store=true, type=integer}, deviceID={store=true, type=text}, lux={store=true, type=integer}, ts={copy_to=datetime, type=long}}}]"
            }
        ],
        "type": "mapper_parsing_exception",
        "reason": "Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters:  [...]",
        "caused_by": {
            "type": "mapper_parsing_exception",
            "reason": "Root mapping definition has unsupported parameters:  [...}]"
        }
    },
    "status": 400
}

I've tried removing the 'type' "sensor_data" attribute and that allowed me to create an index with that mapping,

{
    "acknowledged": true,
    "shards_acknowledged": true,
    "index": "sensors"
}

and then index pattern in OpenSearch Dashboard, but what happens then is that the IoT Core Rule even though it gets triggered does not result in any data ingestion to the OpenSearch domain. So I guess IoT Core Action tries to send that data with type sensor_databut there's no corresponding type in OS. Additionally, when I open the Discovery tab in OS dashboard I get this notice:

"undefined" is not a configured index pattern ID

Showing the default index pattern: "sensors*" (d97775d0-cc44-11ec-acd4-bf72aa2fd725)

I'm using the OpenSearch 1.2 (latest) version. Sample data:

{
  "deviceID": "Tomatoes",
  "Greenhouse": 1,
  "date": "05-05",
  "time": "09:35:39",
  "timestamp": 1651743339,
  "humidity": 60,
  "temperature": 33.3,
  "lux": 9133.333,
  "soil": 78
}

What PUT call I have to make to create a sensor_data type mapping in OS that would match the type specified in IoT Core OpenSearch Action Rule?

1 Answers
0

Hi Lez. It seems mapping types were dropped in ElasticSearch 7.x, a couple of years after that blog.

https://www.elastic.co/guide/en/elasticsearch/reference/7.0/removal-of-types.html

I think types have never existed in OpenSearch. I think the answer is here: https://opensearch.org/docs/latest/opensearch/rest-api/document-apis/index-document/#response-body-fields

The document’s type. OpenSearch supports only one type, which is _doc.

I wanted to try this out before answering, but ran out of time today. I hope this helps.

answered 16 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions