IoT Core OpenSearch Action Rule & "Mapper parsing exception"

0

Hi

I was following this AWS tutorial but with my own sensor data. I've created the following IoT Core Rule OpenSearch Action:

OpenSearchTopicRule:
    Type: AWS::IoT::TopicRule
    Properties:
      TopicRulePayload:
        Actions:
          - OpenSearch:
              Endpoint: !Join ['', ['https://', !GetAtt OpenSearchServiceDomain.DomainEndpoint]]
              Id: '${newuuid()}'
              Index: sensors
              RoleArn: !GetAtt IoTOSActionRole.Arn
              Type: sensor_data
        Sql: SELECT *, timestamp() as ts FROM 'Greenhouse/+/Sensor/Status'

The IoTOSActionRole has propper es:ESHttpPut permission. But when I try to create an index with following command send from Postman that would match the Type: sensor_data attribute:

curl --location --request PUT 'https://search-iot***-avt***i.eu-west-1.es.amazonaws.com/sensors' \
--header 'Content-Type: application/json' \
--data-raw '{
  "mappings": {
    "sensor_data": {
      "properties": {
        "ts": { "type": "long",
          "copy_to": "datetime"},
        "datetime": {"type": "date",
          "store": true},
        "deviceID": {"type": "text",
          "store": true},
        "humidity": {"type": "integer",
          "store": true},
        "temperature": {"type": "integer",
          "store": true},
        "lux": {"type": "integer",
          "store": true},
        "soil": {"type": "integer",
          "store": true}
}}}'

I receive an error:

{
    "error": {
        "root_cause": [
            {
                "type": "mapper_parsing_exception",
                "reason": "Root mapping definition has unsupported parameters:  [sensor_data : {properties={datetime={store=true, type=date}, temperature={store=true, type=integer}, humidity={store=true, type=integer}, soil={store=true, type=integer}, deviceID={store=true, type=text}, lux={store=true, type=integer}, ts={copy_to=datetime, type=long}}}]"
            }
        ],
        "type": "mapper_parsing_exception",
        "reason": "Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters:  [...]",
        "caused_by": {
            "type": "mapper_parsing_exception",
            "reason": "Root mapping definition has unsupported parameters:  [...}]"
        }
    },
    "status": 400
}

I've tried removing the 'type' "sensor_data" attribute and that allowed me to create an index with that mapping,

{
    "acknowledged": true,
    "shards_acknowledged": true,
    "index": "sensors"
}

and then index pattern in OpenSearch Dashboard, but what happens then is that the IoT Core Rule even though it gets triggered does not result in any data ingestion to the OpenSearch domain. So I guess IoT Core Action tries to send that data with type sensor_databut there's no corresponding type in OS. Additionally, when I open the Discovery tab in OS dashboard I get this notice:

"undefined" is not a configured index pattern ID

Showing the default index pattern: "sensors*" (d97775d0-cc44-11ec-acd4-bf72aa2fd725)

I'm using the OpenSearch 1.2 (latest) version. Sample data:

{
  "deviceID": "Tomatoes",
  "Greenhouse": 1,
  "date": "05-05",
  "time": "09:35:39",
  "timestamp": 1651743339,
  "humidity": 60,
  "temperature": 33.3,
  "lux": 9133.333,
  "soil": 78
}

What PUT call I have to make to create a sensor_data type mapping in OS that would match the type specified in IoT Core OpenSearch Action Rule?

Lez
질문됨 2년 전284회 조회
1개 답변
0

Hi Lez. It seems mapping types were dropped in ElasticSearch 7.x, a couple of years after that blog.

https://www.elastic.co/guide/en/elasticsearch/reference/7.0/removal-of-types.html

I think types have never existed in OpenSearch. I think the answer is here: https://opensearch.org/docs/latest/opensearch/rest-api/document-apis/index-document/#response-body-fields

The document’s type. OpenSearch supports only one type, which is _doc.

I wanted to try this out before answering, but ran out of time today. I hope this helps.

profile pictureAWS
전문가
Greg_B
답변함 2년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인