How to automatically apply ILM to an new index created by Firehose

0

We have a Kinesis Firehose sending logs to AWS Elasticsearch and it creates a new index each day.
I created an Index Lifecycle Policy to delete an index after 2 days and that works fine if I apply it via Kibana Index Management manually. What I need to do is apply the Policy to an index when it is created by the firehose.
The Elasticsearch docs shows this can be done with the below template but AWS Kibana does not support the index.lifecycle.name. I get an unknown setting error.
Does anyone know how I can automatically apply an ILM Policy to an Index and I must be able to wildcard it since the Firehose tacks on a date.
Thanks in advance

-------- This is the template I am trying to PUT -----------------
PUT _template/testindex_template
{
"index_patterns": ["test-index*"],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "delete_index_policy"
}
}

------- This is the error -----------------
{
"error": {
"root_cause": [
{
"type": "remote_transport_exception",
"reason": "[b81eac543fece630b97c35b7b84acae3][x.x.x.x:9300][indices:admin/template/put]"
}
],
"type": "illegal_argument_exception",
"reason": "unknown setting [index.lifecycle.name] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
},
"status": 400
}

P.S. I also create a Rollover Lifecycle policy but same issue with that. The Delete policy doesn't get applied to the index that is rolled over.

KevC
질문됨 4년 전768회 조회
3개 답변
0

Figured it out.
Need to us opendistro in the key for the template as follows:
{
"index_patterns": ["test-*"],
"settings": {
"opendistro.index_state_management.policy_id": "zero_replica_and_delete_index"
}
}

KevC
답변함 4년 전
0

See latest reply

KevC
답변함 4년 전
0

Did this work for you? If I do the same, then the templates seems to accept the "opendistro.index_state_management.policy_id" setting, but after a while - if I get the template again through the API - it was removed from the settings.

Is there another way to apply policies automatically to new indices that match a certain pattern?

Regards,
MdB

MdBdVB
답변함 4년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠