Migrating Elastic version 8.9 to OpenSearch latest

Question

Regarding the migration from Cold to UltraWarm storage, would it be possible to get some information on how performant this migration is (understanding performance characteristics would be crucial to analyze feasibility in achieving our business goals using the suggested architecture)? We tried it in a sandbox environment, and it was instant, but we think it was instant because we migrated from UW to Cold, then back within a short time interval.. so, the cache for the index we tried migrating was not evicted yet! We’d like to get at least high-level estimates (could be relative time comparably with migration from Hot to UltraWarm) or how we could measure that ourselves (is there a way to force cache eviction in the UW storage so that we could measure how long it really takes to migrate an index from COLD to UW?).

On Index State Management operations, the only remaining unclear to me thing is how could we define the “start_time” and “end_time” of the cold index within the ISM policy that would do the transition… I know it can be done if we transition index from UltraWarm to COLD via an API (knowing the start and end times upfront)… but since we are planning to rely on ISM policy to manage workflow of the indices… can we specify that “start_time” and “end_time” of the migrated index should be min(@timestamp) and max(@timestamp)?
Is there a way to adjust state of the index within the ISM policy for migrated indices? Currently, we have a process in place where our indices are kept in a frozen state for 730 days before being automatically trashed by an ISM policy. The policy calculates the 730-day threshold from the creation date of each index. However, with the migration of our data to OpenSearch, particularly through a LogStash migration strategy, all migrated indices will be created at the time of the migration. This raises a concern about what can be done to ensure the ISM policy's deletion schedule.

Answer

## Performance of Cold to UltraWarm Migration:

### Performance Characteristics:
> 🙄 The performance of such a migration can vary significantly based on data volume, network conditions, and hardware performance. Since there is no detailed public benchmark specifically for this migration scenario, it is best to perform a controlled test in your environment.

### High-Level Time Estimates:
> ℹ️ There are no standard time estimates for Cold to UltraWarm migrations as it heavily depends on the specific conditions mentioned above. Generally, moving data to UltraWarm is faster than to Cold because UltraWarm is optimized for faster read access.

### Measuring True Migration Time:
> 💡 You can try to simulate a non-cached scenario by accessing different data that was not recently accessed to ensure that the cache does not influence your results. OpenSearch does not explicitly offer a manual cache eviction feature for UltraWarm storage, but accessing large amounts of other data can help 'push out' the cached data.

---

## ISM Operations:

### Defining `start_time` and `end_time`:
> ℹ️ OpenSearch ISM policies do not natively support using the min(@timestamp) and max(@timestamp) as start and end times directly. However, you can write custom scripts or use external automation tools to update ISM policies dynamically based on query results that determine the earliest and latest timestamps in an index.

### Adjusting ISM Policy for Migrated Indices:
> 💡 For indices that have been migrated and thus have a new creation time, consider modifying the ISM policy to base the retention period on a timestamp field within the documents rather than the index creation date. This approach requires custom scripting or adjustments in your ISM policy definition.

Migrating Elastic version 8.9 to OpenSearch latest

Performance of Cold to UltraWarm Migration:

Performance Characteristics:

High-Level Time Estimates:

Measuring True Migration Time:

ISM Operations:

Defining start_time and end_time:

Adjusting ISM Policy for Migrated Indices:

Relevant content

Defining `start_time` and `end_time`: