Master and internal user accounts reset on Amazon OpenSearch Service: why snapshots don't restore them, and what to prepare

Symptoms

• Invalid username or password when signing in to OpenSearch Dashboards with the existing master user credentials
• Internal users, roles, and role mappings you created are no longer listed
• In some cases, business data indexes are missing as well
• Usually noticed immediately after a version upgrade or a configuration change

Cause

Account data is stored in an index inside the domain. With fine-grained access control, the master user and the internal user database live in the security system index. As described in Fine-grained access control, this index is a special index managed by the security plugin and is not treated like a regular data index. The documentation states that the internal user database is stored in an OpenSearch index, so you can't share it with other clusters.

If the number of available copies reaches zero, the index is recreated with default configuration. When the security system index comes up without an available copy, the security plugin populates it with defaults. The master credentials revert to their initial values, and the internal users and role mappings you created are not retained.

Configuring replicas does not guarantee that a copy survives. Replicas assume that node failures are independent of one another. If the nodes holding the copies stop functioning at the same time, no healthy copy remains to promote.

The highest-risk window is a configuration change or an upgrade. Configuration changes and version upgrades run as blue/green deployments, provisioning new nodes and copying shards, which drives load on all data nodes upward at the same time. If the data nodes are burstable instances, the risk is higher still: identical instance types under identical load exhaust their CPU credits at nearly the same time. For more detail, see How to resolve OpenSearch cluster performance issues with t2/t3 burstable instances.

Resolution

1. Reset the master user credentials. In the console, edit the domain's security configuration and specify the master user again to regain Dashboards access. If you previously used an IAM master user, fine-grained access control re-maps the all_access role to the new IAM ARN you specify. If you previously used the internal user database, a new master user is created.

2. Restore your business data indexes. Unlike the security index, data indexes can be restored from a snapshot. Specify the indexes individually.

POST _snapshot/cs-automated/<snapshot-name>/_restore
{ "indices": "my-index" }

Note: If not all primary shards were available when a snapshot was taken, its state is PARTIAL and some shard data was not stored, so a snapshot from the incident window may be incomplete and you may need an earlier one. When restoring to a domain that uses Multi-AZ with Standby, the replica settings of the restored indexes must be compatible with the standby configuration; otherwise the restore fails unless you override them with index_settings.

3. Recreate internal users and role mappings manually. Snapshots will not bring this data back. Four constraints combine.

• The official restore procedure excludes this index. Restoring data from snapshots notes that, due to special permissions on the OpenSearch Dashboards and fine-grained access control indexes, attempts to restore all indexes might fail — especially from an automated snapshot. The recommended command itself excludes them with "indices": "-.kibana*,-.opendistro*", and depending on your version you may also need to exclude -.opensearch-observability* and -.plugins-ml-config*.
• You can't restore into a cluster that already has an index with the same name. After an account reset, the security system index has already been recreated with defaults, so restoring over the same name is not possible.
• Automated snapshots can't be restored to a different domain. Per Creating index snapshots, automated snapshots are for cluster recovery only. Restoring to a new domain to read the account data out is only an option if you have a manual snapshot.
• While the cluster is red, automated snapshots keep failing. The same document states that all automated snapshots fail while the cluster is red, and that if you don't correct the problem within two weeks you can permanently lose the data. When missing indexes have turned the cluster red, your recovery window is limited to 14 days.

Without a prior backup there is no source to work from, so you end up reconstructing usernames and permissions by tracing them back through application configuration. This step accounts for most of the recovery time.

Prevention

Every item below only helps if it is in place before an incident.

1. Export internal users, roles, and role mappings on a schedule.

GET _plugins/_security/api/internalusers
GET _plugins/_security/api/roles
GET _plugins/_security/api/rolesmapping

Keep the responses in a version-controlled repository or another separate location. A PUT to role mappings replaces the current mappings, which is why the documentation tells you to perform a GET first — so this export is already part of normal operational practice. Note that the internal user response does not include passwords, so application credentials must be managed separately in a secret store.

2. Register a manual snapshot repository. Manual snapshots can be restored to a different domain, which is the only path available if the domain itself becomes unrecoverable. This requires an S3 bucket and an IAM role delegated to OpenSearch Service.

3. Monitor automated snapshot health. In CloudWatch metrics, an AutomatedSnapshotFailure value of 1 means no automated snapshot was taken for the domain in the previous 36 hours. An alarm on this metric tells you early that your recovery path has gone away.

4. Run a configuration where copies don't become unavailable together. Operational best practices recommend avoiding T2 and t3.small for production domains. Running data nodes on fixed-performance instances removes the correlated-failure path that leaves an index with no available copy.

5. Enable error log publishing. You can't enable it retroactively, so turn it on ahead of time to have evidence available when you need to determine a cause.

Related information

• Fine-grained access control in Amazon OpenSearch Service
• Restoring data from snapshots
• Creating index snapshots in Amazon OpenSearch Service
• Monitoring OpenSearch cluster metrics with Amazon CloudWatch
• Making configuration changes in Amazon OpenSearch Service
• Operational best practices for Amazon OpenSearch Service
• Can't restore indices from snapshots in Amazon OpenSearch Service
• Troubleshoot fine-grained access control issues in OpenSearch Service