Domain processing status Stuck for serveral days after config change

Question

![attached pictures from the aws opensearch console](/media/postImages/original/IMoAdFLcOXS3ijEywJMisY3g)

![Enter image description here](/media/postImages/original/IMkjnhDdwpQNGQKNO4TYxbAQ)

![Enter image description here](/media/postImages/original/IM0wmUDbgySZKyM_1E1wMECA)

i want to change the volume size from 27 to 30 GB and after that many Chards got unassigned and Cluster Red.
I deleted the Indexes and the indextemplates with it. Så now is every thing green and i have stopped all data injection.

Please Advice ?
BestRegards
DrRask

Describe-domain-change-progress:
{
    "ChangeProgressStatus": {
        "ChangeId": "7cc09dee-84df-4f30-890a-09bf47a8dd7a",
        "StartTime": "2024-03-14T01:45:59.662000+01:00",
        "Status": "COMPLETED",
        "PendingProperties": [],
        "CompletedProperties": [
            "LogPublishingOptions : {\"ES_APPLICATION_LOGS\":{\"cloudWatchLogsLogGroupArn\":\"arn:aws:logs:eu-north-1:99999999992:log-group:/aws/OpenSearchService/domains//application-logs\",\"enabled\":true}}"
        ],
        "TotalNumberOfStages": 1,
        "ChangeProgressStages": [
            {
                "Name": "Dynamic Update",
                "Status": "COMPLETED",
                "Description": "Configuration update(s) complete.",
                "LastUpdated": "2024-03-14T01:48:02.891000+01:00"
            }
        ]
    }
}
    "DomainStatus": {
        "DomainId": "*",
        "DomainName": "*",
        "ARN": "arn:aws:es:eu-north-1",
        "Created": true,
        "Deleted": false,
        "Endpoint": "search-opensearch.eu-north-1.es.amazonaws.com",
        "Processing": false,
        "UpgradeProcessing": false,
        "EngineVersion": "OpenSearch_2.11",
        "ClusterConfig": {
            "InstanceType": "t3.small.search",
            "InstanceCount": 3,
            "DedicatedMasterEnabled": true,
            "ZoneAwarenessEnabled": true,
            "ZoneAwarenessConfig": {
                "AvailabilityZoneCount": 3
            },
            "DedicatedMasterType": "t3.small.search",
            "DedicatedMasterCount": 3,
            "WarmEnabled": false,
            "ColdStorageOptions": {
                "Enabled": false
            },
            "MultiAZWithStandbyEnabled": false
        },
        "EBSOptions": {
            "EBSEnabled": true,
            "VolumeType": "gp3",
            "VolumeSize": 30,
            "Iops": 3000,
            "Throughput": 125
        },

"ServiceSoftwareOptions": [
{
            "CurrentVersion": "OpenSearch_2_11_R20231113-P2",
            "NewVersion": "OpenSearch_2_11_R20231113-P3",
            "UpdateAvailable": false,
            "Cancellable": false,
            "UpdateStatus": "NOT_ELIGIBLE",
            "Description": "A newer release OpenSearch_2_11_R20231113-P3 is available. Your domain is not eligible to be updated. Please refer to the public documentation.",
            "AutomatedUpdateDate": "1970-01-01T01:00:00+01:00",
            "OptionalDeployment": true
        },

"UpgradeHistories": [
        {
            "UpgradeName": "Upgrade from OpenSearch_2.9 to OpenSearch_2.11",
            "StartTimestamp": "2024-02-13T12:37:42+01:00",
            "UpgradeStatus": "SUCCEEDED",
            "StepsList": [
                {
                    "UpgradeStep": "UPGRADE",
                    "UpgradeStepStatus": "SUCCEEDED",
                    "ProgressPercent": 0.0
                },
                {
                    "UpgradeStep": "SNAPSHOT",
                    "UpgradeStepStatus": "SUCCEEDED",
                    "ProgressPercent": 0.0
                },
                {
                    "UpgradeStep": "PRE_UPGRADE_CHECK",
                    "UpgradeStepStatus": "SUCCEEDED",
                    "ProgressPercent": 0.0
                }
            ]
        },
        {
            "UpgradeName": "Pre-Upgrade Check from OpenSearch_2.9 to OpenSearch_2.11",
            "StartTimestamp": "2024-02-13T12:36:23+01:00",
            "UpgradeStatus": "SUCCEEDED",
            "StepsList": [
                {
                    "UpgradeStep": "PRE_UPGRADE_CHECK",
                    "UpgradeStepStatus": "SUCCEEDED",
                    "ProgressPercent": 0.0
                }
            ]
        },

rePost-User-8296001 · Answer

DrRask,

Thank you for your response. As the cluster has been stuck for several days, it will require manual intervention from AWS support to mitigate the issue and retry the failed workflow to get the cluster out of processing state. I would suggest reaching out to AWS Support.

Additional tips:

- I see you are using T type instance. The issue may occur while using T-type instances, which are burstable and not recommended for production environments. CPU spikes exceeding the instance capacity can led to CPU credit depletion, causing node drops and subsequent cluster instability.

- Node drops can cause the primary shards of the cluster to be unassigned leading to Red health status.

- In order to get the cluster out of processing state, it is necessary to first address the red health status.        
 *(which you mitigated by deleting the indices)*

- Once the cluster is in green health, the AWS support should be able to retry the failed activities and get your cluster out of processing state.

*To prevent similar incidents, please review and implement the best practices for cluster stability outlined in our documentation: [+] https://docs.aws.amazon.com/opensearch-service/latest/developerguide/bp.html*

Domain processing status Stuck for serveral days after config change

Add your answer

Relevant content