Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

How do I create and schedule manual snapshots on OpenSearch Service?

10 minute read
0

I want to create a manual snapshot of my Amazon OpenSearch Service domain. I also want to automatically run the snapshot on a recurring basis.

Short description

To create a manual snapshot of your OpenSearch Service domain, set up an Amazon Simple Storage Service (Amazon S3) bucket to store the snapshot data. Then, register the S3 bucket as a snapshot repository. Finally, use the registered repository to create manual snapshots. You can also use OpenSearch Dashboards to take a snapshot with the registered repository.

To automatically schedule manual snapshots for OpenSearch 2.1+, use a snapshot policy. For earlier OpenSearch versions, use an AWS Lambda function with the Amazon EventBridge Scheduler.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Prerequisites:

  • Your domain runs OpenSearch 1.0+.
  • Your user or role's AWS Identity and Access Management (IAM) policy has the CreateBucket and PutBucketPolicy permissions that are required to manage the bucket.
  • Your user or role's IAM policy has the IAMFullAccess permission that's required to create an IAM policy and role.
  • Your user or role's IAM policy has the ESHttpPut permission that's required to register the repository.

Set up an S3 bucket

Complete the following steps:

  1. Create a new bucket in the same AWS Region as your OpenSearch Service domain.

  2. Use the IAM console or the AWS CLI to create an IAM policy that grants the required permissions to access the S3 bucket.
    Example policy:

    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket-name"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "iam:PassRole"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket-name/*"
      ]
    }
  ]
}

    Note: Replace your-bucket-name with your bucket name.

  3. Use the IAM console or the AWS CLI to create an IAM role for OpenSearch Service.
    The following example create-role command creates a role that's named TheSnapshotRole and attaches a trust policy to the role:

    aws iam create-role --role-name TheSnapshotRole --assume-role-policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "opensearchservice.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}'

  4. Use the IAM console or the AWS CLI to attach the Amazon S3 access policy to the role.

  5. Use the IAM console or the AWS CLI to create a policy with iam:PassRole permissions.
    Example policy:

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::account-id:role/TheSnapshotRole"
        },
        {
            "Effect": "Allow",
            "Action": "es:ESHttpPut",
            "Resource": "arn:aws:es:region:account-id:domain/domain-name/*"
        }
    ]
}

    Note: Replace account-id with your AWS account ID, region with your Region, and domain-name with your OpenSearch Service domain name.

  6. Run the following attach-user-policy command to attach the iam:PassRole policy to the IAM user that you configured to use the AWS CLI:

    aws iam attach-user-policy --user-name YOUR_IAMUSERNAME --policy-arn arn:aws:iam::account-id:policy/PassRolePolicy

    Note: Replace YOUR_IAMUSERNAME with the username and account-id with your account ID.

Register the snapshot repository

To register the snapshot repository, you must sign the request with AWS Signature Version 4. Repository names can't start with cs. Also, it isn't a best practice to write to the same repository from multiple domains. Configure only one domain to have write access to the repository.

Use Python to register the snapshot repository in Lambda. Or, to use awscurl and the AWS CLI to register the snapshot repository, run the following command on your instance:

read -r ACCESS_KEY SECRET_KEY SESSION_TOKEN <<< $(aws sts get-session-token --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' --output text)

awscurl \
  --access_key $ACCESS_KEY \
  --secret_key $SECRET_KEY \
  --session_token $SESSION_TOKEN \
  --region us-west-2 \
  --service es \
  -X PUT \
  -H "Content-Type: application/json" \
  'domain-endpoint/_snapshot/my-snapshot-repo' \
  -d '{
    "type": "s3",
    "settings": {
      "bucket": "bucketname",
      "region": "us-west-2",
      "role_arn": "arn:aws:iam::account-id:role/TheSnapshotRole"
    }
  }'

Note: Replace my-snapshot-repo with your repository name, bucketname with your bucket name, and us-west-2 with your Region. Also, replace domain-endpoint with your OpenSearch Service domain endpoint, account-id with your account ID, and TheSnapshotRole with your role name.

To install awscurl, run the following command:

pip install awscurl

For more information, see awscurl on the GitHub website.

Create the manual snapshot

Use curl commands to take a manual snapshot of your OpenSearch Service domain. For OpenSearch 2.5 or later, you can use OpenSearch Dashboards. For instructions, see Take a snapshot on the OpenSearch website.

After you create the snapshot, you can view it in the S3 bucket or through OpenSearch Dashboards.

Schedule automated manual snapshots

For OpenSearch 2.1 or later, you can use an API or OpenSearch Dashboards to create a snapshot policy with the registered repository. For more information, see Snapshot management API and Creating an SM policy on the OpenSearch website.

For OpenSearch versions earlier than 2.1, complete the following steps to schedule manual snapshots:

  1. If your OpenSearch Service domain is in a virtual private cloud (VPC), then create a new Lambda function with the Python 3.13 runtime in the same VPC.
  2. Add the AWSSDKPandas-Python313 layer that includes requests-aws4auth to the function. For more information, see AWS Lambda managed layers on the AWS SDK for pandas website and requests-aws4auth on the Python website.
  3. Enter the following code for your Lambda function: 
    import json
import boto3
import requests
from requests_aws4auth import AWS4Auth
from datetime import datetime

def lambda_handler(event, context):
    # OpenSearch endpoint and repository details
    host = 'domain-endpoint'
    repository = 'my-snapshot-repo'
    snapshot = f"snapshot-{datetime.now().strftime('%Y%m%d%H%m%S')}"
    region = 'us-west-2'
    service = 'es'
    
    # Get AWS credentials
    credentials = boto3.Session().get_credentials()
    awsauth = AWS4Auth(
        credentials.access_key,
        credentials.secret_key,
        region,
        service,
        session_token=credentials.token
    )
    
    # API endpoint for creating snapshot
    url = f'https://{host}/_snapshot/{repository}/{snapshot}'
    
    # Snapshot creation parameters
    payload = {
        "indices": "*",
        "ignore_unavailable": True,
        "include_global_state": False
    }
    
    try:
        # Make the request
        response = requests.put(
            url,
            auth=awsauth,
            json=payload,
            headers={"Content-nt-Type": "application/json"}
        )
        
        # Check if request was successful
        if response.status_code == 200:
            return {
                'statusCode': 200,
                'body': json.dumps({
                    'message': 'Snapshot creation initiated successfully',
                    'snapshot_name': snapshot,
                    'response': response.json()
                })
            }
        else:
            return {
                'statusCode': response.status_code,
                'body': json.dumps({
                    'message': 'Failed to create snapshot',
                    'error': response.text
                })
            }
            
    except Exception as e:
        return {
            'statusCode': 500,
            'body': json.dumps({
                'message': 'Error creating snapshot',
                'error': str(e)
            })
        }
    Note: Replace domain-endpoint with your domain endpoint, but don't include the https://. Also, replace my-snapshot-repo with your snapshot repository and us-west-2 with your Region.
  4. Attach the following permissions to the role that's attached to the Lambda function: 
    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "es:ESHttpPut",
                "es:DescribeElasticsearchDomain"
            ],
            "Resource": "arn:aws:es:region:account-id:domain/domain-name/*"
        }
    ]
}
    Note: Replace region with your Region, account-id with your account ID, my-snapshot-repo with your repository name, and domain-name with your domain.
  5. Use EventBridge Scheduler to invoke the Lambda function on a schedule.

(Optional) Delete unneeded snapshots

For manual snapshots that the snapshot policy creates, use the max_age parameter or the Retention period to set an expiration date. For more information and instructions, see Parameters and Creating an SM policy on the OpenSearch website.

If the Lambda function created the manual snapshot, then you can't configure an expire option. Instead, use another Lambda function to delete OpenSearch snapshots.

The following Lambda function deletes OpenSearch snapshots that are older than 30 days:

import json
import boto3
import requests
from requests_aws4auth import AWS4Auth
from datetime import datetime, timedelta
import re

def parse_date_from_snapshot_name(snapshot_name):
    # Pattern for format: snapshot-YYYY-MM-DDtHH-MM-SS
    try:
        # Skip "snapshot-" prefix and parse the date part
        date_str = snapshot_name[9:]  # Remove "snapshot-"
        return datetime.strptime(date_str, '%Y-%m-%dt%H-%M-%S')
    except (ValueError, IndexError):
        return None

def lambda_handler(event, context):
    # OpenSearch endpoint and repository details
    host = 'domain-endpoint'
    repository = 'my-snapshot-repo'
    region = 'us-west-2'
    service = 'es'
    
    # Get AWS credentials
    credentials = boto3.Session().get_credentials()
    awsauth = AWS4Auth(
        credentials.access_key,
        credentials.secret_key,
        region,
        service,
        session_token=credentials.token
    )
    
    try:
        # First, get all snapshots
        list_url = f'https://{host}/_snapshot/{repository}/_all'
        list_response = requests.get(
            list_url,
            auth=awsauth,
            headers={"Content-Type": "application/json"}
        )
        
        if list_response.status_code != 200:
            raise Exception(f"Failed to list snapshots: {list_response.text}")
            
        response_data = list_response.json()
        snapshots = response_data.get('snapshots', [])
        current_time = datetime.now()
        deletion_results = []
        
        # Process each snapshot
        for snapshot in snapshots:
            snapshot_name = snapshot.get('snapshot')
            try:
                snapshot_date = parse_date_from_snapshot_name(snapshot_name)
                
                if snapshot_date and (current_time - snapshot_date).days > 30:
                    # Delete snapshot
                    delete_url = f'https://{host}/_snapshot/{repository}/{snapshot_name}'
                    delete_response = requests.delete(
                        delete_url,
                        auth=awsauth,
                        headers={"Content-Type": "application/json"}
                    )
                    
                    deletion_results.append({
                        'snapshot': snapshot_name,
                        'status': 'deleted' if delete_response.status_code == 200 else 'failed',
                        'response': delete_response.text
                    })
                else:
                    deletion_results.append({
                        'snapshot': snapshot_name,
                        'status': 'skipped',
                        'reason': 'Not older than 30 days or date not found in name'
                    })
            except Exception as e:
                deletion_results.append({
                    'snapshot': snapshot_name,
                    'status': 'error',
                    'error': str(e)
                })
                
        return {
            'statusCode': 200,
            'body': json.dumps({
                'message': 'Snapshot deletion process completed',
                'results': deletion_results
            })
        }
            
    except Exception as e:
        return {
            'statusCode': 500,
            'body': json.dumps({
                'message': 'Error in snapshot deletion process',
                'error': str(e)
            })
        }

Note: Replace domain-endpoint with your domain endpoint without the https://, region with your Region, and my-snapshot-repo with your repository name. It's a best practice to configure Lambda function timeout to 60 seconds.

Troubleshoot issues

Take the following actions based on the error that you receive.

"No permissions for [cluster:admin/snapshot/create]" error

If you activated fine-grained access control on your OpenSearch Service domain, then you must grant OpenSearch Service access to the Lambda function role. If you don't grant access, then you receive the following permissions error:

""statusCode": 403, "body": "{"message": "Failed to create snapshot", "error": "{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"no permissions for [cluster:admin/snapshot/create] and User [name=arn:aws:iam::123456789012:role/service-role/Lambda_ES_Snapshot-role"

To resolve this issue, run the following command to grant OpenSearch Service access to the Lambda function role:

awscurl \
--access_key ABCDE \
--secret_key EXAMPLESECRET\
--region us-west-2 \
--service es \
-X PUT \
-H "Content-Type: application/json" \
'https://domain-endpoint/_plugins/_security/api/rolesmapping/all_access' \
-d '{"backend_roles": ["arn:aws:iam::account-id:role/service-role/Lambda_AOS_Snapshot-role"], "hosts": [], "users": ["master-user", "arn:aws:iam::account-id:user/second-primary-user" ]}'

Note: Replace ABCDE with your access key, EXAMPLESECRET with your secret, and us-west-2 with your Region. Also, replace domain-endpoint with your OpenSearch Service domain endpoint, account-id with your account ID, and Lambda_AOS_Snapshot-role with your Lambda function role.

"AccessDeniedException" error

If you receive the "AccessDeniedException" error message, then you must configure a trust relationship for the IAM role that accesses the S3 bucket. To troubleshoot this issue, use IAM policy simulator to identify what part of your IAM policy you must update.

"S3Exception" error

If you receive the "S3Exception: The bucket is in region 'us-west-1'" error message, then you must move the bucket to the same Region as the OpenSearch Service domain.

"SignatureDoesNotMatch" error

If you receive the "SignatureDoesNotMatch" error message, then your AWS credentials are expired or you used incorrect information in your signature. Keep your AWS credentials up to date, and make sure that the Region is correct in your signature.

"not authorized to perform: iam:PassRole" error

If you don't use AWS Signature Version 4, then you receive the following error message:

"Message":"User: anonymous is not authorized to perform: iam:PassRole on resource: arn:aws:iam::account-id:role/TheSnapshotRole because no resource-based policy allows the iam:PassRole action"

To troubleshoot this error, see Why do I get a "User: anonymous is not authorized" error when I try to access my OpenSearch Service cluster?

For more troubleshooting information, see How do I resolve the manual snapshot error in my OpenSearch Service cluster?

Related information

Unleash the power of Snapshot Management to take automated snapshots using OpenSearch Service

Take manual snapshots and restore in a different domain spanning across various Regions and accounts in OpenSearch Service

How do I restore data from an OpenSearch Service domain in another AWS account?

Topics
Analytics
Tags
Amazon OpenSearch Service
Language
English
AWS OFFICIALUpdated a month ago
No comments

Relevant content