Questions tagged with Amazon OpenSearch Service

I created a domain in OpenSearch and I've set the Access policy like the following: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "es:*", "Resource": "arn:aws:es:ca-central-1:xxxxxx:domain/xxxxx/*" } ] } From what I understand I give access to all users but I am the only one who can browse the tab Indices in the console, my collegues get this error message: {"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [indices:monitor/stats] and User [name=arn:aws:iam::xxxx:user/users/xyuser, backend_roles=[], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [indices:monitor/stats] and User [name=arn:aws:iam::xxx:user/users/xyuseri, backend_roles=[], requestedTenant=null]"},"status":403} They also get the same error message trying to perform queries with the REST API Please what am I missing?

My app is in beta, so it's not really being used, I received a spike in my bill because of app sync and open search service. I can't figure out why this charge is happening or who triggered it. Could someone help me?

OpenSearch version: 1.2 Service software version: R20220323-P3 3 data nodes with r5.large.search instance type After enable auto-tune, it shows "Error" p.s. the initial version of the domain is ElasticSearch version 6.2

Hello, I'm trying to find documentation/information about foreign tokenizers supported by Opensearch 1.1. I'm aware that Opensearch supports several foreign language tokenizer plugins (such as Japanese, Korean, Chinese, Polish and Ukrainian. ref: https://opensearch.org/docs/1.1/opensearch/install/plugins/) I've came across ElasticSearch documentation of their supported built-in analyzer (https://www.elastic.co/guide/en/elasticsearch/reference/7.10/analysis-lang-analyzer.html) and I'm wondering if there exists such built-in analyzer for Opensearch as well. Or since ElasticSearch ver 7.10 is supported by Opensearch 1.1, is it safe to assume that Opensearch supports all those built-in analyzer mentioned in the above URL? Thank you

Hi, I created an AWS Opensearch domain in a VPC, but I can't figure out how to access it? When I am clicking on AWS Opensearch dashboard it's not opening and there is a timeout. I fould 2 solutions: EC2 instance with SSH tunnel or NGINX proxy on EC2 instance. But can I access the domain in a VPC without spinning the EC2 instance at all?

Hi I was following this [AWS tutorial](https://aws.amazon.com/blogs/mobile/analyze-device-generated-data-with-aws-iot-and-amazon-elasticsearch-service/) but with my own sensor data. I've created the following IoT Core Rule OpenSearch Action: ``` OpenSearchTopicRule: Type: AWS::IoT::TopicRule Properties: TopicRulePayload: Actions: - OpenSearch: Endpoint: !Join ['', ['https://', !GetAtt OpenSearchServiceDomain.DomainEndpoint]] Id: '${newuuid()}' Index: sensors RoleArn: !GetAtt IoTOSActionRole.Arn Type: sensor_data Sql: SELECT *, timestamp() as ts FROM 'Greenhouse/+/Sensor/Status' ``` The IoTOSActionRole has propper es:ESHttpPut permission. But when I try to create an index with following command send from Postman that would match the `Type: sensor_data` attribute: ``` curl --location --request PUT 'https://search-iot***-avt***i.eu-west-1.es.amazonaws.com/sensors' \ --header 'Content-Type: application/json' \ --data-raw '{ "mappings": { "sensor_data": { "properties": { "ts": { "type": "long", "copy_to": "datetime"}, "datetime": {"type": "date", "store": true}, "deviceID": {"type": "text", "store": true}, "humidity": {"type": "integer", "store": true}, "temperature": {"type": "integer", "store": true}, "lux": {"type": "integer", "store": true}, "soil": {"type": "integer", "store": true} }}}' ``` I receive an error: ``` { "error": { "root_cause": [ { "type": "mapper_parsing_exception", "reason": "Root mapping definition has unsupported parameters: [sensor_data : {properties={datetime={store=true, type=date}, temperature={store=true, type=integer}, humidity={store=true, type=integer}, soil={store=true, type=integer}, deviceID={store=true, type=text}, lux={store=true, type=integer}, ts={copy_to=datetime, type=long}}}]" } ], "type": "mapper_parsing_exception", "reason": "Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters: [...]", "caused_by": { "type": "mapper_parsing_exception", "reason": "Root mapping definition has unsupported parameters: [...}]" } }, "status": 400 } ``` I've tried removing the 'type' `"sensor_data"` attribute and that allowed me to create an index with that mapping, ``` { "acknowledged": true, "shards_acknowledged": true, "index": "sensors" } ``` and then index pattern in OpenSearch Dashboard, but what happens then is that the IoT Core Rule even though it gets triggered does not result in any data ingestion to the OpenSearch domain. So I guess IoT Core Action tries to send that data with type `sensor_data`but there's no corresponding type in OS. Additionally, when I open the Discovery tab in OS dashboard I get this notice: ``` "undefined" is not a configured index pattern ID Showing the default index pattern: "sensors*" (d97775d0-cc44-11ec-acd4-bf72aa2fd725) ``` I'm using the OpenSearch 1.2 (latest) version. Sample data: ``` { "deviceID": "Tomatoes", "Greenhouse": 1, "date": "05-05", "time": "09:35:39", "timestamp": 1651743339, "humidity": 60, "temperature": 33.3, "lux": 9133.333, "soil": 78 } ``` What PUT call I have to make to create a `sensor_data` type mapping in OS that would match the type specified in IoT Core OpenSearch Action Rule?

Opensearch's 3 node cluster's diskspace was upgraded from 500GB to 600GB. Step 3 of 4: The copying of shards to new nodes became stuck. "Copying shards to new nodes - 48 of 52 done" is still the status after more than two days. I verified all of the parameters in monitoring (CPU, JVM, and storage) and they all look good. cluster status in green. What can we do now??

Initially the cluster was stuck for more than a week in "Processing.." after applying a service patch update. Therefore I deleted the cluster and tried to re-create it but now it's just stuck on "Step 1 of 3: Creating a new environment" I've tried deleting and re-creating it twice but get's stuck on the same step Can you please help with getting this sorted so I can re-create the cluster?

Hi, my Developer propose for my Video on Demand app for the search function Angolia or Open Search. He is telling me that DynamoDB is not perfect for the Filter. Is it true? What would be the best and the cheapest way?

Trying to migrate existing data from AWS RDS Postgres to AWS managed OpenSearch, but it is not working, no rows were migrated to opensearch, When checking the Cloudwatch log getting below error Bulk request failed. no retry. TotalRecordCount 4080, FailedRecordCount 4080 [1026400] (elasticsearch_bulk_utils.c:181) DMS has the following configuration: { "TargetMetadata": { "TargetSchema": "", "SupportLobs": false, "FullLobMode": false, "LobChunkSize": 0, "LimitedSizeLobMode": false, "LobMaxSize": 0, "InlineLobMaxSize": 0, "LoadMaxFileSize": 0, "ParallelLoadThreads": 5, "ParallelLoadBufferSize": 100, "BatchApplyEnabled": false, "TaskRecoveryTableEnabled": false, "ParallelLoadQueuesPerThread": 0, "ParallelApplyThreads": 0, "ParallelApplyBufferSize": 100, "ParallelApplyQueuesPerThread": 0 }, "FullLoadSettings": { "TargetTablePrepMode": "DO_NOTHING", "CreatePkAfterFullLoad": false, "StopTaskCachedChangesApplied": false, "StopTaskCachedChangesNotApplied": false, "MaxFullLoadSubTasks": 8, "TransactionConsistencyTimeout": 600, "CommitRate": 50000 }, "Logging": { "EnableLogging": true, "LogComponents": [ { "Id": "TRANSFORMATION", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "SOURCE_UNLOAD", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "IO", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "TARGET_LOAD", "Severity": "LOGGER_SEVERITY_DETAILED_DEBUG" }, { "Id": "PERFORMANCE", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "SOURCE_CAPTURE", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "SORTER", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "REST_SERVER", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "VALIDATOR_EXT", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "TARGET_APPLY", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "TASK_MANAGER", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "TABLES_MANAGER", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "METADATA_MANAGER", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "FILE_FACTORY", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "COMMON", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "ADDONS", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "DATA_STRUCTURE", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "COMMUNICATION", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "FILE_TRANSFER", "Severity": "LOGGER_SEVERITY_DEFAULT" } ], "CloudWatchLogGroup": null, "CloudWatchLogStream": null }, "ControlTablesSettings": { "historyTimeslotInMinutes": 5, "ControlSchema": "", "HistoryTimeslotInMinutes": 5, "HistoryTableEnabled": true, "SuspendedTablesTableEnabled": false, "StatusTableEnabled": true, "FullLoadExceptionTableEnabled": false }, "StreamBufferSettings": { "StreamBufferCount": 3, "StreamBufferSizeInMB": 8, "CtrlStreamBufferSizeInMB": 5 }, "ChangeProcessingDdlHandlingPolicy": { "HandleSourceTableDropped": true, "HandleSourceTableTruncated": true, "HandleSourceTableAltered": true }, "ErrorBehavior": { "DataErrorPolicy": "LOG_ERROR", "EventErrorPolicy": null, "DataTruncationErrorPolicy": "LOG_ERROR", "DataErrorEscalationPolicy": "SUSPEND_TABLE", "DataErrorEscalationCount": 0, "TableErrorPolicy": "SUSPEND_TABLE", "TableErrorEscalationPolicy": "STOP_TASK", "TableErrorEscalationCount": 0, "RecoverableErrorCount": -1, "RecoverableErrorInterval": 5, "RecoverableErrorThrottling": true, "RecoverableErrorThrottlingMax": 1800, "RecoverableErrorStopRetryAfterThrottlingMax": true, "ApplyErrorDeletePolicy": "IGNORE_RECORD", "ApplyErrorInsertPolicy": "LOG_ERROR", "ApplyErrorUpdatePolicy": "LOG_ERROR", "ApplyErrorEscalationPolicy": "LOG_ERROR", "ApplyErrorEscalationCount": 0, "ApplyErrorFailOnTruncationDdl": false, "FullLoadIgnoreConflicts": true, "FailOnTransactionConsistencyBreached": false, "FailOnNoTablesCaptured": true }, "ChangeProcessingTuning": { "BatchApplyPreserveTransaction": true, "BatchApplyTimeoutMin": 1, "BatchApplyTimeoutMax": 30, "BatchApplyMemoryLimit": 500, "BatchSplitSize": 0, "MinTransactionSize": 1000, "CommitTimeout": 1, "MemoryLimitTotal": 1024, "MemoryKeepTime": 60, "StatementCacheSize": 50 }, "PostProcessingRules": null, "CharacterSetSettings": null, "LoopbackPreventionSettings": null, "BeforeImageSettings": null, "FailTaskWhenCleanTaskResourceFailed": false, "TTSettings": null } Opensearch have index with following settings { "settings": { "index.max_ngram_diff" :8, "analysis": { "analyzer": { "my_ngram_analyzer": { "type": "custom", "tokenizer": "standard", "filter": [ "lowercase", "mynGram" ] } }, "filter": { "mynGram": { "type": "nGram", "min_gram": 6, "max_gram": 14, "token_chars": [ "letter", "digit", "whitespace", "symbol" ] } } }, "number_of_shards": 6, "number_of_replicas": 1 }, "mappings" : { "properties" : { "created_at" : { "type" : "date" }, "id" : { "type" : "long" }, "name" : { "type" : "text", "analyzer":"my_ngram_analyzer" , "search_analyzer": "my_ngram_analyzer" }, "phone" : { "type" : "text", "analyzer":"my_ngram_analyzer" , "search_analyzer": "my_ngram_analyzer" }, "updated_at" : { "type" : "date" } } } } I have tried to insert a sample document using _bulk API on opensearch console and it worked, below is the thing I had tried over opensearch, which worked POST _bulk {"index":{"_index":"contacts"}} {"name": "name","phone" : "11111111","created_at" : "2021-12-21T12:12:59","updated_at" : "2021-12-21T12:12:59","id": 101}

Hi, How can you scale AWS OpenSearch e.g. to switch off the cluster at night and cut some costs? I see that Step Functions does not have this ability.

I’m having trouble setting up multi-region replication between two aws opensearch domains. I’ve been following the documentation (https://docs.aws.amazon.com/opensearch-service/latest/developerguide/replication.html) but once I start the replication via PUT call and query the replication status I see that the status is stuck at ‘BOOTSTRAPPING’ and I can see that no bytes have been transferred, even hours later. PUT request payload to start the replication: ``` { "leader_alias": "dss-pre-prod-use2-usw2", "leader_index": "devportal.resourcev2", "use_roles":{ "leader_cluster_role": "all_access", "follower_cluster_role": "all_access" } } ``` GET response: ``` { "status": "BOOTSTRAPPING", "reason": "User initiated", "leader_alias": "dss-pre-prod-use2-usw2", "leader_index": "devportal.resourcev2", "follower_index": "devportal.resourcev2", "bootstrap_details": { "bytes_total": 0, "bytes_recovered": 0, "bytes_percent": 0.0, "files_total": 0, "files_recovered": 0, "files_percent": 0.0, "start_time": 1650317767399, "running_time": 18585600 } } ``` There are only 5 documents in the leader index, so the replication seems stuck. I've tried deleting the follower index and restarting the replication multiple times but with the same result. How can I debug this issue?

I am trying to create Domain in open search, I used the Below IAM permission but everytime it is giving me this error-: Before you can proceed, you must enable a service-linked role to give Amazon OpenSearch Service permissions to create and manage resources on your behalf I have also attached the Service Linked Role but still I am facing the Issue I am using this IAM policy { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "es:ESHttpDelete", "es:ESHttpGet", "es:ESHttpHead", "es:ESHttpPost", "es:ESHttpPut", "es:ESHttpPatch", "ec2:AuthorizeSecurityGroupIngress", "ec2:CreateNetworkInterface", "ec2:CreateSecurityGroup", "ec2:DeleteNetworkInterface", "ec2:DeleteSecurityGroup", "ec2:DescribeAvailabilityZones", "ec2:DescribeNetworkInterfaces", "ec2:DescribeSecurityGroups", "ec2:DescribeSubnets", "ec2:DescribeVpcs", "ec2:ModifyNetworkInterfaceAttribute", "ec2:RevokeSecurityGroupIngress", "elasticloadbalancing:AddListenerCertificates", "elasticloadbalancing:RemoveListenerCertificates" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "es:AddTags", "es:AssociatePackage", "es:CreateDomain", "es:CreateOutboundConnection", "es:DeleteDomain", "es:DescribeDomain", "es:DescribeDomainAutoTunes", "es:DescribeDomainConfig", "es:DescribeDomains", "es:DissociatePackage", "es:ESCrossClusterGet", "es:GetCompatibleVersions", "es:GetUpgradeHistory", "es:GetUpgradeStatus", "es:ListPackagesForDomain", "es:ListTags", "es:RemoveTags", "es:StartServiceSoftwareUpdate", "es:UpdateDomainConfig", "es:UpdateNotificationStatus", "es:UpgradeDomain" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "es:AcceptInboundConnection", "es:CancelServiceSoftwareUpdate", "es:CreatePackage", "es:CreateServiceRole", "es:DeletePackage", "es:DescribeInboundConnections", "es:DescribeInstanceTypeLimits", "es:DescribeOutboundConnections", "es:DescribePackages", "es:DescribeReservedInstanceOfferings", "es:DescribeReservedInstances", "es:GetPackageVersionHistory", "es:ListDomainNames", "es:ListDomainsForPackage", "es:ListInstanceTypeDetails", "es:ListInstanceTypes", "es:ListNotifications", "es:ListVersions", "es:PurchaseReservedInstanceOffering", "es:RejectInboundConnection", "es:UpdatePackage" ], "Resource": "*" }, { "Sid": "AllowCreationOfServiceLinkedRoleForOpenSearch", "Effect": "Allow", "Action": [ "iam:CreateServiceLinkedRole", "iam:PassRole" ], "Resource": [ "arn:aws:iam::*:role/aws-service-role/opensearchservice.amazonaws.com/AWSServiceRoleForAmazonOpenSearchService*", "arn:aws:iam::*:role/aws-service-role/es.amazonaws.com/AWSServiceRoleForAmazonOpenSearchService*" ], "Condition": { "StringLike":{ "iam:AWSServiceName": [ "opensearchservice.amazonaws.com", "es.amazonaws.com" ] } } } ] }

We have Appsync as our primary db. We are using Opensearch for aggregarion, visualisation and reporting purposes. But the Opensearch is showing some data anamoly like the values showing sql aggregation query is different from what coming on the visualised dashboard of opensearch. Also, the sql plugin of the opensearch has certain set of limitation that are not letting us to use the full potential of it. Like, if we put a limit of more than 10000 to fetch the records we are getting an error of "Shards failed". Also, the aggregations are not working properly on more than 2 joins and there are various other such limitations. Currently we are using t3.medium.search instance with 5 nodes. Can anyone please help me out with this.

I have a QuickSight Enterprise instance that produces a timeout error when I try to validate the connection to both RDS(Postgresql) and OpenSearch instances on my vpc. I have setup the sg inbound and outbound rules, the role policies, and am able to auto discover both the RDS and OpenSearch instances. So, what am I missing?

I have created the domain in VPC form with enabled Fine-Grained with internal database configuration (non-Cognito). Now, I was tried to make an additional user with limited permissions to a single index by following this tutorial: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/fgac-walkthrough-basic.html but in result: * I can see only documents matched by `"FlightDelay": true` rule * I can see anonymized `Dest` field * I still can use the `_search/` query, though I have limited it by the pattern `opensearch_dashboards_sample_data_fl*` for search only on the `opensearch_dashboards_sample_data_flights` index Additionally, I have checked the same rules and settings on my local OpenSearch (also as 1.2) and there everything works as I expected. Did I forget about something or is there anything else that I must do?

I am using AWS lambda function to save response in S3 bucket. I am able to save response size upto 10000. But, when increased beyond 10000 I end up getting the error - Result window is too large, from + size must be less than or equal to: [10000] but was [25000]. I am unable to increase the limit of the index max result window size greater than 10000. How do I proceed with the issue?

Posting https://github.com/aws/aws-cdk/issues/18239 here for increased visibility. Stacks using attributes of Opensearch domains will randomly fail to deploy. The message shown in Cloudformation is similar to "Unable to retrieve Arn attribute for AWS::OpenSearchService::Domain, with error message Internal error occurred". The actual cause as per discussion in that issue is that the DescribeDomain calls are being throttled, and Cloudformation is not handling the throttling sensibly. This is a pernicious bug because of it's unpredictability. In my case, I've deployed my application successfully many times, and have never seen this issue before. I hit this issue whilst upgrading from Elasticsearch 7.10 to Opensearch 1.2; I was making no other changes. I'm stuck unable to rollback cleanly either, as it's not possible to go from Opensearch 1.2 to Elasticsearch 7.10. So I'm having to skip rolling back the cluster, and attempt to roll forward until either it succeeds, or I'm forced to hardcode the domain attributes I should be looking up. According to the CDK issue, the internal tracking code for this issue is V498467686.

Hi, Instance connection status is coming red for open search domain and status is coming as Processing updates . How can I resolve this issue.

Good evening everyone. I'm trying to migrate some tables from RDS MySQL to OpenSearch through DMS. However, I have a problem with decimal columns. When configuring the task and sending it to OpenSearch, the DMS is treating it as a Float. But Float doesn't have the precision I need (decimal(10,2)) and it distorts the data. OpenSearch supports both double and scaled_float, but if in my index I map the column with these types I get DMS error like: > "cannot be changed from type [scaled_float] to [float]" From the documentation it seems that DMS only supports int and float for numbers. (https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Elasticsearch.html) I would like to know if there is any alternative to this.

We have a single node Opensearch 1.1 cluster. After selecting to upgrade it to 1.2 the cluster has been stuck on "Preparing to process updates". I found out that it is stuck because the default settings for .opendistro-* indices are to create 5 primaries and 1 replica shard. As our cluster only consists of a single shards those some of the .opendistro-* indices are not unallocated. I created index templates for those indices in question with 1 primary and 0 replicas and deleted some of the .opendistro indices but there are .opendistro-alerting-alerts indices which I can not delete even though I have all the permissions possible. Also followed this guide but still getting 403 https://aws.amazon.com/premiumsupport/knowledge-center/opensearch-disable-index-alerts/ GET _cat/shards .opendistro-alerting-alerts 3 r UNASSIGNED .opendistro-alerting-alerts 1 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822 .opendistro-alerting-alerts 1 r UNASSIGNED .opendistro-alerting-alerts 2 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822 .opendistro-alerting-alerts 2 r UNASSIGNED .opendistro-alerting-alerts 4 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822 .opendistro-alerting-alerts 4 r UNASSIGNED .opendistro-alerting-alerts 0 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822 .opendistro-alerting-alerts 0 r UNASSIGNED .opendistro-alerting-alert-history-2022.03.16-1 3 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822 .opendistro-alerting-alert-history-2022.03.16-1 3 r UNASSIGNED .opendistro-alerting-alert-history-2022.03.16-1 1 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822 .opendistro-alerting-alert-history-2022.03.16-1 1 r UNASSIGNED .opendistro-alerting-alert-history-2022.03.16-1 2 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822 .opendistro-alerting-alert-history-2022.03.16-1 2 r UNASSIGNED .opendistro-alerting-alert-history-2022.03.16-1 4 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822 .opendistro-alerting-alert-history-2022.03.16-1 4 r UNASSIGNED .opendistro-alerting-alert-history-2022.03.16-1 0 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822 .opendistro-alerting-alert-history-2022.03.16-1 0 r UNASSIGNED DELETE .opendistro-alerting-alert-history-2022.03.16-1 DELETE .opendistro-alerting-alerts { "error" : { "root_cause" : [ { "type" : "security_exception", "reason" : "no permissions for [] and User [name=xxx, backend_roles=[], requestedTenant=]" } ], "type" : "security_exception", "reason" : "no permissions for [] and User [name=xxx, backend_roles=[], requestedTenant=]" }, "status" : 403 } I would gladly add some additional data nodes to the cluster in order to fix this but this also fails as the cluster is stuck on "Preparing to process updates". EDIT: I tried deleting the .opendistro indices with "m aster" user but even "m aster" user receives error 403.

Hi I've created test opensearch domain according to guide https://docs.aws.amazon.com/opensearch-service/latest/developerguide/gsgcreate-domain.html It has Public access, fine grained access control with mster-user/password But when I try to connect to opensearch with those command > curl -XPOST -u 'my-mster-user:my-mster-password' 'https://search-wb-elk-xqfw2gbhc4ie6ormpmsf3gb7ae.us-east-2.es.amazonaws.com/_bulk' --data-binary @movies.json -H 'Content-Type: application/json' I get > "Message":"User: anonymous is not authorized to perform: es:ESHttpPost because no resource-based policy allows the es:ESHttpPost action" Whereas I've setup everything by default like it is said in guide What's the problem here? here is my access policy `{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "es:*", "Resource": "arn:aws:es:us-east-my-arn:domain/wb_elk/*" } ] }`

I am trying to create Data Stream -> Firehose -> OpenSearch infrastructure using the AWS CDK v2. In my CDK Stack I have created an [OpenSearch `Domain`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_opensearchservice.Domain.html), and am trying to create a Kinesis Firehose [`DeliveryStream`](https://docs.aws.amazon.com/cdk/api/v2/docs/@aws-cdk_aws-kinesisfirehose-alpha.DeliveryStream.html) with that domain as the destination. However, kinesisfirehose-destinations package seems to only have a ready-to-use [construct for S3 buckets](https://docs.aws.amazon.com/cdk/api/v2/docs/@aws-cdk_aws-kinesisfirehose-destinations-alpha.S3BucketProps.html), so there is no obvious way to do this easily. I have tried implementing my own [`IDestination`](https://docs.aws.amazon.com/cdk/api/v2/docs/@aws-cdk_aws-kinesisfirehose-alpha.IDestination.html), but I am at a complete loss since I have no idea how to implement `bind`. So, my questions are: 1. How are you supposed to implement `bind` in `IDestination`? 2. Are there any complete working examples of creating a Firehose to OpenSearch using the non-alpha L1 constructs?

I started to notice there are random error throw at my OpenSearch Log with : ``` Cannot index event publisher.Event ``` My cluster, indices are all in health state, with no jvm or cpu above 80%. Not sure what is causing it, and in the index block, i see ``` `"blocks" : { "write" : "false" },` ``` Wondering if because it is missing the setting ` "read_only_allow_delete" : "false",`

I have a pipeline resolver for an AppSync Mutation. It contains two functions, the first one is a Lambda sending updates to RDS, the second one should take the result from `$ctx.prev.result` and index it via OpenSearch datasource. In the request resolver mapping template of the second one, I am composing the bulk body in NDJSON similar to the following mannar: ```vtl #set($bulk = $util.toJson({ "index": { "_id": "${$ctx.prev.result.id}" } })) #set($bulk = "${bulk} ${util.toJson($ctx.prev.result)}") { "version": "2017-02-28", "operation": "POST", "path": "/_bulk", "params": { "body": $util.toJson($bulk) } } ``` Lacking proper debugging tools, I have been using `$util.error` as a logging method to get my `$bulk` contents. And it looks like the following format, which seems correct. ```ndjson {"index":{"_id":"A8DEF210-C342-48CB-9A4A-DA7D1E4D6AF1"}} {"foo":123,"bar":999,"baz":1234567} ``` But when I actually runs the mutation via AppSync, I got a `MappingTemplate` error `Unable to transform for the body: $[params][body].` and I have no idea why. EDIT: I took a look at [[re:Post] Appsync HTTP resolver supported content types](https://repost.aws/questions/QUFqBom4iFQ-6ePM1z11Sthw/appsync-http-resolver-supported-content-types), which inspired me to take another look at [Resolver Mappping Template for OpenSearch (params)](https://docs.aws.amazon.com/appsync/latest/devguide/resolver-mapping-template-reference-elasticsearch.html#params-field). It seems POST body only accepts a single JSON object, NSJSON required by the bulk request is not supported yet. Am I correct? If so, is supporting the bulk API in the upcoming plans? Also, what is the currently recommended way to index multiple "normalized" documents from the same resolver?

Hello, I have setup a domain that uses IAM auth via cognito. I created the below ingest pipeline. However, the user property is always an empty string. Does aws open search support the _user security variable. ``` PUT _ingest/pipeline/post { "description": "post", "processors": [ { "set": { "description": "Add user info", "field": "user", "value": "{{_user.username}}" } } ] } ```

I'm trying to connect AWS OpenSearch to QuickSight and export data to SPICE, but I get ELASTICSEARCH_CURSOR_NOT_ENABLED exception. I was not able to find how to enable cursor for opensearch distro 1.1 - region:eu-west-1 - timestamp:1649003264494 - requestId:95b8a137-9839-453c-affc-f50c0e32ae4e - sourceErrorCode:ELASTICSEARCH_CURSOR_NOT_ENABLED - sourceErrorMessage:The domain monei-prod has not enabled cursor.

Currently, I have an OpenSearch cluster that has a **Max Clause Count **of 1024. I want this value updated to 4096. I have updated the value within the cluster configuration settings and the dashboard has confirmed that the update was completed. However, when testing, I get the following error: `{"type":"too_many_clauses","reason":"maxClauseCount is set to 1024"}` Indicating that the update was somehow not successful. Any suggestions as to what I need to do to remedy this? Thanks!

Understanding Active Passive Cross Cluster replication for AWS Open Search[1] is pretty straightforward. You have a leader index in the active cluster that replicates everything to the follower index in the passive cluster. In case there is a DR and we witness a region failure, we can continue to process read operations through the follower index on the passive cluster. However, the documentation provided does not go through the following two points. 1. How do we handle write operations that come during the time the active cluster is down. Is it possible to write on the passive cluster during this time and the primary would catch up when it comes back online or only read is possible during this time, and we need to batch process all the writes once active cluster comes back up ? 2. When active cluster comes back online, does Open Search automatically starts redirecting search queries back to active cluster instead of the passive cluster ? 3. In addition, in case we wish to test the DR failover setup for Open Search, is it possible to simulate active cluster failure and test ? [1] https://docs.aws.amazon.com/opensearch-service/latest/developerguide/replication.html

Hi, We are designing a data indexing pipeline to OpenSearch using Firehose, like : data source ----(putRecordBatch)----> firehose ---(Lambda transformation)--> OpenSearch New data added to data store will be indexed via this flow in near-real time. There is requirement of both creating new documents, and doing partial update on existing documents(ie, updating select attributes of a specified document id) Does Firehose integration with OpenSearch support Upsert(Update if document is seen in OpenSearch, else insert)? Thanks

I'm working on Centralized Logging of my environnement AWS presented in this guide:https://docs.aws.amazon.com/solutions/latest/centralized-logging/overview.html I'm using the same architecture as presented in AWS Doc. I'm searching the possibility to send metrics from CloudWatch of each account to OpenSearch in centralized account. I was thinking of a Lambda function that retrieves each X minutes the data with get_metric_data ? I'm collecting the metrics with CloudWatch Agent Is there an another method to do it ? Thank you

Hello 1. I was facing ES EBS space issue (Prod Env) since last few days. Further i have increased ES EBS volume 10 to 128 GB using "Configure cluster", keep all configration same as, only increased EBS size up, and submitted. now 24 hours has been gone, still i can see Domain Status "Processing" . Could please help me how to fix it? or why is it taking too much time. 2. I tried to reset the settings and now I have next message *UpdateDomainConfig: {"message":"Update has failed. Reason(s): The Cognito User Pool does not have the domain name set. Please set it and try again.The Cognito User Pool configured with this domain is unavailable. Please update the User Pool ID and try again. If the issue persists, please contact AWS Support."}* 3. I reset the ES security configuration and I still have the same message. so I do not have any Cognito configuration in my ES, but it still asks me about the domain name

Hello I was facing ES EBS space issue (Prod Env) since last few days. Further i have increased ES EBS volume 10 to 128 GB using "Configure cluster", keep all configration same as, only increased EBS size up, and submitted. now 24 hours has been gone, still i can see Domain Status "Processing" . Could please help me how to fix it? or why is it taking too much time.

What I'm trying to do: 1. I am using a kinesis data stream to ingest data from a python client sending in JSON's. 2. I setup a kinesis firehose delivery stream with source as the kinesis-data-stream from the previous step and destination as an index on opensearch. I also use a lambda to transform the stream before delivering to opensearch. Now i would like to set the document id for the record I'm transforming in the lambda . I tried setting the key on the transformed record object to be `id` but that creates a document attribute named `id` and sets a value to it. What i would like is to set the _id on the search doc.When i try to set the _id attribute directly by returning it within the transformed record back to firehose delivery stream it generated a destination error : ``` "message": "{\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [_id] of type [_id] in document with id \\u002749627661727302149016659535428367209810930350656512327682.0\\u0027. Preview of field\\u0027s value: \\u00279b428743-b19a-4fb1-90f2-f132a423b2e8\\u0027\",\"caused_by\":{\"type\":\"mapper_parsing_exception\",\"reason\":\"Field [_id] is a metadata field and cannot be added inside a document. Use the index API request parameters.\"}}", "errorCode": "400", ``` Is there anyway to the doc _id for the documents loaded from firehose to opensearch or would i need to take a different approach by selecting the destination to be a http endpoint and using the rest API's provided by Opensearch (which would kinda be a hassle compared to directly being able to just set the _id attrib) ? What I'm really trying to do is update the indexed documents on a change event. I understand that firehose uses the bulk api from Opensearch , but am unsure about how the upsert operation is handled internally by the kinesis destination connector to opensearch. Hence specifying a fixed id from another DB would be ideal for both insert and update ops to Opensearch in my case.It would be super useful to atleast be able to dictate the type of operation based on some attribute of the kinesis record to opensearch with a reference id for the doc to update.

We still seem to be experiencing the bug outlined [this question](https://www.repost.aws/questions/QUDlAar3N6QqCE7o3YDVI1dQ/open-search-sns-destination-support-for-alerting), despite being updated to v1.1 P5. One of the workarounds suggested was to use the API to create the destination, but, since we don't have any existing SNS destinations, I'm not sure what exactly the syntax is. I managed to get a destination created using the following, but it's registering as an "unsupported type". Not sure if that's a symptom of the bug or if I screwed up the syntax of the alert destination creation. ``` POST _plugins/_alerting/destinations { "name": "sns destination", "type": "sns", "sns": { "topic_arn":"arn:aws:sns:us-east-1:000000000000:opensearch-alerting-topic", "role_arn": "arn:aws:iam::000000000000:instance-profile/opensearch-sns" } } ```

Hi folks, we have created an Elasticsearch (version 7.10) cluster with Cognito enabled fined grained authentication for Kibana. We have replicated (via terraform IaC) a configuration already present on another AWS account that is working fine. In this new cluster we have Kibana session issues, for example when we click "Log out" in Kibana we got an 403 error (I'll post it at the end of my message as LOGOUT_ERROR). Another error happens when the Cognito session token expires and we got another 403 error (another small snippet at the end of my message as LOGIN_ERROR). We cannot see the Cognito login page if we don't remove the cookie in the browser (we tested different browsers). Everything else is working fine: we are able to use Elasticsearch and Kibana with no other issues. Since the error is originating from "AWSSecurityTokenService" (an interface in the AWS Java SDK) we smell this as a possible bug that we're not able to address on our side. The fact that the Elasticsearch cluster is in eu-south-1 region and Cognito is in eu-west-1 could be an issue? Can you please help us by pointing us to any resource that may help resolve this issue? Regards Marco LOGOUT_ERROR ``` <!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="description" content="403 Forbidden"> <title>Kibana Authentication Error</title> <link rel="stylesheet" href="//maxcdn.bootstrapcdn.com/bootstrap/3.3.4/css/bootstrap.min.css "> <link rel="stylesheet" href="//maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css "> <script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js "></script> <script src="//maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js "></script> <style> body { padding-top: 20px; } .jumbotron { font-size: 21px; font-weight: 200; line-height: 2.1428571435; color: inherit; padding: 10px 0px; } .body-content { padding-left: 15px; padding-right: 15px; } .jumbotron { text-align: center; background-color: transparent; } .jumbotron .btn { font-size: 21px; padding: 14px 24px; } .blue {color:#00bfff;} .red {color:#d9534f;} </style> </head> <div class="container"> <div class="jumbotron"> <h1><i class="fa fa-frown-o red"></i> Sorry!</h1> <p class="lead">Something went wrong during authentication between Kibana and Amazon Cognito.</p> <p><a href="https://kibana.linkemswarm.com/_plugin/kibana " class="btn btn-default btn-lg"><span class="blue">Log in to Kibana</span></a> </p> </div> </div> <div class="container"> <div class="body-content"> <div class="row"> <h2>What happened?</h2> <p>cognito:revoke_tokens:ErrorUser: x:x:x::xx:x is not authorized to perform: sts:AssumeRole on resource: x:x:x::xx:x (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied; Request ID: 432dbf6a-126f-43f4-be02-69c3657b3016; Proxy: null)</p> </div> <div class="row"> <h2>What should I do?</h2> <p>Try logging in again. If the problem persists, please review the <a href="https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-cognito-auth.html#es-cognito-auth-troubleshooting ">troubleshooting guide </a>for information on resolving common issues.</p> </div> </div> </div> </body> </html> ``` END LOGOUT_ERROR LOGIN_ERROR ``` ... <h2>What happened?</h2> <p>com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: User: x:x:x::xx:x is not authorized to perform: sts:AssumeRole on resource: x:x:x::xx:x (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied; Request ID: d3bbebd3-a5ba-4406-9d75-5b78cd1c92bc; Proxy: null)</p> ... ``` END LOGIN_ERROR

ElasticSearch Service have a useful metric **JVM memory pressure** that specifies the percentage of the Java heap occupied in a cluster node. **CMSInitiatingOccupancyFraction** is the threshold value on which Garbage collection activity starts to clean deleted items from ES node memory to reduce JVM Memory pressure. [refer article for more information](https://medium.com/naukri-engineering/garbage-collection-in-elasticsearch-and-the-g1gc-16b79a447181) Our application have clean up tasks executed on weekly basis and we are exploring options to lower down CMSInitiatingOccupancyFraction (which is by default set to 75%) value. As cleanup tasks gets executed we are thinking of setting this threshold to 65-70%. We didn't found any configuration options to fix this value. Is there any other mechanism to activate Garbage collector? Does AWS Elastic search service provide options to configure these default values? If yes then guide us through steps.

I went to bed last night and everything was ok, woke up this morning and tried to start working and All my calls to Opensearch where timing out: opensearchpy.exceptions.ConnectionTimeout: ConnectionTimeout caused by - ReadTimeout(HTTPSConnectionPool(host='search-XXXXXXXX-jt5tcucdjcsa-xedweehyupm7z3zn5q6tqrt57q.us-east-2.es.amazonaws.com', port=443): Read timed out. (read timeout=10)) e65029b2-6ce0-4b34-a137-a6f598bfe877 ERROR ConnectionTimeout: ConnectionTimeout caused by - ReadTimeout(HTTPSConnectionPool(host='search-XXXXXXXXXX-jt5tcucdjcsa-xedweehyupm7z3zn5q6tqrt57q.us-east-2.es.amazonaws.com', port=443): Read timed out. (read timeout=10)) Checked the service health dashboard and no errors listed there. Went to the Opensearch Dashboard in the AWS UI and The cluster Health field was empty as was Searchable Doucments, total Free Space, and minimum free space. Clicking on Cluster Health gives a 404 Source View in the summary and no other useful information. There is no useful button anywhere like reboot cluster. I am at a complete loss of what to do. Luckily this is a development environment, but if this is happening here, what's to keep this from happening in a production enviornment? Anyone have any ideas? BTW as an aside, why can't we post screenshots on this support site? As a second aside the tag for Opensearch doesn't exist you have to use elasticsearch

Hi Everyone, We use Elasticsearch 7.9, time to time we may need to do re-indexing. We are using aliases and the applications are pointing to alias instead of real index. I read some articles and it looks in most cases we would lose some update's/deletes that happen during the re-indexing process. based on what I read, Below are the steps that I am planning to do. Note : old_index is pointing to index-alias(it's alias name) 1- create new index(new_index) 2- do re-indexing from old_index to new_index 3- point the alias(index-alias) to new_index 4- Do second round of re-indexing from old_index to new_index below are some of issues with this approach: 1- Two round of re-indexing needs to be done. This will become problematic, if we have large data in that index. 2- After step (3) , we may lose some of the deletes/updates on new_index. What is the best method to do re-indexing with zero downtime and minimal impact for end-users? Thanks in advance.

This question was targeted at collect following blog's questions --- 1. Amazon WAF Implementation Guide (1) WAF priciple, default implemtation and log store 2. Amazon WAF Implementation Guide (2) A cost effective way - Log Insight 3. Amazon WAF Implementation Guide (3) Leverage OpenSearch to start security investigation 4. Amazon WAF Implementation Guide (4) Use Log Hub solution to start WAF security operation

Hi Everyone, We use Elasticsearch 7.9 and on monthly bases we create new indexes. Generally the last 4-5 indexes(months) receive large number of updates, so we end up with lots of deleted documents. I read that in order to improve the search performance, we have to do force merge. But from other side its been recommended to do the force merge on indexes that don't receive any write. Here are my questions. 1- Per (https://aws.amazon.com/premiumsupport/knowledge-center/opensearch-deleted-documents/) merge cause blocking for whole cluster, so how can we periodically do the merging? Wont this cause any prolem for end-user? 2- Since in our case the indexes still receive write, is there any work around to eliminate the deleted documents? Thanks in advance.

I'm trying to move an Elasticsearch domain from one VPC to another. The Elasticsearch domain contains two data nodes, each in a different AZ, and I'm using a ZoneAwarenessConfig with enabled set to True. My update fails with: > The subnets must be in the same VPC. (Service: AWSElasticsearch; Status Code: 400; Error Code: ValidationException; Request ID: 1382cf1e-961a-40c0-b336-c43887d07c73; Proxy: null) The change is itself is incredibly simple, the diff from the CDK is: ``` [~] AWS::Elasticsearch::Domain Elasticsearch/DocumentSearch ElasticsearchDocumentSearch78F67497 └─ [~] VPCOptions └─ [~] .SubnetIds: └─ @@ -1,8 +1,8 @@ [ ] [ [ ] { [-] "Ref": "VpcPrivateSubnet1Subnet67A4DBCB" [+] "Ref": "VpcTempVpcPrivateSubnet1SubnetF8F05905" [ ] }, [ ] { [-] "Ref": "VpcPrivateSubnet2SubnetC8EB537D" [+] "Ref": "VpcTempVpcPrivateSubnet2SubnetFF8C4018" [ ] } ``` The new subnets are both in the same VPC. It's hard to read this as anything other than a bug in Cloudformation. It's also not possible to remove an Elasticsearch domain from a subnet. The update of the AWS::Elasticsearch::Domain fails with: > Internal Failure The change once again is trivially simple: ``` [~] AWS::Elasticsearch::Domain Elasticsearch/DocumentSearch ElasticsearchDocumentSearch78F67497 └─ [-] VPCOptions └─ {"SecurityGroupIds":[{"Fn::GetAtt":,"SubnetIds":} ```

Hi, The AWS docs at https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ism.html#:~:text=OpenSearch%20Service%20supports%20three%20unique%20ISM%20operations%2C%20warm_migration%2C%20cold_migration%2C%20and%20cold_delete state there are 3 special actions supported by ISM for Amazon OpenSearch Service: `OpenSearch Service supports three unique ISM operations, warm_migration, cold_migration, and cold_delete.` However, I can only see warm_migration in the OpenSearch Policy Management Visual Editor. See https://imgur.com/a/kzZscks If I use the API to create a policy including cold_migration and cold_delete, it succeeds, but I can't view it in the UI. So I am unsure if this is really supported, or if the UI hasn't been updated to reflect this. Curious to hear others thoughts on this.

Hello, Our cluster is in update status since the 16th of December 2021 and is stuck at 50%: ``` Upgrade from 7.10 to OpenSearch_1.0 In progress Thu, 16 Dec 2021 20:02:38 GMT Check eligibility In progress (50%) - ``` What can we do and what is the status of the upgrade? Thank you

Hello, Is there any way to enable cors on an open search public domain? If not are there any plans for possible future support. It would be ideal if clients could communicate with open search domains using zero-trust signed urls directly in the browser rather than requiring a proxy. This is not possible without a proxy at the moment due to cors limitations. However, on other services like s3 direct secure communication with s3 rest api can be achieved using zero-trust signed urls directly in the browser. It would be nice to be able to do the same thing with open search domain. Thanks

1) IHAC would like to enable cross-cluster search, but the docs limit that to clusters created after June 2020. How to understand the limitation? Is that something we can work around? Creating a new cluster for their 100B documents is going to be quite painful. 2) What a given autotune actually does? Right now it’s just lines on a graph, and sometimes the graph gets worse afterwards. 3) What would it take to get shard allocation control? Not being able to control shard routing has been a real painpoint for them when allocation goes wrong. 4) Similar, what’s the recommended way to kick a bad node from the cluster? Really appreciate your help!

We have wrote custom plugins, example - https://logz.io/blog/opensearch-plugins/ Is it possible to add additional/custom plugins on AWS managed OpenSearch? Without this we won't be able to migrate to AWS. If not allowed currently, any idea if it's going to be included in future? Thanks!

I've loaded 1600+ JSON files via the Bulk API into an OpenSearch domain and have verified that via the Developer Tools console. Now I wanted to use the _dashboards interface to create an index pattern and then use the 'Discover' option to search. I'm following the instructions from here --> https://docs.aws.amazon.com/opensearch-service/latest/developerguide/gsgsearch.html I am able to successfully create the index pattern, but then when I click on 'Discover', it takes me to the 'Create Index Pattern' screen again. Hopefully there is something trivial that I've overlooked. Thanks.

Is it possible to to enable compatibility mode on existing Opensearch domain? I found this article https://docs.aws.amazon.com/opensearch-service/latest/developerguide/rename.html and added in the cluster settings which has changed the master nodes but not the data nodes. I tried scaling the nodes up but they still report the opensearch version number instead of the equivalent Elasticsearch version.

On 2/4, it seems that all my searchable documents in Elasticsearch got deleted automatically by itself \[1]. Was AWS doing some upgrading/migration at the time that causing it? Is there any ways I could do to recover all the documents? \[1] [Link to screenshot](https://ibb.co/wLDcL8t)

When calling endpoints beneath the `/_security/*` endpoint I get the response: > {"Message":"Your request: '/_security/..' is not allowed."}

Hello, I am currently upgrading my OpenSearch instance and it has been stuck at "Check eligibility" with status " In progress (28.57%)" for multiple hours. Is it expected to take so long? Blue-green deployments on that cluster usually complete within an hour. I have looked at the [OpenSearch FAQ](https://aws.amazon.com/opensearch-service/faqs/) and there is a question that mentions the following: > For detailed list of the tests we run to validate upgrade eligibility, please refer to [our documentation](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html). However, the documentation mentions no useful information about what happens during that eligibility check. Not sure what to do here. Wait for a full day? Does it eventually timeout?

## Main problem I understand that is no need to add Auto Scaling to an EMR cluster launched by Data Pipeline. Instead, we can specify the **capacity up-front** and it will be used for the duration of the job. But what if I am doing transformation on some data weekly basis and the instance type size needed do this every week, and I can't be sure about how many nodes are required in the cluster for better performance? ### Possible solution? At the moment, I can predict the amount of data that EMR could process thanks to the number of events tracked in a period of time on **OpenSearch** (where EMR will extract data), E.g. *If 1 EMR node can handle 1,000 events and the actual number of events was 10,000 then create 10 nodes*. I though on create an EventBridge cron job to execute a Lambda function ~10 minutes before the **Data pipeline** process and calculate the number of nodes, then store the value in a service like SSM parameter store. So when the Data Pipeline starts, I can be able to retrieve the value and pass it as a parameter for the task. This may sound a little complicated so I would like to know if there's maybe an easier way to achieve this, thanks in advance!

I have an Amplify app that uses Dynamodb database and an OpenSearch cluster indexes the dynamodb database. Anytime my Amplify app build fails, it erases the OpenSearch index and makes my app unusable. This is turning out to be disastrous for my app, does anyone know to prevent the OpenSearch index from dropping?

Hi, I tried to delete one of my ElasticSearch services 2 days ago and it has been stuck at 0% on "being deleted" since then. None of the options on the console seem to be enabled since it is "being deleted" and the instance always returns a 502 when I try to anything via the API.

When I use OpenSearch v1.0 (R20211203-P2 - latest as of 1/30/22), I have the SNS Destination available to me (for alerting notifications). In v1.1 (again latest as of 1/30/22), I don't see SNS as Destination - only the usual Elasticsearch ones are available I'm confused because documentation says SNS as Destination is available if I use OpenSearch v1.1 (again to clarify, I'm not using Elasticsearch BUT OpenSearch). Any insights?

We have been using the original ElasticSearch for a few years, works great. Today it stopped working. When I logged into the AWS console and went to the OpenSearch page our domain had a status of Processing and at first I thought it was stuck but after several minutes it finished. It then did the whole update process again which took another several minutes. Now it is back to Active but it still isn't working. I think the indices have been deleted? I have no documents any more. Is there a back up to restore from? under the Cluster configuration it says Snapshots - frequency - hourly, but I don't see anyway to restore it. Under the error logs I see: ``` [2022-01-29T09:05:58,417][WARN ][o.e.c.NodeConnectionsService] [bmDFGK_] failed to connect to node {5fSCYAe}{5fSCYAeFQbefAxIiS7YyVA}{03U8UySaTI6gDrlMk5RP2Q}{__IP__}{__IP__}{__AMAZON_INTERNAL__, __AMAZON_INTERNAL__, distributed_snapshot_deletion_enabled=true} (tried [1] times) ConnectTransportException[[5fSCYAe][__IP__] connect_exception]; nested: AnnotatedConnectException[Connection refused: __PATH__]; nested: ConnectException[Connection refused]; at org.elasticsearch.transport.TcpChannel.awaitConnected(TcpChannel.java:165) at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:643) at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:542) at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:316) at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:153) at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:106) at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:749) Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: __PATH__ at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633) at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ... 1 more Caused by: java.net.ConnectException: Connection refused ... 10 more [2022-01-29T09:05:58,417][WARN ][o.e.c.NodeConnectionsService] [bmDFGK_] failed to connect to node {5fSCYAe}{5fSCYAeFQbefAxIiS7YyVA}{03U8UySaTI6gDrlMk5RP2Q}{__IP__}{__IP__}{__AMAZON_INTERNAL__, __AMAZON_INTERNAL__, distributed_snapshot_deletion_enabled=true} (tried [1] times) ConnectTransportException[[5fSCYAe][__IP__] connect_exception]; nested: AnnotatedConnectException[Connection refused: __PATH__]; nested: ConnectException[Connection refused]; at org.elasticsearch.transport.TcpChannel.awaitConnected(TcpChannel.java:165) at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:643) at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:542) at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:316) at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:153) at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:106) at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:749) Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: __PATH__ at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633) at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ... 1 more Caused by: java.net.ConnectException: Connection refused ... 10 more ```

Our ElasticSearch Domain unexpectedly stopped responding. Since we cannot look into the logs anymore we do not have the chance to find out the reason for that. We noticed that the ebs volume appeared to be at 3GB size, however we cannot verify if this indeed is the reason why the elasticsearch domain is not responsive anymore. We also do not see any possibility to find out why this happend (as mentioned above, logs are not accessible). We started an update of the ebs volume to double in size using the AWS webconsole, as we expected that new space might help. The update was marked as 'green' so we agreed on starting it. Even after 24 hours the update is stuck in the state 'Processing'. As described in the documentation, this means that the update of the ebs volume failed (https://aws.amazon.com/premiumsupport/knowledge-center/opensearch-domain-stuck-processing/). We have found a long list of reasons for why this update may have failed: - CDI Failure: creation of new DI failed - Customer makes simultaneous configuration changes on the cluster - Failed CDI due to overloaded mastr node or any other activity failures like insufficient IP addresses in the customer subnet - Nodes went out of service due to heavy processing load or internal hardware failure - Previous DDI Failure: New CDI executed before Previous DDI activities are completed. - Large number of shards and continues node failure due to high JVM memory Pressure and CPU Usage. - Nodes went out of service due to heavy processing load or internal hardware failure - Stuck shard relocation due to insufficient free storage in the new DI, custom shard routing and service/es issues - Single domain can usually have 2 DIs at maximum. If CDI happens before previous DDI is completed, new CDI is queued but won't be executed. But this list does not help to solve the issue that our ElasticSearch Domain is currently in a state that we cannot change. The only action that we see is deleting the domain and start a new one. However we would like to ask if there is some action that AWS can do for bringing the domain back to life. We need the logs that are within the domain.

Our Opensearch cluster is stuck in "processing" for more than 8 hours now & does not seem to be recovering. I am not able to make any changes because of this. My cluster is in **EU-WEST-2(London)** region. How do I get this domain to go back to normal?

Hi, Who do I request to change the ElasticSearch search.max_open_scroll_context parameter to unlimited? Thanks.

I'm migrating some data from postgres to OpenSearch, but i'm struggling with migrating a set of coordinates. In postgres i have `latitude` and `longitude`, and i know DMS does not support geo types when using OpenSearch as a target. I wanted to work around this by using an ingest pipeline and an index template: ``` PUT _ingest/pipeline/my-pipeline-id { "description": "My optional pipeline description", "processors": [ { "set": { "field": "location.lon", "value": "{{{longitude}}}" } }, { "set": { "field": "location.lat", "value": "{{{latitude}}}" } } ] } ``` ``` PUT /_index_template/jobs_template { "index_patterns": [ "jobs*" ], "template": { "settings": { "index.default_pipeline": "my-pipeline-id" }, "mappings":{ "properties": { "location": { "type": "geo_point" } } } } } ``` I first tested having only the pipeline without the mappings and that part works. However, when i add a mapping to the template, and re run the migration task i get the following error ``` 2022-01-20T15:23:01 [TARGET_LOAD ]E: Elasticsearch:FAILED SourceTable:jobs TargetIndex:jobs Operation:INSERT_ENTRY RecordPKKey:93904e3c-5565-4469-94d6-e58fbecdc5a3 RecordPKID:217D5CE32D4EC983FE2C3CFD6048821EA2A95F3658122A80A7EEB3A6088EA89CES HttpCode:400 ESErrorResponse: { "error": { "root_cause": [ { "type": "illegal_argument_exception", "reason": "Rejecting mapping update to [jobs] as the final mapping would have more than 1 type: [_doc, doc]" } ], "type": "illegal_argument_exception", "reason": "Rejecting mapping update to [jobs] as the final mapping would have more than 1 type: [_doc, doc]" }, "status": 400 } ``` Is there any way to make this work? Any way to create a geo point so I can perform geo queries in OpenSearch?

Source: postgres Target: OpenSearch I have a `text` column called `description` in one of my postgres tables. Per the documentation, this data type is mapped to a `NCLOB`. Since OpenSearch does not not offer LOB support, my `description` is missing in my OpenSearch documents. I tried using the mapping rule bellow, but does not seem to be doing anything ``` { "rule-type": "transformation", "rule-id": "3", "rule-name": "3", "rule-target": "column", "object-locator": { "schema-name": "public", "table-name": "jobs", "column-name": "description" }, "rule-action": "change-data-type", "data-type": { "type": "string", "length": 500 } } ``` When i check the logs i see the following ``` Column 'description' is unsupported in table def 'public.jobs' since the LOB support is disabled ``` However, i do have LOB enabled under task settings: ``` "TargetMetadata": { "ParallelApplyBufferSize": 0, "ParallelApplyQueuesPerThread": 0, "ParallelApplyThreads": 0, "TargetSchema": "", "InlineLobMaxSize": 0, "ParallelLoadQueuesPerThread": 0, "SupportLobs": true, "LobChunkSize": 10, "TaskRecoveryTableEnabled": false, "ParallelLoadThreads": 0, "BatchApplyEnabled": false, "FullLobMode": true, "LimitedSizeLobMode": false, "LoadMaxFileSize": 0, "ParallelLoadBufferSize": 0 }, ``` Is that transformation rule supposed to work? Or will any LOB column be skipped because OpenSearch does not have LOB support? Any way to make this work? Thanks!

I have an OpenSearch cluster in VPC A, I need applications deployed in VPC B and VPC C to access OpenSearch in VPC A. The problem is that VPC B and VPC C have the same IP range (even the subnets have the same range and I can't change it), so I cannot do a vpc peering or use transit gateway. A solution would be to put an Nginx proxy in VPC A and then via VPC endpoints allows applications in VPC B and VPC C to access Nginx, but I'd like to avoid to need to support the Nginx proxy. Any other solution?

I am running this kibana query: I have this query in Kibana: GET nearby/_search { "from": 20, "size":20, "query": { "bool": { "must": { "match": { "X": "B" } }, "filter": { "geo_distance": { "distance": "3.0km", "PO": { "lat": 26.8466937, "lon": 80.94616599999999 } } } } } } and response to this is: all the responses are with X=B: 20 results are there, i have removed some fields and some docs to keep the post short { "took" : 228, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 71, "relation" : "eq" }, "max_score" : 2.5032558, "hits" : [ { "_index" : "nearby", "_type" : "_doc", "_id" : "n3YeKvJqvpu1okE7QDBp", "_score" : 2.2831507, "_source" : { "PO" : "tuc89gfn0", "X" : "B" } }, { "_index" : "nearby", "_type" : "_doc", "_id" : "5FPJ2eyr0YoQ9F0xPYzW", "_score" : 2.2831507, "_source" : { "PO" : "tuc89gfn0", "X" : "B" } }, { "_index" : "nearby", "_type" : "_doc", "_id" : "QJflnqGKF1dpOjEaY8vy", "_score" : 2.2831507, "_source" : { "PO" : "tuc89gvr8", "X" : "B" } }] } } This is the browser REQUEST, QUERY REMAINS SAME: https://search-wul8888888.ap-south-1.es.amazonaws.com/nearby/_search?q="{"from":20,"size":20,"query":{"bool":{"must":{"match":{"X":"B"}},"filter":{"geo_distance":{"distance":"3km","PO":{"lat":26.8466937,"lon":80.94616599999999}}}}}}" This is the response: as u can see there are mostly X=I docs i.e. must-match isnt honoured, SECOND THING IS THAT I AM SENDING SIZE=20 BUT I GET 10 REULTS ONLY WHICH IS DEFAULT(BELOW I HAVE REMOVED EXTRA docs TO KEEP THE POST SHORT) {"took":149,"timed_out":false, "_shards":{"total":5,"successful":5,"skipped":0,"failed":0}, "hits":{"total":{"value":802,"relation":"eq"},"max_score":8.597985, "hits":[ {"_index":"nearby","_type":"_doc","_id":"iJ71MNq4a4TCkcT4vWSP","_score":8.597985,"_source":{//EXTRA FIELDS REMOVED "PO":"tuc8unwp7","X":"I","BI":"tRhKrWiDxFSt57tIH7g5"}}, {"_index":"nearby","_type":"_doc","_id":"PmngNe8WcC8aSraDMluz","_score":7.3973455,"_source":{"PO":"tuc8uhc5z",**"X":"I"**,"BI":"m3S6yEicvu1HFI1UOTIb"}}, {"_index":"nearby","_type":"_doc","_id":"lDqjflPZGYsymPGU8iHD","_score":7.1520696,"_source":{"PO":"tuc89wpg5","X":"B"}}, {"_index":"nearby","_type":"_doc","_id":"QIf2KsO4FpCjT3m7kH4I","_score":6.402881,"_source":{"PO":"tuc8uhc5z","X":"I","BI":"m3S6yEicvu1HFI1UOTIb"}}]}} PLEASE HELP. I TRIED EVERYTHING BUT NOT ABLE TO UNDERSTAND. MY HUNCH IS THAT EVERY TIME I M BEING RETURNED A STALE/old RESULT BUT dont know how to fix that. even in chrome incognito mode result for browser is same as above. Even if i change the radius in browser, result remains same which says clearly browser queries are getting the stale result.

My OpenSearch cluster has been stuck in processing since the last auto-tune event. the cluster status is green across the board. The cluster is usable without issue (reading, writing, Kibana), but this prevents me from performing an upgrade or applying other config changes. Monitoring shows: * Cluster status green * Instance count is 9, as expected: 3 master and 6 data nodes * JVM memory pressure looks good: seeing the expected "sawtooth" curve never exceed 75% and going as low as 45% * Most recent update was a Service software update to R20211203-P2. It seems to have taken 5 days, but looks like it completely well. (Judging by the instance count graph) * The cluster is usable without issue, Kibana is reachable and responsive, constantly writing to the cluster without error, nothing seems off Rough timeline: * 19.12.2021 - update to R20211203-P2, instance count is doubled to 18 (expected blue/green deployment) * 24.12.2021 - instance count drops back to the expect 9, cluster status green * 26.12.2021 - Notification "Auto-Tune is applying new settings to your domain", instance count doesn't rise, still at 9 * now - Cluster still stuck at "processing" even though everything is green What I tried: * `GET /_cluster/allocation/explain` responds with "unable to find any unassigned shards to explain" which makes sense * ` GET /_cat/indices?v` shows everything green, as expected * I also tried modifying the disk size to try and "kick" the cluster into doing a blue/green deployment and hopefully getting unstuck but that didn't seem to happen The only possible clue was in CloudWatch error logs, a repeating message appears since the last auto-tune event started on 26.12.2021: with "master not discovered yet", I'll try to pretty-print it below: ``` [2022-01-11T06:36:23,761][WARN ][o.o.c.c.ClusterFormationFailureHelper] [52cb02d8573b17516f7756d5fe05484d] master not discovered yet: have discovered [ {***}{***}{***}{__IP__}{__IP__}{dir}{dp_version=20210501, distributed_snapshot_deletion_enabled=false, cold_enabled=false, adv_sec_enabled=false, __AMAZON_INTERNAL__, shard_indexing_pressure_enabled=true, __AMAZON_INTERNAL__}, {***}{***}{***}{__IP__}{__IP__}{imr}{dp_version=20210501, distributed_snapshot_deletion_enabled=false, cold_enabled=false, adv_sec_enabled=false, __AMAZON_INTERNAL__, shard_indexing_pressure_enabled=true, __AMAZON_INTERNAL__}, {***}{***}{***}{__IP__}{__IP__}{imr}{dp_version=20210501, distributed_snapshot_deletion_enabled=false, cold_enabled=false, adv_sec_enabled=false, __AMAZON_INTERNAL__, shard_indexing_pressure_enabled=true, __AMAZON_INTERNAL__}, {***}{***}{***}{__IP__}{__IP__}{imr}{dp_version=20210501, distributed_snapshot_deletion_enabled=false, cold_enabled=false, adv_sec_enabled=false, __AMAZON_INTERNAL__, shard_indexing_pressure_enabled=true, __AMAZON_INTERNAL__}, {***}{***}{***}{__IP__}{__IP__}{imr}{dp_version=20210501, distributed_snapshot_deletion_enabled=false, cold_enabled=false, adv_sec_enabled=false, __AMAZON_INTERNAL__, shard_indexing_pressure_enabled=true, __AMAZON_INTERNAL__}, {***}{***}{***}{__IP__}{__IP__}{imr}{dp_version=20210501, distributed_snapshot_deletion_enabled=false, cold_enabled=false, adv_sec_enabled=false, __AMAZON_INTERNAL__, shard_indexing_pressure_enabled=true, __AMAZON_INTERNAL__}, {***}{***}{***}{__IP__}{__IP__}{imr}{dp_version=20210501, distributed_snapshot_deletion_enabled=false, cold_enabled=false, adv_sec_enabled=false, __AMAZON_INTERNAL__, shard_indexing_pressure_enabled=true, __AMAZON_INTERNAL__} ]; discovery will continue using [__IP__, __IP__, __IP__, __IP__, __IP__, [__IP__]:9301, [__IP__]:9302, [__IP__]:9303, [__IP__]:9304, [__IP__]:9305, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__] from hosts providers and [] from last-known cluster state; node term 36, last-accepted version 0 in term 0 ``` I masked the node IDs and replaced them with `***`. The log message lists 7 of them above, I can only recognize 3 IDs as my master nodes, cannot recognize the rest of the IDs (not my data nodes) and not sure I understand what's going on here. Any help would be appreciated.

Hi everyone, I was trying to make an alerting system that every time a specific value is stored in one OpenSearch index, that triggers an email alarm configured with AWS Simple Notification Service. The problem is that, when I go to create the destination, in the “Alerting” configuration section, it’s supposed to show me the option to integrate it with SNS but it doesn’t, it only shows me 3 options: Slack, custom webhook, and Amazon Chime (as mentioned in their [doc](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/alerting.html): *“Compared to OpenSearch, alerting in Amazon OpenSearch Service has some notable differences… OpenSearch Service supports Amazon Simple Notification Service ([Amazon SNS](https://aws.amazon.com/sns/?whats-new-cards.sort-by=item.additionalFields.postDateTime&whats-new-cards.sort-order=desc)) for notifications.”*) Does anyone know what should I do to integrate SNS with the AWS OpenSearch Service? Because also, supposing that it is not possible for any reason, if I wanted to do the same thing, using a Lambda function, it’s nonsense because as OpenSearch is a service hosted in AWS, why should I go to the Internet, to consume an API (also hosted in AWS) with all that implies (time, security, etc.), just to send an alert mail message? Thanks for your time.

I'm trying to work with cloudwatch log data in opensearch. While building some dashboards, I'm querying by "@log_stream". My log streams vary slightly, like "productName_filterlog" and "productName_messagelog". However, my query for one "@log_stream" yields results with many different log streams mixed in. I'm doing a term search with explain enabled in hopes of getting some reasoning for this: ### Request ``` POST myindex/_search?explain=true { "query":{ "term": { "@log_stream": "productName_filterlog" } } } ``` ### Response ``` ... "hits": [ { ..., "_explanation" : { "value" : 1.0, "description" : "*:*", "details" : [ ] } } ] ``` **My question** : what does this _explanation object mean? Why is it so useless? Is there a better way to determine why some unwanted results are showing up? I'm assuming the reason is due to some analyzer shenanigans, but how would one determine this? thanks!

Hello, We have an AppSync API and OpenSearch datasource and resolvers. When we run OpenSearch queries from AppSync resolvers, we get an error in response "Communication error while executing a request to OpenSearch." without any further details. There are no OpenSearch error logs for erred request. We have a DynamoDB datasource that is working without issues. The OpenSearch setup also is working fine with requests from outside AppSync. We are able to access OpenSearch through an EC2 instance launched in the same VPC as OpenSearch. What could be the issue? Regards,

If disaster recovery is not a requirement, what would be the best strategy for setting up API Gateway to server global customers. Here are three options that I can think of, not able to land on one. **Option 1**: Single Edge Optimized API Gateway serving traffic * Pros: save cost and avoid complexity of data replication (backend is opensearch) * Cons: Latency? not sure how much edge optimized API will help with latency, as customer will be hitting the API at nearest edge (ssl handshake, etc) and traffic flowing via backbone network. ( Question 1) **Option 2** Multiple Regional API Gateway with Route53 Latency based routing * Pros: customers hitting closest API. * Cons: Data replication, Cost. Also, since there is no cloud front here, traffic will be flowing via internet to closest region API, lets say we have API deployed in two regions , US and Singapore, would users in Europe see latency , worse than the Option 1, where requests are going to nearest edge location and reaches API via backbone? **Option 3** Multiple Edge Optimized API Gateway with Route53 Latency based routing * Pros: customers hitting closest API. Not sure how latency based routing works on an edge optimized endpoint, would it even help, since both endpoints are edge optimized. Not sure how smart is Route53 (Question 2) * Cons: Data replication, cost and uncertainty of Latency based routing. and Finally , one that I can think of could work, but haven't found too many solutions where people implemented. **Option 4** Multiple Regional API Gateway with single custom Cloudfront on top with cloudfront functions to do the routing. * Pros: customers hitting closest edge optimized location and routed to nearest API, this routing will be based on country of origin header from cloudfront. * Cons: Same Data Replication, Cost and predefined list of countries based routing. I need to spend time and run tests with multiple solutions. But wanted to seek community advise first. To summarize everything, if cost, complexity and disaster recovery are kept out of discussion, what would be best architecture for API Gateway to avoid latency issues.

Service Name: Amazon OpenSearch Service Version Name: Elasticsearch 6.3 Service software version: updating to R20211203-P2 (latest) Start time: 16/12/2021, 17:11:14 (UTC), checked from the Notifications tab Cluster Status: 16-17/12 are green, then red from 18-29/12, I have then restored it from the healthy automated snapshot, becomes green again until now Instance Status: processing Data node: 2 (desired to be 1) What could I do when I have such situation? The status is stuck in processing. Any method to handle it as I would like to perform version update from 6.3 to 6.8 and to opensearch? Remarks: I have another production instance that finished the service software update within 1 hour.

Hi I'm trying to start an elasticsearch container via - docker compose (aws-cli and switching to ecs context), but it fails to start - AccessDeinedExcception - cant write to /usr/share/elasticsearch/data/nodes/ directory. I have researched the issue on google and its because of the permission on that folder - from my understanding I need to fix the permissions in the host directory mapped to /usr/share/elasticsearch/data/nodes/ (I think) running sudo chown -R 1000:1000 [directory} However my container shuts down and how am I supposed to update the permission on that directory? this is my docker-compose - any help appreciated version: '3.8' services: elasticsearch01: user: $USER image: docker.elastic.co/elasticsearch/elasticsearch:7.14.1 #image: 645694603269.dkr.ecr.eu-west-2.amazonaws.com/smpn_ecr:latest container_name: es02 restart: unless-stopped environment: cluster.name: docker-es-cluster discovery.type: single-node bootstrap.memory_lock: "true" # ES_JAVA_OPTS: "-Xms2g -Xmx2g" xpack.security.enabled: "false" xpack.monitoring.enabled: "false" xpack.watcher.enabled: "false" node.name: es01 network.host: 0.0.0.0 logger.level: DEBUG ulimits: memlock: soft: -1 hard: -1 volumes: - es_data01:/usr/share/elasticsearch/data:rw ports: - "9200:9200" - "9300:9300" healthcheck: test: "curl -f http://localhost:9200 || exit 1" networks: - smpn_network volumes: es_data01: driver: local networks: smpn_network: driver: bridge

We have Amazon ElasticSearch Cluster of version 7.10.0. We would like to export some data from ElasticSearch to RDS Table. We are already using Logstash to import some data into ElasticSearch. There is amazon_es plugin which can only be used as output plugin in logstash pipeline but for export, we are not able to find anything. In our use case, there is some summarize data in Amazon ElasticSearch which we want to insert into RDS table. Let us know how can we do it?

I have the following setup for logs: * Fargate Service -> Firelens/Fluent Bit -> OpenSearch (ElasticSearch) Cluster **or** * Fargate Service -> Cloudwatch -> OpenSearch (ElasticSearch) Cluster I know about [Fluent Bit Throttle](https://docs.fluentbit.io/manual/pipeline/filters/throttle), but that is on client side. What would be good mechanism on OpenSearch (ElasticSearch) Cluster side to throttle that Client/Index if he pushes to much data?

We used this guide: https://opensearch.org/docs/latest/opensearch/rest-api/scroll/, and the amazon elasticsearch docs: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/supported-operations.html to try and find out how to perform scroll search on amazon elasticsearch. We recently switched from elasticsearch on one of our local servers to amazon elasticsearch. We came to realize that scroll performs differently. See below a snippet on how we now try to perform scroll (in python3): params = {"size": 5000, "q": query, "scroll": "5m"} resp = requests.get(ELASTICSEARCH_URL + "/_search", params=params, auth=auth) resp.status code: 200 scroll_id = resp.json().get("_scroll_id") In [23]: resp = requests.get(ELASTICSEARCH_URL + "_search/scroll", params={"scro ...: ll": "5m", "scroll_id": scroll_id}, auth=auth) In [24]: resp Out[24]: <Response [400]> In [25]: resp._content Out[25]: b'{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"request [/users/_search/scroll] contains unrecognized parameters: [scroll], [scroll_id]"}],"type":"illegal_argument_exception","reason":"request [/users/_search/scroll] contains unrecognized parameters: [scroll], [scroll_id]"},"status":400}' We've tried about all the variations, remove "/"'s or tweaking params, but getting this same error. Thank you in advance.

I have Fargate Service with FireLens and Fluent Bit. I want to send logs to OpenSearch (or ElasticSearch). Two good options: 1. Firelens / FluentBit -> [kinesis plugin](https://github.com/aws/amazon-kinesis-streams-for-fluent-bit) with **compression and aggregation** -> Kinesis Data Stream -> Kinesis Firehose with Lambda that decompresses and parses logs -> OpenSearch (or ElasticSearch). 2. Firelens / FluentBit -> [es plugin](https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch) -> Open Search (or ElasticSearch). First version saves bandwidth over the network and custom log parsing can be done. But if no custom log parsing is needed, how can I measure that the second solution is better? Is it better by the way?

What's the log4j version on R20211203- P2 (today, 20-Dec-2021)?

Yesterday we wanted to update our Elasticsearch domains to the latest software update (R20211203-P2) but after clicking on the update button and confirming it, the system acknowledged the decision and update only 1 of the 4 domains. Now I am in a situation where the dashboard shows me that the software is up to date but if I check the details of the single instances only one has the software update I was expecting. Also the instance update is the only one that showed a "update in progress" status after a while, the others didn't. The update option in the dropdown menu is disabled so I don't know if the server has been update successfully. Do you have any suggestion or, if you were in the same situation, did you manage to sort it? some images: [dashboard showing all domains up to date](https://drive.google.com/file/d/1dAH2rLm2mZHnGjEw6ggDYA-Dppj_dUbN/view?usp=sharing) [one domain shows the "right" software update](https://drive.google.com/file/d/1bAKo4DAvPQ4nnsdzBYO_wnNz2vnU-hU1/view?usp=sharing) [one domain shows the old/previous software update and menu option is disabled](https://drive.google.com/file/d/1IWPzcIX_qC1f11EFx3ASgL1H_uCZ67nP/view?usp=sharing) Who is telling the truth? Thanks FL

We have an ElasticSearch domain that has been stuck in "Being Deleted" status for several months ago. How can we get this instance fully deleted? Any help on this please.

There is a cluster that, due to losing a couple of nodes has a single shard in UNASSIGNED state. **TL;DR;**: The shard can not be rerouted due to AWS limitations, index can not be deleted due to running snapshot (for over 18 hours now), cluster has scaled to double its regular size for no obvious reason and snapshot can not be cancelled because it is one of the automated ones. What could be done to get the cluster back to green health? Data loss of that single index should not be a problem. ## Detailed explanation ### Symptom Cluster in red health status due to a single unnasigned shard. A call to `/_cluster/allocation/explain` returns the following: ``` { "index": "REDACTED", "shard": 1, "primary": true, "current_state": "unassigned", "unassigned_info": { "reason": "NODE_LEFT", "at": "2021-12-01T21:27:04.905Z", "details": "node_left[REDACTED]", "last_allocation_status": "no_valid_shard_copy" }, "can_allocate": "no_valid_shard_copy", "allocate_explanation": "cannot allocate because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster", ... ``` ### Cluster rerouting Regular troubleshooting on the matter indicates that one could take the data loss by reallocating the shard to empty using something like: ``` $ curl -XPOST '/_cluster/reroute' -d '{"commands": [{ "allocate_empty_primary": { "index": "REDACTED", "shard": 1, "node": "REDACTED", "accept_data_loss": true }}] }' {"Message":"Your request: '/_cluster/reroute' is not allowed."} ``` But that endpoint is not available in AWS. ### Closing/Deleting the index Other suggestions include closing the index for operations, but that is not supported by AWS: ``` $ curl -X POST '/REDACTED/_close' {"Message":"Your request: '/REDACTED/_close' is not allowed by Amazon Elasticsearch Service."} ``` Another solution is to delete the index. But, as there is a running snapshot, it can not be deleted: ``` $ curl -X DELETE '/REDACTED' {"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[REDACTED][indices:admin/delete]"}],"type":"illegal_argument_exception","reason":"Cannot delete indices that are being snapshotted: [[REDACTED]]. Try again after snapshot finishes or cancel the currently running snapshot."},"status":400} ``` ### Cancelling the snapshot As the previous error message states, you can try cancelling the snapshot: ``` curl -X DELETE '/_snapshot/cs-automated-enc/REDACTED' {"Message":"Your request: '/_snapshot/cs-automated-enc/REDACTED' is not allowed."} ``` Apparently that is because the snapshot is part of the automated ones. Had it been a manual snapshot I would have been able to cancel it. Problem is that the snapshot has been running for over 10 hours and is still initializing: ``` $ curl '/_snapshot/cs-automated-enc/REDACTED/_status' { "snapshots": [ { "snapshot": "2021-12-12t20-38-REDACTED", "repository": "cs-automated-enc", "uuid": "REDACTED", "state": "INIT", "shards_stats": { "initializing": 0, "started": 0, "finalizing": 0, "done": 0, "failed": 0, "total": 0 }, "stats": { "number_of_files": 0, "processed_files": 0, "total_size_in_bytes": 0, "processed_size_in_bytes": 0, "start_time_in_millis": 0, "time_in_millis": 0 }, "indices": {} } ]} ``` As it can be seen from the timestamp, it has been that way for almost 20 hours now (for reference, previous snapshots show to have run in a couple of minutes).

I have two OpenSearch Service deployed in AWS, one for PreProduction and the other one for production purpose. I have just received an AWS notification telling me both of my services have an update release (R20211203). The update release is available only onto 1 of my 2 services. The AWS portal is showing the following message "Your domain is not eligible to be updated. Please see [Learn more](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/service-software.html) for more information." The "Learn more" documentation shows reasons why my domain might be ineligible for a service software update but none of the reasons seems to apply to my domain as both are pretty in the same state while i can upgrade one. Any clues?

I've created a Amazon OpenSearch Service(**t3.small.search**).It was working good quite a few months. But suddenly the instance went to red status after that it didn't become yellow. I've increased the instance type **t3.medium.search** it was still in processing state and didn't recover it is already two day in processing state and also I'm not able to reach (giving 502 bad gateway error)

They simply do not exist. An attempt to reconfigure the cluster, increase capacity, and select other types of virtual machines did not lead to anything. The cluster has been "processing" for more than a day Why did this happen? Can this cluster be repaired or its data restored?

Howdy, It looks like the R20210426-P2 software upgrade is being pushed on us by AWS and is a mandatory upgrade. My question: - does R20210426-P2 software upgrade actually upgrade the ES version number? It would be bad if that happened because our application - Magento 2.3.3 can only accept 6.8/6.9 and not 7.x+ Additional question: Is the there a way to backup your domain / nodes / ES instance? Thanks! Paul

Hi everyone, I would like to connect an Elastic Search cluster to my graphQL API (AppSync) to be able to perform a geosearch, but I'm quite confused by the pricing model... On the AWS Elastic Search page, I find "Amazon Elasticsearch Service lets you pay only for what you use – there are no upfront costs or usage requirements." and on its pricing page, the pricing table's header says "on-demand pricing". For a t2.small in Frankfurt, it costs $0.048/h. But is this for a continuously running server (hence $30/month) or only for the time you need the server to be running (genuinely on-demand)? BR, Arne

Hi, I’m using the ElasticSearch service on a VPC with basic authentication. I am able to make ES REST API calls, but Kibana API calls fail with 401 status code - "Authentication required". Kibana UI works well. For example, the following API call works: curl -u user:pass -H 'Content-Type: application/json' -X GET <vpc-es-endpoint>/_search But this one fails with status 401: curl -u user:pass -H 'Content-Type: application/json' -X GET <vpc-es-endpoint>/_plugin/kibana/api/saved_objects/index-pattern/<id> Any idea what could be the problem? Thanks

The Elasticsearch client for golang uses info derived from the follow ES api call: GET /_node/http <https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-info.html> Normally this call returns lot of information about the node including IP address but when making the call to the AWS service this information is missing. This causes the golang client to fail to be able to _discover_ any nodes. Here are two examples which highlight the problem: regular node i have running locally: curl "http://localhost:9200/_nodes/http?pretty" { "_nodes" : { "total" : 1, "successful" : 1, "failed" : 0 }, "cluster_name" : "docker-cluster", "nodes" : { "hnsnurlLTG2-XbBQFyYg9w" : { "name" : "16c3f8d15864", "transport_address" : "172.25.0.4:9300", "host" : "172.25.0.4", "ip" : "172.25.0.4", "version" : "7.8.0", "build_flavor" : "default", "build_type" : "docker", "build_hash" : "757314695644ea9a1dc2fecd26d1a43856725e65", "roles" : \[ "data", "ingest", "master", "ml", "remote_cluster_client", "transform" ], "attributes" : { "ml.machine_memory" : "25225474048", "xpack.installed" : "true", "transform.node" : "true", "ml.max_open_jobs" : "20" }, "http" : { "bound_address" : \[ "0.0.0.0:9200" ], "publish_address" : "172.25.0.4:9200", "max_content_length_in_bytes" : 104857600 } } } } AWS equivalent: curl -XGET https://vpc-address....amazonaws.com/_nodes/http?pretty { "_nodes" : { "total" : 3, "successful" : 3, "failed" : 0 }, "cluster_name" : "XXX", "nodes" : { "8oxT_9maSMif5iXoQptpKA" : { "name" : "d026e0070020663bdd93028c2c801292", "version" : "7.7.0", "build_flavor" : "oss", "build_type" : "tar", "build_hash" : "unknown", "roles" : \[ "ingest", "master", "data", "remote_cluster_client" ] }, "8joBdfy6Q-aULryA3aP00w" : { "name" : "3992f18f3b80c28fbb6d5b5624c4bdfe", "version" : "7.7.0", "build_flavor" : "oss", "build_type" : "tar", "build_hash" : "unknown", "roles" : \[ "ingest", "master", "data", "remote_cluster_client" ] }, "2v572nbWQK6zrXmwGUiRmQ" : { "name" : "41d42ffdc2fdaf9c7a425e29c6ba92d6", "version" : "7.7.0", "build_flavor" : "oss", "build_type" : "tar", "build_hash" : "unknown", "roles" : \[ "ingest", "master", "data", "remote_cluster_client" ] } } } Obviously there are going to be some differences local Vs AWS with things like plugins but returning the actually http information seems pretty fundemental no?

What is the impact to AWS Elasticsearch service on the licensing [change ](https://www.elastic.co/blog/licensing-change) to Elasticsearch and Kibana?

I am trying to use the Nginx module for filebeat/metricbeat, which in turn seems to require ingest-geoip This is the error they got: Dec 03 08:37:45 ip-10-1-2-5 filebeat[30775]: 2020-12-03T08:37:45.077Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://<OUR_AWS_ELK_INSTANCE>)): Connection marked as failed because the onConnect callback failed: Error loading pipeline for fileset nginx/access: This module requires the following Elasticsearch plugins: ingest-geoip. You can install them by running the following commands on all the Elasticsearch nodes: Dec 03 08:37:45 ip-10-1-2-5 filebeat[30775]: sudo bin/elasticsearch-plugin install ingest-geoip Is there a way to install ingest-geoip or any other workaround to use Nginix module for filebeat/metricbeat on ES?

Hi, I can not see my indices in the kibana>management>create new index view. I've waited a day, thinking there was a data issue somewhere, but I still cannot see them. I created several new indices, 2 by command line curl and the other directly inside kibana, all of which returned successful messages and also ran `_cat/indices` on my elasticsearch domain and verified that they exist. What am I doing wrong here?

Hi all, I'm trying to figure it out if/how is possible to import Filebeat/Metricbeat dashboards on ES Kibana. It seems possible, at least with older version of ES: https://discuss.elastic.co/t/installing-metricbeat-dashboards-on-kibana-for-amazon-elasticsearch/102180 But in other case seems to fail: https://forums.aws.amazon.com/thread.jspa?threadID=279402 Do you know if that is possible and which are the limitations? Thanks!