Questions tagged with Amazon OpenSearch Service

We are using AWS SSO with Google Workspace to log into AWS. We also have OpenSerch in use. We want to use AWS SSO as an IDP for OpenSearch however we get the following error ``` Oops, something went wrong Provide your administrator with the following info: No access Request ID: xxxx HTTP status: 403 ``` The 403 call is when a GET is made on https://portal.sso.eu-west-1.amazonaws.com/saml/v2/assertion/xxxxxredactedxxxxx/ HTTP/1.1lg...

I have a set of what I presumed to be fairly small elasticsearch indices in a self-hosted cluster of ec2 instances. I'm in the middle of trying to migrate this date into an AWS-managed elasticesearch cluster, and I've been having trouble getting the reindex tasks to _consistently_ reproduce the documents in the target cluster. My cluster has about 330,000 documents in it - my approach has been to expose the source cluster to the destination cluster and to issue a reindex operation to the destination cluster, using some scripts written in ruby: ```ruby def reindex(index:) logger.info("reindexing #{index}") task = dest_es.reindex( body: { source: { remote: { host: source, username: source_username, password: source_password, socket_timeout: '5m', connect_timeout: '30s', external: true }, index: index }, dest: { index: index }, conflicts: "proceed" }, refresh: true, timeout: '3m', wait_for_completion: wait_for_completion? # set to false by default ) logger.info(task) end ``` This task, however, never reindexes 100% of the documents, and frequently times out with the task status stating: ```json { "completed" : true, "task" : { ... "type" : "transport", "action" : "indices:data/write/reindex", "status" : { "total" : 317554, "updated" : 0, "created" : 184000, "deleted" : 0, "batches" : 184, "version_conflicts" : 0, "noops" : 0, "retries" : { "bulk" : 0, "search" : 0 }, "throttled_millis" : 0, "requests_per_second" : -1.0, "throttled_until_millis" : 0 }, "description" : """ ... """, "start_time_in_millis" : 1660058838837, "running_time_in_nanos" : 312625819581, "cancellable" : true, "headers" : { } }, "error" : { "type" : "socket_timeout_exception", "reason" : null } } ``` If i try to break this reindex operation into multiple tasks, running asyncronously, i can issue a number of async tasks to reindex, say, 2000 documents at a tim. . By looking at `POST /.tasks/_search` it seems that they all complete without issue, but I still can't quite get to 100% - in fact the total number of docs varies between 60% and 99% Are there config settings that are generally used with reindexing across clusters? I feel like this should be a lot more straightforward than it's turning out to be. Thanks!lg...

I'm using OpenSearch services through AWS Amplify. Today suddenly stopped working all my apps, after a little researching I found that the OpenSearch domain is not working at all. When I check the "Cluster Health" using AWS Console, just behind "Summary" there is an error: { "message": "No server available to handle the request", } And in "Instance Health" for today there isn't any "Data node", but if I list the previous week there is a "Data node". So, it seems like the Data note was removed. Amplify uses ElasticeSearch 6.2, and my instance type is "t2.medium.search". What should I do to recover all my data and to fix this issue? Thanks!lg...

I am not able to connect metricbeat in my ec2 instance and aws managed opensearch for infra monitoring. How to achieve this?lg...

Elastic Search managed ignores X-Opaque-id header [Elastic Search docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/api-conventions.html#x-opaque-id) mentions with the use of an `X-Opaque-id` in the requests we can track the queries. The X-Opaque-id header should be forwarded in the response. However this doesn't work with AWS managed Elastic Search to aws elasticsearch: ``` curl -i --http1.1 -H 'X-Opaque-Id: 12345' "http://aws_elastic_sarch_url/_cat/health" HTTP/1.1 200 OK Date: Wed, 03 Aug 2022 07:56:35 GMT Content-Type: text/plain;charset=UTF-8 Content-Length: 100 Connection: keep-alive Access-Control-Allow-Origin: * 1659513395 07:56:35 aws-elasticsearch green 7 4 true 7976 3978 0 0 0 0 - 100.0% ``` to raw elasticsearch: ``` curl -i --http1.1 -H 'X-Opaque-Id: 12345' "http://raw_elastic_sarch_url/_cat/health" HTTP/1.1 200 OK X-Opaque-Id: 12345 content-type: text/plain; charset=UTF-8 content-length: 66 1659516213 08:43:33 docker-cluster green 1 1 0 0 0 0 0 0 - 100.0% ``` Any solutions for using X-opaque-id headers with AWS ElasticSearch?lg...

We are evaluating search technology options and doing research I've bumped into some suggestions that the CloudSearch product is unofficially deprecated in favor of OpenSearch (still supported but not receiving active development/improvements). It appears the product announcements for CloudSearch over the last couple of years (compared to OpenSearch) have been nearly non-existent. I'm wondering if anyone can comment on/confirm this?lg...

We're having an issue where our UAT environment AWS OpenSearch setup seems to drop nodes occasionally, but doesn't seem to drop to zero. As it's our testing environment, we only have one node for OpenSearch, but for some reason it'll disappear and then come back with only the `.kibana` index still available, we can then restore from our last snapshot. I tried to setup an automated task to watch for when Nodes dropped to zero, creating an event bridge alert, but the problem is, it doesn't drop to zero as you can see from the metric snapshot below, so the event bridge alert never triggers.  Why doesn't it drop to zero? How can I automate the restoration when the node eventually comes back? how can I trigger an alert at the very least when the nodes disappear?lg...

Amazon recommends running an opensearch domain that contains 3 regular nodes and 3 master nodes distributed accross 3 AZ zones. The lowest instance type that is still suitable for a production environment is the `t3 medium.search` instance type. I've run this setup for about 1.5 days, using `t3 small.search` instead of `medium`. When i looked at the bill afterwards i could see that running such an instance for merely 1.5 days already costs 9 dollars. That's way too expensive for me. According to the [amazon cost calculator](https://calculator.aws/#/addService/OpenSearchService) the monthly cost for this setup would have been well over 350 dollars. My web application will use the open search server only for serving autocomplete suggestions and finding documents whose coordinates reside within a certain geographical area. When the webapplicatoin is launched the open search server will start out with only 5 indexes containing only a small number of documents, no more then 200 mb in total. Of these only one index is used to preform geospatial queries on. I don't think i need a t3 medium instance for this. So my question is: With what kind of open search instance can i start out with? The setup needs to be economical because it will take a while before my web application starts making money. I was thinking about setting up a `t2 micro.search` domain service with 2 micro master instances and 2 micro worker instances. That would cost me about 50 dollars a month in total. Could this be a good setup to start with? If so then i would like to know how i can setup a domain that uses `t2 micro.search` instances. When i go to the domain creation page in my aws console i'm not able to select `t2 micro.search` from the instance type list. The smallest i can select is `t3 small.search` but thats already too expensive for me because i want to run nodes in atleast two availability zones. I could opt for running only one `t3 small.search` master node and one worker node, which would cost 50 dollars a month as well, but then the domain service is no longer highly available. If the availability zone it sits in crashes then i can't serve autocomplete suggestions anymore, nor can i return documents based on their coordinates. I'd love to hear your opinions on this. Thank youlg...

When using Kinesis firehose to opensearch / elasticsearch, while the delivery stream is super convenient, one major limitation I find is that one cannot override the timestamp field that is use to decide on the destination index (it always uses the estimated arrival time). This means that for backfill jobs (which are super common for us) all the data ends up in the current daily index, which reduces read/write scalability and also makes index-based management much more difficult (e.g. compact / archive index older than N days). Another example is when querying dashboards, some indices have data for a large range of dates and end up becoming a bottleneck for performance. Ideally one could pick a timestamp field from the events, or set it as part of the lambda processing, so that a record with date D goes to index of date D. Thanks for suggestions!lg...

I reconfigured my OpenSearch cluster from T2 to C5 instance type. The migration remains in "Step 3 of 4: Copying shards to new nodes" status for 2 days. Any idea to escape from this situation?lg...

I read this documentation: https://docs.amplify.aws/cli-legacy/graphql-transformer/searchable/#definition But how to impliment the query in android studio using java.What are the methods of Amplify to get search query? Can any one share documentation?lg...

we are trying to add an OpenSearch datasource to our existing AppSync setup. all other datasources for AppSync have been DynamDB up to this point is this error type indicating a problem with the resolver mapping templates or a problem re AppSync accessing OpenSearch ie misconfigured access policies? we are using the nodejs CDK to deploy via CodePipeline we have added an OpenSearch Domain, and it is being populated by a lambda listening for writes to a DynamoDB table - this part is working we have created an OpenSearch Datasource, and created a resolver using the following as request and response mapping templates ``` const requestTemplate = `{ "version":"2017-02-28", "operation":"GET", "path":"/id/shipments/_search", "params":{ "body":{ "from":0, "to": 20 } } }`; const responseTemplate = `{ "shipments": [ #foreach($entry in $context.result.hits.hits) #if( $velocityCount > 1 ) , #end $utils.toJson($entry.get("_source")) #end ], "total": -1 }`; const datasourceId = `ShipmentsListDatasource`; const datasource = new appsync.OpenSearchDataSource( this, getId(datasourceId, this.props), { domain: this.openSearchDomain, api: this.api, } ); datasource.createResolver({ typeName: GraphQLTypeName.Query, fieldName: "listShipments", requestMappingTemplate: appsync.MappingTemplate.fromString( requestTemplate ), responseMappingTemplate: appsync.MappingTemplate.fromString( responseTemplate ), }); ``` access to the OpenSearch domain has been configured like this ``` private configureOpenSearch() { const openSearchAppsyncRole = new iam.Role( this, "a-role-name-here", { assumedBy: new iam.ServicePrincipal("appsync.amazonaws.com"), } ); const lambdaPolicyStatement = new iam.PolicyStatement({ actions: ["es:*"], resources: [`${this.openSearchDomain.domainArn}/*`], principals: [new iam.ArnPrincipal(this.baseStack.lambdaRole.roleArn)], }); const rolePolicyStatement = new iam.PolicyStatement({ actions: ["es:*"], resources: [`${this.openSearchDomain.domainArn}/*`], effect: iam.Effect.ALLOW, notPrincipals: [openSearchAppsyncRole] }) this.openSearchDomain.addAccessPolicies(lambdaPolicyStatement, rolePolicyStatement); this.openSearchDomain.grantRead(openSearchAppsyncRole); this.openSearchDomain.grantIndexReadWrite( "id", this.shipmentsOpenSearchLambda ); } ``` any requests to the graphql endpoint for this particular query return the following ``` { "data": null, "errors": [{ "path": ["listShipments"], "data": null, "errorType": "OpenSearch:UserIllegalArgumentException", "errorInfo": null, "locations": [{ "line": 3, "column": 5, "sourceName": null }], "message": "OpenSearch responded with an error: Bad Request" }] } ```lg...