By using AWS re:Post, you agree to the Terms of Use
/Amazon OpenSearch Service/

Questions tagged with Amazon OpenSearch Service

Sort by most recent
  • 1
  • 90 / page

Browse through the questions and answers listed below or filter and sort to narrow down your results.

IoT Core OpenSearch Action Rule & "Mapper parsing exception"

Hi I was following this [AWS tutorial](https://aws.amazon.com/blogs/mobile/analyze-device-generated-data-with-aws-iot-and-amazon-elasticsearch-service/) but with my own sensor data. I've created the following IoT Core Rule OpenSearch Action: ``` OpenSearchTopicRule: Type: AWS::IoT::TopicRule Properties: TopicRulePayload: Actions: - OpenSearch: Endpoint: !Join ['', ['https://', !GetAtt OpenSearchServiceDomain.DomainEndpoint]] Id: '${newuuid()}' Index: sensors RoleArn: !GetAtt IoTOSActionRole.Arn Type: sensor_data Sql: SELECT *, timestamp() as ts FROM 'Greenhouse/+/Sensor/Status' ``` The IoTOSActionRole has propper es:ESHttpPut permission. But when I try to create an index with following command send from Postman that would match the `Type: sensor_data` attribute: ``` curl --location --request PUT 'https://search-iot***-avt***i.eu-west-1.es.amazonaws.com/sensors' \ --header 'Content-Type: application/json' \ --data-raw '{ "mappings": { "sensor_data": { "properties": { "ts": { "type": "long", "copy_to": "datetime"}, "datetime": {"type": "date", "store": true}, "deviceID": {"type": "text", "store": true}, "humidity": {"type": "integer", "store": true}, "temperature": {"type": "integer", "store": true}, "lux": {"type": "integer", "store": true}, "soil": {"type": "integer", "store": true} }}}' ``` I receive an error: ``` { "error": { "root_cause": [ { "type": "mapper_parsing_exception", "reason": "Root mapping definition has unsupported parameters: [sensor_data : {properties={datetime={store=true, type=date}, temperature={store=true, type=integer}, humidity={store=true, type=integer}, soil={store=true, type=integer}, deviceID={store=true, type=text}, lux={store=true, type=integer}, ts={copy_to=datetime, type=long}}}]" } ], "type": "mapper_parsing_exception", "reason": "Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters: [...]", "caused_by": { "type": "mapper_parsing_exception", "reason": "Root mapping definition has unsupported parameters: [...}]" } }, "status": 400 } ``` I've tried removing the 'type' `"sensor_data"` attribute and that allowed me to create an index with that mapping, ``` { "acknowledged": true, "shards_acknowledged": true, "index": "sensors" } ``` and then index pattern in OpenSearch Dashboard, but what happens then is that the IoT Core Rule even though it gets triggered does not result in any data ingestion to the OpenSearch domain. So I guess IoT Core Action tries to send that data with type `sensor_data`but there's no corresponding type in OS. Additionally, when I open the Discovery tab in OS dashboard I get this notice: ``` "undefined" is not a configured index pattern ID Showing the default index pattern: "sensors*" (d97775d0-cc44-11ec-acd4-bf72aa2fd725) ``` I'm using the OpenSearch 1.2 (latest) version. Sample data: ``` { "deviceID": "Tomatoes", "Greenhouse": 1, "date": "05-05", "time": "09:35:39", "timestamp": 1651743339, "humidity": 60, "temperature": 33.3, "lux": 9133.333, "soil": 78 } ``` What PUT call I have to make to create a `sensor_data` type mapping in OS that would match the type specified in IoT Core OpenSearch Action Rule?
1
answers
0
votes
3
views
asked 14 days ago

Not able to do one time load, from postgres to opensearch using DMS

Trying to migrate existing data from AWS RDS Postgres to AWS managed OpenSearch, but it is not working, no rows were migrated to opensearch, When checking the Cloudwatch log getting below error Bulk request failed. no retry. TotalRecordCount 4080, FailedRecordCount 4080 [1026400] (elasticsearch_bulk_utils.c:181) DMS has the following configuration: { "TargetMetadata": { "TargetSchema": "", "SupportLobs": false, "FullLobMode": false, "LobChunkSize": 0, "LimitedSizeLobMode": false, "LobMaxSize": 0, "InlineLobMaxSize": 0, "LoadMaxFileSize": 0, "ParallelLoadThreads": 5, "ParallelLoadBufferSize": 100, "BatchApplyEnabled": false, "TaskRecoveryTableEnabled": false, "ParallelLoadQueuesPerThread": 0, "ParallelApplyThreads": 0, "ParallelApplyBufferSize": 100, "ParallelApplyQueuesPerThread": 0 }, "FullLoadSettings": { "TargetTablePrepMode": "DO_NOTHING", "CreatePkAfterFullLoad": false, "StopTaskCachedChangesApplied": false, "StopTaskCachedChangesNotApplied": false, "MaxFullLoadSubTasks": 8, "TransactionConsistencyTimeout": 600, "CommitRate": 50000 }, "Logging": { "EnableLogging": true, "LogComponents": [ { "Id": "TRANSFORMATION", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "SOURCE_UNLOAD", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "IO", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "TARGET_LOAD", "Severity": "LOGGER_SEVERITY_DETAILED_DEBUG" }, { "Id": "PERFORMANCE", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "SOURCE_CAPTURE", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "SORTER", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "REST_SERVER", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "VALIDATOR_EXT", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "TARGET_APPLY", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "TASK_MANAGER", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "TABLES_MANAGER", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "METADATA_MANAGER", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "FILE_FACTORY", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "COMMON", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "ADDONS", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "DATA_STRUCTURE", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "COMMUNICATION", "Severity": "LOGGER_SEVERITY_DEFAULT" }, { "Id": "FILE_TRANSFER", "Severity": "LOGGER_SEVERITY_DEFAULT" } ], "CloudWatchLogGroup": null, "CloudWatchLogStream": null }, "ControlTablesSettings": { "historyTimeslotInMinutes": 5, "ControlSchema": "", "HistoryTimeslotInMinutes": 5, "HistoryTableEnabled": true, "SuspendedTablesTableEnabled": false, "StatusTableEnabled": true, "FullLoadExceptionTableEnabled": false }, "StreamBufferSettings": { "StreamBufferCount": 3, "StreamBufferSizeInMB": 8, "CtrlStreamBufferSizeInMB": 5 }, "ChangeProcessingDdlHandlingPolicy": { "HandleSourceTableDropped": true, "HandleSourceTableTruncated": true, "HandleSourceTableAltered": true }, "ErrorBehavior": { "DataErrorPolicy": "LOG_ERROR", "EventErrorPolicy": null, "DataTruncationErrorPolicy": "LOG_ERROR", "DataErrorEscalationPolicy": "SUSPEND_TABLE", "DataErrorEscalationCount": 0, "TableErrorPolicy": "SUSPEND_TABLE", "TableErrorEscalationPolicy": "STOP_TASK", "TableErrorEscalationCount": 0, "RecoverableErrorCount": -1, "RecoverableErrorInterval": 5, "RecoverableErrorThrottling": true, "RecoverableErrorThrottlingMax": 1800, "RecoverableErrorStopRetryAfterThrottlingMax": true, "ApplyErrorDeletePolicy": "IGNORE_RECORD", "ApplyErrorInsertPolicy": "LOG_ERROR", "ApplyErrorUpdatePolicy": "LOG_ERROR", "ApplyErrorEscalationPolicy": "LOG_ERROR", "ApplyErrorEscalationCount": 0, "ApplyErrorFailOnTruncationDdl": false, "FullLoadIgnoreConflicts": true, "FailOnTransactionConsistencyBreached": false, "FailOnNoTablesCaptured": true }, "ChangeProcessingTuning": { "BatchApplyPreserveTransaction": true, "BatchApplyTimeoutMin": 1, "BatchApplyTimeoutMax": 30, "BatchApplyMemoryLimit": 500, "BatchSplitSize": 0, "MinTransactionSize": 1000, "CommitTimeout": 1, "MemoryLimitTotal": 1024, "MemoryKeepTime": 60, "StatementCacheSize": 50 }, "PostProcessingRules": null, "CharacterSetSettings": null, "LoopbackPreventionSettings": null, "BeforeImageSettings": null, "FailTaskWhenCleanTaskResourceFailed": false, "TTSettings": null } Opensearch have index with following settings { "settings": { "index.max_ngram_diff" :8, "analysis": { "analyzer": { "my_ngram_analyzer": { "type": "custom", "tokenizer": "standard", "filter": [ "lowercase", "mynGram" ] } }, "filter": { "mynGram": { "type": "nGram", "min_gram": 6, "max_gram": 14, "token_chars": [ "letter", "digit", "whitespace", "symbol" ] } } }, "number_of_shards": 6, "number_of_replicas": 1 }, "mappings" : { "properties" : { "created_at" : { "type" : "date" }, "id" : { "type" : "long" }, "name" : { "type" : "text", "analyzer":"my_ngram_analyzer" , "search_analyzer": "my_ngram_analyzer" }, "phone" : { "type" : "text", "analyzer":"my_ngram_analyzer" , "search_analyzer": "my_ngram_analyzer" }, "updated_at" : { "type" : "date" } } } } I have tried to insert a sample document using _bulk API on opensearch console and it worked, below is the thing I had tried over opensearch, which worked POST _bulk {"index":{"_index":"contacts"}} {"name": "name","phone" : "11111111","created_at" : "2021-12-21T12:12:59","updated_at" : "2021-12-21T12:12:59","id": 101}
1
answers
0
votes
8
views
asked 21 days ago

IAM Policy To Create Domain in OpenSearch

I am trying to create Domain in open search, I used the Below IAM permission but everytime it is giving me this error-: Before you can proceed, you must enable a service-linked role to give Amazon OpenSearch Service permissions to create and manage resources on your behalf I have also attached the Service Linked Role but still I am facing the Issue I am using this IAM policy { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "es:ESHttpDelete", "es:ESHttpGet", "es:ESHttpHead", "es:ESHttpPost", "es:ESHttpPut", "es:ESHttpPatch", "ec2:AuthorizeSecurityGroupIngress", "ec2:CreateNetworkInterface", "ec2:CreateSecurityGroup", "ec2:DeleteNetworkInterface", "ec2:DeleteSecurityGroup", "ec2:DescribeAvailabilityZones", "ec2:DescribeNetworkInterfaces", "ec2:DescribeSecurityGroups", "ec2:DescribeSubnets", "ec2:DescribeVpcs", "ec2:ModifyNetworkInterfaceAttribute", "ec2:RevokeSecurityGroupIngress", "elasticloadbalancing:AddListenerCertificates", "elasticloadbalancing:RemoveListenerCertificates" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "es:AddTags", "es:AssociatePackage", "es:CreateDomain", "es:CreateOutboundConnection", "es:DeleteDomain", "es:DescribeDomain", "es:DescribeDomainAutoTunes", "es:DescribeDomainConfig", "es:DescribeDomains", "es:DissociatePackage", "es:ESCrossClusterGet", "es:GetCompatibleVersions", "es:GetUpgradeHistory", "es:GetUpgradeStatus", "es:ListPackagesForDomain", "es:ListTags", "es:RemoveTags", "es:StartServiceSoftwareUpdate", "es:UpdateDomainConfig", "es:UpdateNotificationStatus", "es:UpgradeDomain" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "es:AcceptInboundConnection", "es:CancelServiceSoftwareUpdate", "es:CreatePackage", "es:CreateServiceRole", "es:DeletePackage", "es:DescribeInboundConnections", "es:DescribeInstanceTypeLimits", "es:DescribeOutboundConnections", "es:DescribePackages", "es:DescribeReservedInstanceOfferings", "es:DescribeReservedInstances", "es:GetPackageVersionHistory", "es:ListDomainNames", "es:ListDomainsForPackage", "es:ListInstanceTypeDetails", "es:ListInstanceTypes", "es:ListNotifications", "es:ListVersions", "es:PurchaseReservedInstanceOffering", "es:RejectInboundConnection", "es:UpdatePackage" ], "Resource": "*" }, { "Sid": "AllowCreationOfServiceLinkedRoleForOpenSearch", "Effect": "Allow", "Action": [ "iam:CreateServiceLinkedRole", "iam:PassRole" ], "Resource": [ "arn:aws:iam::*:role/aws-service-role/opensearchservice.amazonaws.com/AWSServiceRoleForAmazonOpenSearchService*", "arn:aws:iam::*:role/aws-service-role/es.amazonaws.com/AWSServiceRoleForAmazonOpenSearchService*" ], "Condition": { "StringLike":{ "iam:AWSServiceName": [ "opensearchservice.amazonaws.com", "es.amazonaws.com" ] } } } ] }
0
answers
0
votes
3
views
asked 24 days ago

Opensearch upgrade stuck

We have a single node Opensearch 1.1 cluster. After selecting to upgrade it to 1.2 the cluster has been stuck on "Preparing to process updates". I found out that it is stuck because the default settings for .opendistro-* indices are to create 5 primaries and 1 replica shard. As our cluster only consists of a single shards those some of the .opendistro-* indices are not unallocated. I created index templates for those indices in question with 1 primary and 0 replicas and deleted some of the .opendistro indices but there are .opendistro-alerting-alerts indices which I can not delete even though I have all the permissions possible. Also followed this guide but still getting 403 https://aws.amazon.com/premiumsupport/knowledge-center/opensearch-disable-index-alerts/ GET _cat/shards .opendistro-alerting-alerts 3 r UNASSIGNED .opendistro-alerting-alerts 1 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822 .opendistro-alerting-alerts 1 r UNASSIGNED .opendistro-alerting-alerts 2 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822 .opendistro-alerting-alerts 2 r UNASSIGNED .opendistro-alerting-alerts 4 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822 .opendistro-alerting-alerts 4 r UNASSIGNED .opendistro-alerting-alerts 0 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822 .opendistro-alerting-alerts 0 r UNASSIGNED .opendistro-alerting-alert-history-2022.03.16-1 3 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822 .opendistro-alerting-alert-history-2022.03.16-1 3 r UNASSIGNED .opendistro-alerting-alert-history-2022.03.16-1 1 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822 .opendistro-alerting-alert-history-2022.03.16-1 1 r UNASSIGNED .opendistro-alerting-alert-history-2022.03.16-1 2 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822 .opendistro-alerting-alert-history-2022.03.16-1 2 r UNASSIGNED .opendistro-alerting-alert-history-2022.03.16-1 4 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822 .opendistro-alerting-alert-history-2022.03.16-1 4 r UNASSIGNED .opendistro-alerting-alert-history-2022.03.16-1 0 p STARTED x.x.x.x 92b97e40e5fe8565ff6a7bdf023bb822 .opendistro-alerting-alert-history-2022.03.16-1 0 r UNASSIGNED DELETE .opendistro-alerting-alert-history-2022.03.16-1 DELETE .opendistro-alerting-alerts { "error" : { "root_cause" : [ { "type" : "security_exception", "reason" : "no permissions for [] and User [name=xxx, backend_roles=[], requestedTenant=]" } ], "type" : "security_exception", "reason" : "no permissions for [] and User [name=xxx, backend_roles=[], requestedTenant=]" }, "status" : 403 } I would gladly add some additional data nodes to the cluster in order to fix this but this also fails as the cluster is stuck on "Preparing to process updates". EDIT: I tried deleting the .opendistro indices with "m aster" user but even "m aster" user receives error 403.
2
answers
0
votes
10
views
asked a month ago

How do I compose a BULK request to OpenSearch via AppSync resolver mapping templates?

I have a pipeline resolver for an AppSync Mutation. It contains two functions, the first one is a Lambda sending updates to RDS, the second one should take the result from `$ctx.prev.result` and index it via OpenSearch datasource. In the request resolver mapping template of the second one, I am composing the bulk body in NDJSON similar to the following mannar: ```vtl #set($bulk = $util.toJson({ "index": { "_id": "${$ctx.prev.result.id}" } })) #set($bulk = "${bulk} ${util.toJson($ctx.prev.result)}") { "version": "2017-02-28", "operation": "POST", "path": "/_bulk", "params": { "body": $util.toJson($bulk) } } ``` Lacking proper debugging tools, I have been using `$util.error` as a logging method to get my `$bulk` contents. And it looks like the following format, which seems correct. ```ndjson {"index":{"_id":"A8DEF210-C342-48CB-9A4A-DA7D1E4D6AF1"}} {"foo":123,"bar":999,"baz":1234567} ``` But when I actually runs the mutation via AppSync, I got a `MappingTemplate` error `Unable to transform for the body: $[params][body].` and I have no idea why. EDIT: I took a look at [[re:Post] Appsync HTTP resolver supported content types](https://repost.aws/questions/QUFqBom4iFQ-6ePM1z11Sthw/appsync-http-resolver-supported-content-types), which inspired me to take another look at [Resolver Mappping Template for OpenSearch (params)](https://docs.aws.amazon.com/appsync/latest/devguide/resolver-mapping-template-reference-elasticsearch.html#params-field). It seems POST body only accepts a single JSON object, NSJSON required by the bulk request is not supported yet. Am I correct? If so, is supporting the bulk API in the upcoming plans? Also, what is the currently recommended way to index multiple "normalized" documents from the same resolver?
0
answers
0
votes
2
views
asked a month ago

How to set document id when delivering data from Kinesis Data Firehose to Opensearch index

What I'm trying to do: 1. I am using a kinesis data stream to ingest data from a python client sending in JSON's. 2. I setup a kinesis firehose delivery stream with source as the kinesis-data-stream from the previous step and destination as an index on opensearch. I also use a lambda to transform the stream before delivering to opensearch. Now i would like to set the document id for the record I'm transforming in the lambda . I tried setting the key on the transformed record object to be `id` but that creates a document attribute named `id` and sets a value to it. What i would like is to set the _id on the search doc.When i try to set the _id attribute directly by returning it within the transformed record back to firehose delivery stream it generated a destination error : ``` "message": "{\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [_id] of type [_id] in document with id \\u002749627661727302149016659535428367209810930350656512327682.0\\u0027. Preview of field\\u0027s value: \\u00279b428743-b19a-4fb1-90f2-f132a423b2e8\\u0027\",\"caused_by\":{\"type\":\"mapper_parsing_exception\",\"reason\":\"Field [_id] is a metadata field and cannot be added inside a document. Use the index API request parameters.\"}}", "errorCode": "400", ``` Is there anyway to the doc _id for the documents loaded from firehose to opensearch or would i need to take a different approach by selecting the destination to be a http endpoint and using the rest API's provided by Opensearch (which would kinda be a hassle compared to directly being able to just set the _id attrib) ? What I'm really trying to do is update the indexed documents on a change event. I understand that firehose uses the bulk api from Opensearch , but am unsure about how the upsert operation is handled internally by the kinesis destination connector to opensearch. Hence specifying a fixed id from another DB would be ideal for both insert and update ops to Opensearch in my case.It would be super useful to atleast be able to dictate the type of operation based on some attribute of the kinesis record to opensearch with a reference id for the doc to update.
1
answers
0
votes
8
views
asked 2 months ago

Elasticsearch + Kibana + Cognito session error 403

Hi folks, we have created an Elasticsearch (version 7.10) cluster with Cognito enabled fined grained authentication for Kibana. We have replicated (via terraform IaC) a configuration already present on another AWS account that is working fine. In this new cluster we have Kibana session issues, for example when we click "Log out" in Kibana we got an 403 error (I'll post it at the end of my message as LOGOUT_ERROR). Another error happens when the Cognito session token expires and we got another 403 error (another small snippet at the end of my message as LOGIN_ERROR). We cannot see the Cognito login page if we don't remove the cookie in the browser (we tested different browsers). Everything else is working fine: we are able to use Elasticsearch and Kibana with no other issues. Since the error is originating from "AWSSecurityTokenService" (an interface in the AWS Java SDK) we smell this as a possible bug that we're not able to address on our side. The fact that the Elasticsearch cluster is in eu-south-1 region and Cognito is in eu-west-1 could be an issue? Can you please help us by pointing us to any resource that may help resolve this issue? Regards Marco LOGOUT_ERROR ``` <!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="description" content="403 Forbidden"> <title>Kibana Authentication Error</title> <link rel="stylesheet" href="//maxcdn.bootstrapcdn.com/bootstrap/3.3.4/css/bootstrap.min.css "> <link rel="stylesheet" href="//maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css "> <script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js "></script> <script src="//maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js "></script> <style> body { padding-top: 20px; } .jumbotron { font-size: 21px; font-weight: 200; line-height: 2.1428571435; color: inherit; padding: 10px 0px; } .body-content { padding-left: 15px; padding-right: 15px; } .jumbotron { text-align: center; background-color: transparent; } .jumbotron .btn { font-size: 21px; padding: 14px 24px; } .blue {color:#00bfff;} .red {color:#d9534f;} </style> </head> <div class="container"> <div class="jumbotron"> <h1><i class="fa fa-frown-o red"></i> Sorry!</h1> <p class="lead">Something went wrong during authentication between Kibana and Amazon Cognito.</p> <p><a href="https://kibana.linkemswarm.com/_plugin/kibana " class="btn btn-default btn-lg"><span class="blue">Log in to Kibana</span></a> </p> </div> </div> <div class="container"> <div class="body-content"> <div class="row"> <h2>What happened?</h2> <p>cognito:revoke_tokens:ErrorUser: x:x:x::xx:x is not authorized to perform: sts:AssumeRole on resource: x:x:x::xx:x (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied; Request ID: 432dbf6a-126f-43f4-be02-69c3657b3016; Proxy: null)</p> </div> <div class="row"> <h2>What should I do?</h2> <p>Try logging in again. If the problem persists, please review the <a href="https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-cognito-auth.html#es-cognito-auth-troubleshooting ">troubleshooting guide </a>for information on resolving common issues.</p> </div> </div> </div> </body> </html> ``` END LOGOUT_ERROR LOGIN_ERROR ``` ... <h2>What happened?</h2> <p>com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: User: x:x:x::xx:x is not authorized to perform: sts:AssumeRole on resource: x:x:x::xx:x (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied; Request ID: d3bbebd3-a5ba-4406-9d75-5b78cd1c92bc; Proxy: null)</p> ... ``` END LOGIN_ERROR
1
answers
0
votes
21
views
asked 2 months ago
0
answers
0
votes
2
views
asked 2 months ago
0
answers
0
votes
6
views
asked 3 months ago

Elasticsearch 6.4 - R20211203-P3 update deleted all documents?

We have been using the original ElasticSearch for a few years, works great. Today it stopped working. When I logged into the AWS console and went to the OpenSearch page our domain had a status of Processing and at first I thought it was stuck but after several minutes it finished. It then did the whole update process again which took another several minutes. Now it is back to Active but it still isn't working. I think the indices have been deleted? I have no documents any more. Is there a back up to restore from? under the Cluster configuration it says Snapshots - frequency - hourly, but I don't see anyway to restore it. Under the error logs I see: ``` [2022-01-29T09:05:58,417][WARN ][o.e.c.NodeConnectionsService] [bmDFGK_] failed to connect to node {5fSCYAe}{5fSCYAeFQbefAxIiS7YyVA}{03U8UySaTI6gDrlMk5RP2Q}{__IP__}{__IP__}{__AMAZON_INTERNAL__, __AMAZON_INTERNAL__, distributed_snapshot_deletion_enabled=true} (tried [1] times) ConnectTransportException[[5fSCYAe][__IP__] connect_exception]; nested: AnnotatedConnectException[Connection refused: __PATH__]; nested: ConnectException[Connection refused]; at org.elasticsearch.transport.TcpChannel.awaitConnected(TcpChannel.java:165) at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:643) at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:542) at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:316) at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:153) at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:106) at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:749) Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: __PATH__ at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633) at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ... 1 more Caused by: java.net.ConnectException: Connection refused ... 10 more [2022-01-29T09:05:58,417][WARN ][o.e.c.NodeConnectionsService] [bmDFGK_] failed to connect to node {5fSCYAe}{5fSCYAeFQbefAxIiS7YyVA}{03U8UySaTI6gDrlMk5RP2Q}{__IP__}{__IP__}{__AMAZON_INTERNAL__, __AMAZON_INTERNAL__, distributed_snapshot_deletion_enabled=true} (tried [1] times) ConnectTransportException[[5fSCYAe][__IP__] connect_exception]; nested: AnnotatedConnectException[Connection refused: __PATH__]; nested: ConnectException[Connection refused]; at org.elasticsearch.transport.TcpChannel.awaitConnected(TcpChannel.java:165) at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:643) at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:542) at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:316) at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:153) at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:106) at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:749) Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: __PATH__ at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633) at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ... 1 more Caused by: java.net.ConnectException: Connection refused ... 10 more ```
1
answers
1
votes
15
views
asked 4 months ago

ElasticSearch Domain not responsive anymore

Our ElasticSearch Domain unexpectedly stopped responding. Since we cannot look into the logs anymore we do not have the chance to find out the reason for that. We noticed that the ebs volume appeared to be at 3GB size, however we cannot verify if this indeed is the reason why the elasticsearch domain is not responsive anymore. We also do not see any possibility to find out why this happend (as mentioned above, logs are not accessible). We started an update of the ebs volume to double in size using the AWS webconsole, as we expected that new space might help. The update was marked as 'green' so we agreed on starting it. Even after 24 hours the update is stuck in the state 'Processing'. As described in the documentation, this means that the update of the ebs volume failed (https://aws.amazon.com/premiumsupport/knowledge-center/opensearch-domain-stuck-processing/). We have found a long list of reasons for why this update may have failed: - CDI Failure: creation of new DI failed - Customer makes simultaneous configuration changes on the cluster - Failed CDI due to overloaded mastr node or any other activity failures like insufficient IP addresses in the customer subnet - Nodes went out of service due to heavy processing load or internal hardware failure - Previous DDI Failure: New CDI executed before Previous DDI activities are completed. - Large number of shards and continues node failure due to high JVM memory Pressure and CPU Usage. - Nodes went out of service due to heavy processing load or internal hardware failure - Stuck shard relocation due to insufficient free storage in the new DI, custom shard routing and service/es issues - Single domain can usually have 2 DIs at maximum. If CDI happens before previous DDI is completed, new CDI is queued but won't be executed. But this list does not help to solve the issue that our ElasticSearch Domain is currently in a state that we cannot change. The only action that we see is deleting the domain and start a new one. However we would like to ask if there is some action that AWS can do for bringing the domain back to life. We need the logs that are within the domain.
1
answers
0
votes
3
views
asked 4 months ago

AWS DMS + OpenSearch + Index templates

I'm migrating some data from postgres to OpenSearch, but i'm struggling with migrating a set of coordinates. In postgres i have `latitude` and `longitude`, and i know DMS does not support geo types when using OpenSearch as a target. I wanted to work around this by using an ingest pipeline and an index template: ``` PUT _ingest/pipeline/my-pipeline-id { "description": "My optional pipeline description", "processors": [ { "set": { "field": "location.lon", "value": "{{{longitude}}}" } }, { "set": { "field": "location.lat", "value": "{{{latitude}}}" } } ] } ``` ``` PUT /_index_template/jobs_template { "index_patterns": [ "jobs*" ], "template": { "settings": { "index.default_pipeline": "my-pipeline-id" }, "mappings":{ "properties": { "location": { "type": "geo_point" } } } } } ``` I first tested having only the pipeline without the mappings and that part works. However, when i add a mapping to the template, and re run the migration task i get the following error ``` 2022-01-20T15:23:01 [TARGET_LOAD ]E: Elasticsearch:FAILED SourceTable:jobs TargetIndex:jobs Operation:INSERT_ENTRY RecordPKKey:93904e3c-5565-4469-94d6-e58fbecdc5a3 RecordPKID:217D5CE32D4EC983FE2C3CFD6048821EA2A95F3658122A80A7EEB3A6088EA89CES HttpCode:400 ESErrorResponse: { "error": { "root_cause": [ { "type": "illegal_argument_exception", "reason": "Rejecting mapping update to [jobs] as the final mapping would have more than 1 type: [_doc, doc]" } ], "type": "illegal_argument_exception", "reason": "Rejecting mapping update to [jobs] as the final mapping would have more than 1 type: [_doc, doc]" }, "status": 400 } ``` Is there any way to make this work? Any way to create a geo point so I can perform geo queries in OpenSearch?
0
answers
0
votes
3
views
asked 4 months ago

AWS DMS Postgres to OpenSearch LOB handling

Source: postgres Target: OpenSearch I have a `text` column called `description` in one of my postgres tables. Per the documentation, this data type is mapped to a `NCLOB`. Since OpenSearch does not not offer LOB support, my `description` is missing in my OpenSearch documents. I tried using the mapping rule bellow, but does not seem to be doing anything ``` { "rule-type": "transformation", "rule-id": "3", "rule-name": "3", "rule-target": "column", "object-locator": { "schema-name": "public", "table-name": "jobs", "column-name": "description" }, "rule-action": "change-data-type", "data-type": { "type": "string", "length": 500 } } ``` When i check the logs i see the following ``` Column 'description' is unsupported in table def 'public.jobs' since the LOB support is disabled ``` However, i do have LOB enabled under task settings: ``` "TargetMetadata": { "ParallelApplyBufferSize": 0, "ParallelApplyQueuesPerThread": 0, "ParallelApplyThreads": 0, "TargetSchema": "", "InlineLobMaxSize": 0, "ParallelLoadQueuesPerThread": 0, "SupportLobs": true, "LobChunkSize": 10, "TaskRecoveryTableEnabled": false, "ParallelLoadThreads": 0, "BatchApplyEnabled": false, "FullLobMode": true, "LimitedSizeLobMode": false, "LoadMaxFileSize": 0, "ParallelLoadBufferSize": 0 }, ``` Is that transformation rule supposed to work? Or will any LOB column be skipped because OpenSearch does not have LOB support? Any way to make this work? Thanks!
1
answers
0
votes
20
views
asked 4 months ago

AWS ElasticSearch returning DIFFERENT results in Kibana and http request in browser for the exact same query

I am running this kibana query: I have this query in Kibana: GET nearby/_search { "from": 20, "size":20, "query": { "bool": { "must": { "match": { "X": "B" } }, "filter": { "geo_distance": { "distance": "3.0km", "PO": { "lat": 26.8466937, "lon": 80.94616599999999 } } } } } } and response to this is: all the responses are with X=B: 20 results are there, i have removed some fields and some docs to keep the post short { "took" : 228, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 71, "relation" : "eq" }, "max_score" : 2.5032558, "hits" : [ { "_index" : "nearby", "_type" : "_doc", "_id" : "n3YeKvJqvpu1okE7QDBp", "_score" : 2.2831507, "_source" : { "PO" : "tuc89gfn0", "X" : "B" } }, { "_index" : "nearby", "_type" : "_doc", "_id" : "5FPJ2eyr0YoQ9F0xPYzW", "_score" : 2.2831507, "_source" : { "PO" : "tuc89gfn0", "X" : "B" } }, { "_index" : "nearby", "_type" : "_doc", "_id" : "QJflnqGKF1dpOjEaY8vy", "_score" : 2.2831507, "_source" : { "PO" : "tuc89gvr8", "X" : "B" } }] } } This is the browser REQUEST, QUERY REMAINS SAME: https://search-wul8888888.ap-south-1.es.amazonaws.com/nearby/_search?q="{"from":20,"size":20,"query":{"bool":{"must":{"match":{"X":"B"}},"filter":{"geo_distance":{"distance":"3km","PO":{"lat":26.8466937,"lon":80.94616599999999}}}}}}" This is the response: as u can see there are mostly X=I docs i.e. must-match isnt honoured, SECOND THING IS THAT I AM SENDING SIZE=20 BUT I GET 10 REULTS ONLY WHICH IS DEFAULT(BELOW I HAVE REMOVED EXTRA docs TO KEEP THE POST SHORT) {"took":149,"timed_out":false, "_shards":{"total":5,"successful":5,"skipped":0,"failed":0}, "hits":{"total":{"value":802,"relation":"eq"},"max_score":8.597985, "hits":[ {"_index":"nearby","_type":"_doc","_id":"iJ71MNq4a4TCkcT4vWSP","_score":8.597985,"_source":{//EXTRA FIELDS REMOVED "PO":"tuc8unwp7","X":"I","BI":"tRhKrWiDxFSt57tIH7g5"}}, {"_index":"nearby","_type":"_doc","_id":"PmngNe8WcC8aSraDMluz","_score":7.3973455,"_source":{"PO":"tuc8uhc5z",**"X":"I"**,"BI":"m3S6yEicvu1HFI1UOTIb"}}, {"_index":"nearby","_type":"_doc","_id":"lDqjflPZGYsymPGU8iHD","_score":7.1520696,"_source":{"PO":"tuc89wpg5","X":"B"}}, {"_index":"nearby","_type":"_doc","_id":"QIf2KsO4FpCjT3m7kH4I","_score":6.402881,"_source":{"PO":"tuc8uhc5z","X":"I","BI":"m3S6yEicvu1HFI1UOTIb"}}]}} PLEASE HELP. I TRIED EVERYTHING BUT NOT ABLE TO UNDERSTAND. MY HUNCH IS THAT EVERY TIME I M BEING RETURNED A STALE/old RESULT BUT dont know how to fix that. even in chrome incognito mode result for browser is same as above. Even if i change the radius in browser, result remains same which says clearly browser queries are getting the stale result.
0
answers
0
votes
5
views
asked 4 months ago

OpenSearch cluster green but stuck in processing for 2 weeks

My OpenSearch cluster has been stuck in processing since the last auto-tune event. the cluster status is green across the board. The cluster is usable without issue (reading, writing, Kibana), but this prevents me from performing an upgrade or applying other config changes. Monitoring shows: * Cluster status green * Instance count is 9, as expected: 3 master and 6 data nodes * JVM memory pressure looks good: seeing the expected "sawtooth" curve never exceed 75% and going as low as 45% * Most recent update was a Service software update to R20211203-P2. It seems to have taken 5 days, but looks like it completely well. (Judging by the instance count graph) * The cluster is usable without issue, Kibana is reachable and responsive, constantly writing to the cluster without error, nothing seems off Rough timeline: * 19.12.2021 - update to R20211203-P2, instance count is doubled to 18 (expected blue/green deployment) * 24.12.2021 - instance count drops back to the expect 9, cluster status green * 26.12.2021 - Notification "Auto-Tune is applying new settings to your domain", instance count doesn't rise, still at 9 * now - Cluster still stuck at "processing" even though everything is green What I tried: * `GET /_cluster/allocation/explain` responds with "unable to find any unassigned shards to explain" which makes sense * ` GET /_cat/indices?v` shows everything green, as expected * I also tried modifying the disk size to try and "kick" the cluster into doing a blue/green deployment and hopefully getting unstuck but that didn't seem to happen The only possible clue was in CloudWatch error logs, a repeating message appears since the last auto-tune event started on 26.12.2021: with "master not discovered yet", I'll try to pretty-print it below: ``` [2022-01-11T06:36:23,761][WARN ][o.o.c.c.ClusterFormationFailureHelper] [52cb02d8573b17516f7756d5fe05484d] master not discovered yet: have discovered [ {***}{***}{***}{__IP__}{__IP__}{dir}{dp_version=20210501, distributed_snapshot_deletion_enabled=false, cold_enabled=false, adv_sec_enabled=false, __AMAZON_INTERNAL__, shard_indexing_pressure_enabled=true, __AMAZON_INTERNAL__}, {***}{***}{***}{__IP__}{__IP__}{imr}{dp_version=20210501, distributed_snapshot_deletion_enabled=false, cold_enabled=false, adv_sec_enabled=false, __AMAZON_INTERNAL__, shard_indexing_pressure_enabled=true, __AMAZON_INTERNAL__}, {***}{***}{***}{__IP__}{__IP__}{imr}{dp_version=20210501, distributed_snapshot_deletion_enabled=false, cold_enabled=false, adv_sec_enabled=false, __AMAZON_INTERNAL__, shard_indexing_pressure_enabled=true, __AMAZON_INTERNAL__}, {***}{***}{***}{__IP__}{__IP__}{imr}{dp_version=20210501, distributed_snapshot_deletion_enabled=false, cold_enabled=false, adv_sec_enabled=false, __AMAZON_INTERNAL__, shard_indexing_pressure_enabled=true, __AMAZON_INTERNAL__}, {***}{***}{***}{__IP__}{__IP__}{imr}{dp_version=20210501, distributed_snapshot_deletion_enabled=false, cold_enabled=false, adv_sec_enabled=false, __AMAZON_INTERNAL__, shard_indexing_pressure_enabled=true, __AMAZON_INTERNAL__}, {***}{***}{***}{__IP__}{__IP__}{imr}{dp_version=20210501, distributed_snapshot_deletion_enabled=false, cold_enabled=false, adv_sec_enabled=false, __AMAZON_INTERNAL__, shard_indexing_pressure_enabled=true, __AMAZON_INTERNAL__}, {***}{***}{***}{__IP__}{__IP__}{imr}{dp_version=20210501, distributed_snapshot_deletion_enabled=false, cold_enabled=false, adv_sec_enabled=false, __AMAZON_INTERNAL__, shard_indexing_pressure_enabled=true, __AMAZON_INTERNAL__} ]; discovery will continue using [__IP__, __IP__, __IP__, __IP__, __IP__, [__IP__]:9301, [__IP__]:9302, [__IP__]:9303, [__IP__]:9304, [__IP__]:9305, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__, __IP__] from hosts providers and [] from last-known cluster state; node term 36, last-accepted version 0 in term 0 ``` I masked the node IDs and replaced them with `***`. The log message lists 7 of them above, I can only recognize 3 IDs as my master nodes, cannot recognize the rest of the IDs (not my data nodes) and not sure I understand what's going on here. Any help would be appreciated.
1
answers
0
votes
8
views
asked 4 months ago
3
answers
0
votes
17
views
asked 4 months ago

Multi Region strategy for API Gateway

If disaster recovery is not a requirement, what would be the best strategy for setting up API Gateway to server global customers. Here are three options that I can think of, not able to land on one. **Option 1**: Single Edge Optimized API Gateway serving traffic * Pros: save cost and avoid complexity of data replication (backend is opensearch) * Cons: Latency? not sure how much edge optimized API will help with latency, as customer will be hitting the API at nearest edge (ssl handshake, etc) and traffic flowing via backbone network. ( Question 1) **Option 2** Multiple Regional API Gateway with Route53 Latency based routing * Pros: customers hitting closest API. * Cons: Data replication, Cost. Also, since there is no cloud front here, traffic will be flowing via internet to closest region API, lets say we have API deployed in two regions , US and Singapore, would users in Europe see latency , worse than the Option 1, where requests are going to nearest edge location and reaches API via backbone? **Option 3** Multiple Edge Optimized API Gateway with Route53 Latency based routing * Pros: customers hitting closest API. Not sure how latency based routing works on an edge optimized endpoint, would it even help, since both endpoints are edge optimized. Not sure how smart is Route53 (Question 2) * Cons: Data replication, cost and uncertainty of Latency based routing. and Finally , one that I can think of could work, but haven't found too many solutions where people implemented. **Option 4** Multiple Regional API Gateway with single custom Cloudfront on top with cloudfront functions to do the routing. * Pros: customers hitting closest edge optimized location and routed to nearest API, this routing will be based on country of origin header from cloudfront. * Cons: Same Data Replication, Cost and predefined list of countries based routing. I need to spend time and run tests with multiple solutions. But wanted to seek community advise first. To summarize everything, if cost, complexity and disaster recovery are kept out of discussion, what would be best architecture for API Gateway to avoid latency issues.
2
answers
0
votes
30
views
asked 5 months ago

ElasticSearch Container failed to start - ECS deployment using docker compose up - /usr/share/elasticsearch/data/nodes/ AccessDeinedException

Hi I'm trying to start an elasticsearch container via - docker compose (aws-cli and switching to ecs context), but it fails to start - AccessDeinedExcception - cant write to /usr/share/elasticsearch/data/nodes/ directory. I have researched the issue on google and its because of the permission on that folder - from my understanding I need to fix the permissions in the host directory mapped to /usr/share/elasticsearch/data/nodes/ (I think) running sudo chown -R 1000:1000 [directory} However my container shuts down and how am I supposed to update the permission on that directory? this is my docker-compose - any help appreciated version: '3.8' services: elasticsearch01: user: $USER image: docker.elastic.co/elasticsearch/elasticsearch:7.14.1 #image: 645694603269.dkr.ecr.eu-west-2.amazonaws.com/smpn_ecr:latest container_name: es02 restart: unless-stopped environment: cluster.name: docker-es-cluster discovery.type: single-node bootstrap.memory_lock: "true" # ES_JAVA_OPTS: "-Xms2g -Xmx2g" xpack.security.enabled: "false" xpack.monitoring.enabled: "false" xpack.watcher.enabled: "false" node.name: es01 network.host: 0.0.0.0 logger.level: DEBUG ulimits: memlock: soft: -1 hard: -1 volumes: - es_data01:/usr/share/elasticsearch/data:rw ports: - "9200:9200" - "9300:9300" healthcheck: test: "curl -f http://localhost:9200 || exit 1" networks: - smpn_network volumes: es_data01: driver: local networks: smpn_network: driver: bridge
0
answers
0
votes
15
views
asked 5 months ago

Red ElasticSearch 5.5 cluster due to NODE_LEFT with running snapshot

There is a cluster that, due to losing a couple of nodes has a single shard in UNASSIGNED state. **TL;DR;**: The shard can not be rerouted due to AWS limitations, index can not be deleted due to running snapshot (for over 18 hours now), cluster has scaled to double its regular size for no obvious reason and snapshot can not be cancelled because it is one of the automated ones. What could be done to get the cluster back to green health? Data loss of that single index should not be a problem. ## Detailed explanation ### Symptom Cluster in red health status due to a single unnasigned shard. A call to `/_cluster/allocation/explain` returns the following: ``` { "index": "REDACTED", "shard": 1, "primary": true, "current_state": "unassigned", "unassigned_info": { "reason": "NODE_LEFT", "at": "2021-12-01T21:27:04.905Z", "details": "node_left[REDACTED]", "last_allocation_status": "no_valid_shard_copy" }, "can_allocate": "no_valid_shard_copy", "allocate_explanation": "cannot allocate because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster", ... ``` ### Cluster rerouting Regular troubleshooting on the matter indicates that one could take the data loss by reallocating the shard to empty using something like: ``` $ curl -XPOST '/_cluster/reroute' -d '{"commands": [{ "allocate_empty_primary": { "index": "REDACTED", "shard": 1, "node": "REDACTED", "accept_data_loss": true }}] }' {"Message":"Your request: '/_cluster/reroute' is not allowed."} ``` But that endpoint is not available in AWS. ### Closing/Deleting the index Other suggestions include closing the index for operations, but that is not supported by AWS: ``` $ curl -X POST '/REDACTED/_close' {"Message":"Your request: '/REDACTED/_close' is not allowed by Amazon Elasticsearch Service."} ``` Another solution is to delete the index. But, as there is a running snapshot, it can not be deleted: ``` $ curl -X DELETE '/REDACTED' {"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[REDACTED][indices:admin/delete]"}],"type":"illegal_argument_exception","reason":"Cannot delete indices that are being snapshotted: [[REDACTED]]. Try again after snapshot finishes or cancel the currently running snapshot."},"status":400} ``` ### Cancelling the snapshot As the previous error message states, you can try cancelling the snapshot: ``` curl -X DELETE '/_snapshot/cs-automated-enc/REDACTED' {"Message":"Your request: '/_snapshot/cs-automated-enc/REDACTED' is not allowed."} ``` Apparently that is because the snapshot is part of the automated ones. Had it been a manual snapshot I would have been able to cancel it. Problem is that the snapshot has been running for over 10 hours and is still initializing: ``` $ curl '/_snapshot/cs-automated-enc/REDACTED/_status' { "snapshots": [ { "snapshot": "2021-12-12t20-38-REDACTED", "repository": "cs-automated-enc", "uuid": "REDACTED", "state": "INIT", "shards_stats": { "initializing": 0, "started": 0, "finalizing": 0, "done": 0, "failed": 0, "total": 0 }, "stats": { "number_of_files": 0, "processed_files": 0, "total_size_in_bytes": 0, "processed_size_in_bytes": 0, "start_time_in_millis": 0, "time_in_millis": 0 }, "indices": {} } ]} ``` As it can be seen from the timestamp, it has been that way for almost 20 hours now (for reference, previous snapshots show to have run in a couple of minutes).
1
answers
0
votes
7
views
asked 5 months ago

_nodes/http info missing lots of info

The Elasticsearch client for golang uses info derived from the follow ES api call: GET /_node/http <https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-info.html> Normally this call returns lot of information about the node including IP address but when making the call to the AWS service this information is missing. This causes the golang client to fail to be able to _discover_ any nodes. Here are two examples which highlight the problem: regular node i have running locally: curl "http://localhost:9200/_nodes/http?pretty" { "_nodes" : { "total" : 1, "successful" : 1, "failed" : 0 }, "cluster_name" : "docker-cluster", "nodes" : { "hnsnurlLTG2-XbBQFyYg9w" : { "name" : "16c3f8d15864", "transport_address" : "172.25.0.4:9300", "host" : "172.25.0.4", "ip" : "172.25.0.4", "version" : "7.8.0", "build_flavor" : "default", "build_type" : "docker", "build_hash" : "757314695644ea9a1dc2fecd26d1a43856725e65", "roles" : \[ "data", "ingest", "master", "ml", "remote_cluster_client", "transform" ], "attributes" : { "ml.machine_memory" : "25225474048", "xpack.installed" : "true", "transform.node" : "true", "ml.max_open_jobs" : "20" }, "http" : { "bound_address" : \[ "0.0.0.0:9200" ], "publish_address" : "172.25.0.4:9200", "max_content_length_in_bytes" : 104857600 } } } } AWS equivalent: curl -XGET https://vpc-address....amazonaws.com/_nodes/http?pretty { "_nodes" : { "total" : 3, "successful" : 3, "failed" : 0 }, "cluster_name" : "XXX", "nodes" : { "8oxT_9maSMif5iXoQptpKA" : { "name" : "d026e0070020663bdd93028c2c801292", "version" : "7.7.0", "build_flavor" : "oss", "build_type" : "tar", "build_hash" : "unknown", "roles" : \[ "ingest", "master", "data", "remote_cluster_client" ] }, "8joBdfy6Q-aULryA3aP00w" : { "name" : "3992f18f3b80c28fbb6d5b5624c4bdfe", "version" : "7.7.0", "build_flavor" : "oss", "build_type" : "tar", "build_hash" : "unknown", "roles" : \[ "ingest", "master", "data", "remote_cluster_client" ] }, "2v572nbWQK6zrXmwGUiRmQ" : { "name" : "41d42ffdc2fdaf9c7a425e29c6ba92d6", "version" : "7.7.0", "build_flavor" : "oss", "build_type" : "tar", "build_hash" : "unknown", "roles" : \[ "ingest", "master", "data", "remote_cluster_client" ] } } } Obviously there are going to be some differences local Vs AWS with things like plugins but returning the actually http information seems pretty fundemental no?
1
answers
0
votes
1
views
asked a year ago
  • 1
  • 90 / page