using ingest-geoip or Nginx filebeat/metricbeat module on Amazon ElasticSearch
I am trying to use the Nginx module for filebeat/metricbeat, which in turn seems to require ingest-geoip This is the error they got:
Dec 03 08:37:45 ip-10-1-2-5 filebeat[30775]: 2020-12-03T08:37:45.077Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://<OUR_AWS_ELK_INSTANCE>)): Connection marked as failed because the onConnect callback failed: Error loading pipeline for fileset nginx/access: This module requires the following Elasticsearch plugins: ingest-geoip. You can install them by running the following commands on all the Elasticsearch nodes:
Dec 03 08:37:45 ip-10-1-2-5 filebeat[30775]: sudo bin/elasticsearch-plugin install ingest-geoip
Is there a way to install ingest-geoip or any other workaround to use Nginix module for filebeat/metricbeat on ES?
As of now Amazon Elasticsearch service does not have the ingest-geoip module built in. So, there are 2 ways you can tackle this error:
1) Use logstash: In this method instead of sending data from Filebeat -> Elasticsearch, send it via logstash. You can do something like Filebeat -> Logstash -> Elasticsearch.
In this case add the geoip filter in logstash and enrich the data for IP. A sample conf may look like:
input {
beat { .. }
}
filter {
geoip {
source => "ip_field_name"
}
}
output {
elasticsearch { .. }
}
2) Skip the geoip parsing and just send the data to Elasticsearch. You won't get the geo details extracted, but you can still send the rest of data to Elasticsearch.
For this go to your filebeat installation path, for example: filebeat-7.10.0-darwin-x86_64/module/nginx/access/ingest/pipeline.yml and comment out or remove the section related to geoip.
- geoip:
field: source.ip
target_field: source.geo
ignore_missing: true
- geoip:
database_file: GeoLite2-ASN.mmdb
field: source.ip
target_field: source.as
properties:
- asn
- organization_name
ignore_missing: true
Relevant questions
using ingest-geoip or Nginx filebeat/metricbeat module on Amazon ElasticSearch
Accepted Answerasked a year agoHow do you install dnf on Amazon Linux 2?
asked 3 years agoBeanstalk node.js amazon linux 2 running on nginx 1.20.0 which has critical vulnerability CVE-2021-23017
asked 3 months agoCrypto Module Error when using AWS4Axios
asked a month agoUnable to resolve "client intended to send too large body" using extensions
asked 2 years agoUnable to modify nginx config on CodeBuild deployment to EBS
asked a year agoError when trying to import package installed using lambda layer on nodejs14 ES module enabled lambda function
asked 4 months agoAmazon CloudFront serving compressed files with Origin server running NGINX
Accepted Answerasked 2 years agoLoad Balancer HTTP 413 Error
asked a month agoHow to correctly quote task command / entrypoint ?
asked 2 years ago