How do I use Filebeat and Logstash on Amazon Linux to connect to OpenSearch Service?

Resolution

Set up security ports

To use Logstash to connect to OpenSearch Service, set up your security ports to forward logs from Logstash.

Complete the following steps:

1. Launch an Amazon Elastic Compute Cloud (Amazon EC2) instance for Filebeat, and then launch another instance for Logstash to send logs from.
2. Confirm that your EC2 instances are in the same security group as your virtual private cloud (VPC) for OpenSearch Service.
3. Make sure that ports 80, 443, and 5044 are open in your security group so that you can send data between Logstash and OpenSearch Service.

Update Filebeat, Logstash, and OpenSearch Service configurations

It's a best practice to use Filebeat and Logstash versions that match your OpenSearch Service with a legacy Elasticsearch version. Use Filebeat and Logstash versions that match your domain version.

For example, if you use OpenSearch Service 1.x with compatibility mode, then OpenSearch Service reports the version as 7.10. Check the compatibility matrix to choose the correct version of Filebeat and Logstash. For more information, see Support matrix on the Elastic website or Compatibility matrices on the OpenSearch website.

If you use OpenSearch Service version 1.0 or later, then turn on compatibility mode when you launch your domain.

To make sure that the downloaded software remains in sync, download Red Hat Package Managers (RPMs) to each instance. To prevent a single point of failure in your pipeline, don't run Filebeat and Logstash on the same instance.

Install Filebeat on the source EC2 instance

To install Filebeat on the source instance, complete the following steps:

1. Use SSH to connect to the instance that you launched for Filebeat.

2. Run the following command to download the RPM for the compatible Filebeat version:

   wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-oss-8.15.2-x86_64.rpm

3. Run the following command to install the Filebeat RPM file:

   rpm -ivh filebeat-oss-8.15.2-x86_64.rpm

Install and configure Logstash on a separate EC2 instance

Complete the following steps:

1. Use SSH to connect to your instance that you launched for Logstash.

2. Run the following command to download the compatible Logstash RPM file:

   wget https://artifacts.elastic.co/downloads/logstash/logstash-oss-8.15.2-x86_64.rpm

3. Run the following command to install Java or OpenJDK on your EC2 instance:

   yum install java-1.8.0-*

   Note: The preceding command uses Java version 8 (OpenJDK 1.8). All Logstash versions support Java version 8.

4. Run the following command to install the Logstash RPM file:

   rpm -ivh logstash-oss-8.15.2-x86_64.rpm

5. Run the following command to open the Logstash data directory:

   cd /usr/share/logstash/

6. Run the following command to install the OpenSearch Service plugin for Logstash:

   bin/logstash-plugin install logstash-output-opensearch

7. To verify the Logstash plugin installation, run the following command:

   bin/logstash-plugin list

   Note: For Logstash OSS 7.13 and later, use only the logstash-output-opensearch plugin. Version 7.12.1 supports both logstash-output-elasticsearch and logstash-output-opensearch plugins.

8. To verify that the configuration directories exist, check the /etc/filebeat and /etc/logstash directories.

9. To send logs through Logstash, create a Filebeat configuration file at /etc/filebeat/filebeat.yml.
   Example Filebeat configuration file:

   filebeat.inputs:
   - type: log
     enabled: true
     paths:
       - /var/log/*.log
    
   filebeat.config.modules:
     path: ${path.config}/modules.d/*.yml
     reload.enabled: false
    
   setup.template.settings:
    
   index.number_of_shards: 1
   index.codec: best_compression
    
   #output.elasticsearch:
   #hosts: ["your-domain-endpoint"]
   #protocol: "https"
    
   output.logstash:
     # The Logstash hosts
     hosts: ["Logstash-EC2-InstanceIP:5044"]
    
   setup.ilm.enabled: false 
   ilm.enabled: false

   Note: Replace Logstash-EC2-InstanceIP:5044 with your Logstash instance's IP address and your-domain-endpoint with your domain endpoint. You can also send logs directly to OpenSearch Service. For more information, see logstash-output-opensearch on the GitHub website.

10. Create a Logstash configuration file at /etc/logstash/conf.d/logstash.conf.
    Example Logstash configuration file:

    input {
      beats {
        port => 5044
      }
    }
    
    output {
      opensearch {
        hosts => ["https://domain-endpoint:443"]
        ssl => true
        index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
        user => "my-username"
        password => "my-password"
      }
    }

    Note: Replace domain-endpoint, my-username and my-password with your values.

11. To test connectivity to your OpenSearch Service endpoint, run the following command:

    curl -XGET https://your-domain-endpoint

    Note: Replace your-domain-endpoint with your domain endpoint.

12. (Optional) If you activated fine-grained access control (FGAC), then run the following command:

    curl -u my-username:my-password -XGET https://your-domain-endpoint

    Note: Replace your-domain-endpoint with your domain endpoint.

13. Run the following commands to start the Filebeat and Logstash services:

    systemctl start filebeat

    systemctl start logstash

14. To confirm that the Filebeat logs are sent, run the following curl command to your OpenSearch Service domain:

    curl -XGET https://your-domain-endpoint/_cat/indices

    Note: Replace your-domain-endpoint with your domain endpoint.
    Example outputs of a Filebeat index:

    green open filebeat-7.16.2-2022.01.27 f97c4WnuQ-CtsAJJaJHUlg
    1 1 1511515 0 249.7mb 124.7mb
    green open .kibana_1                 Ioco6fUoSCGkaOvHNCL39g 1
    1       1 0   7.4kb  
    3.7kb

    green open filebeat-7.16.2-2022.01.27 f97c4WnuQ-CtsAJJaJHUlg
    1 1 1511515 0 249.7mb 124.7mb
    green open .kibana_1                 Ioco6fUoSCGkaOvHNCL39g 1
    1       1 0   7.4kb  
    3.7kb
    green open filebeat-7.16.2-2022.01.28 4i8W0smlRGGFcQOaDMxonA
    1 1      89 0 207.1kb 118.1kb

If you successfully configured Filebeat, Logstash, and OpenSearch Dashboards with Amazon EC2 Linux, then your pipeline looks similar to the following example:

Filebeat > Logstash > Amazon OpenSearch Service/Dashboards

Troubleshoot 401, 403, and Logstash x-pack installation errors

401 Unauthorized error

If Logstash returns a "401 Unauthorized" error, then your OpenSearch Service domain uses one of the following security features:

• FGAC
• Amazon Cognito authentication

To resolve this issue, add authentication credentials to your Logstash configuration file at /etc/logstash/conf.d/logstash.conf.

Example configuration:

output {
  opensearch {
    hosts => ["https://your-domain-endpoint"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    ilm_enabled => false
    user => "my-username"
    password => "my-password"
  }
}

Note: Replace your-domain-endpoint, my-username, and my-password with your OpenSearch Service domain's values.

403 Forbidden error

If Logstash doesn't have the required AWS Identity and Access Management (IAM) permissions to access OpenSearch Service, then a "403 Forbidden" error occurs.

To resolve this issue, complete the following steps:

1. Run the following command to install the OpenSearch Service plugin for Logstash:

   bin/logstash-plugin install logstash-output-opensearch

2. Create an IAM role.

3. Create the following policy, and then attach it to the IAM role:

   {  
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "es:ESHttp*"
         ],
         "Resource": "[Amazon-OpenSearch-Domain-ARN]"
       }
     ]
   }

   Note: Replace Amazon-OpenSearch-Domain-ARN with your OpenSearch Service domain's Amazon Resource Name (ARN).

4. Update your Logstash configuration file at /etc/logstash/conf.d/logstash.conf.
   Example Logstash configuration file:

   output {
     opensearch {
       hosts => ["domain-endpoint"]
       ssl => true
       region => "example_region"
       index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
     }
   }

   Note: Replace domain-endpoint and example-region with your OpenSearch Service domain's values.

Logstash x-pack installation error

If Logstash fails to start because of a "x-pack plugin" error, then complete the following steps:

1. Open the following registry file:

   /usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb

2. Find the following line of code:

   load_xpack unless LogStash::OSS

3. Update the code to the following example:

   #load_xpack unless LogStash::OSS

4. To prevent x-pack plugin startup errors, turn off Index Life Management (ILM) settings on your configuration files:
   In Filebeat, configure ilm.enabled: false.
   In Logstash, configure ilm_enabled => false.

Related information

Loading data into Amazon OpenSearch Service with Logstash