Loading data to ES cost comparison

0

A customer is currently crawling the web and writing to S3 + loading to Amazon Elasticsearch. The current architecture looks like this.

ECS -> EC2 -> S3 -> EC2 -> ES

Looks like they set this up a long time ago, and would like to optimize the workflow. They are using ECS (they have containerized old code written in C#), running that through EC2 to write to S3, and then using another EC2 instance to load data into ES. This seems like a very inefficient solution. There also seems to be a lot of unknown around ECS and crawler code, and they would not like to touch it. Given this information, I thought the following architecture would make more sense.

ECS + Kinesis agent -> Kinesis Firehose + Lambda -> ES (and S3 through backup)

What I would like to know is

  1. Does the above workflow make sense?
  2. Customer is concerned about price. While it is difficult to get an exact estimate, I was hoping to get some insight into if anyone has done a similar comparison. This customer has a small team and would not like to commit resources to end up with a more expensive solution.
AWS
EXPERT
john_l
asked 4 years ago218 views
1 Answer
0
Accepted Answer

Yes, proposed workflow makes lot of sense. Migrating this workload on Kinesis Firehose would provide lot of benefits over current architecture like auto scaling, high availability, no administration, ease of use etc.

Firehose pricing is based on volume of data you ingest into Firehose plus charges for data format conversion to Parquet or ORC which is not the requirement here.

Refer to link https://aws.amazon.com/kinesis/data-firehose/pricing/

Pricing example#4 explains how to calculate pricing if Firehose loads data into Elasticsearch cluster. You need to collect volumetric in terms of size of record, count of records/second to be able to calculate monthly pricing of Firehose delivery stream and compare the same with existing solution.

answered 4 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions