Loading data to ES cost comparison

0

A customer is currently crawling the web and writing to S3 + loading to Amazon Elasticsearch. The current architecture looks like this.

ECS -> EC2 -> S3 -> EC2 -> ES

Looks like they set this up a long time ago, and would like to optimize the workflow. They are using ECS (they have containerized old code written in C#), running that through EC2 to write to S3, and then using another EC2 instance to load data into ES. This seems like a very inefficient solution. There also seems to be a lot of unknown around ECS and crawler code, and they would not like to touch it. Given this information, I thought the following architecture would make more sense.

ECS + Kinesis agent -> Kinesis Firehose + Lambda -> ES (and S3 through backup)

What I would like to know is

  1. Does the above workflow make sense?
  2. Customer is concerned about price. While it is difficult to get an exact estimate, I was hoping to get some insight into if anyone has done a similar comparison. This customer has a small team and would not like to commit resources to end up with a more expensive solution.
AWS
전문가
john_l
질문됨 4년 전225회 조회
1개 답변
0
수락된 답변

Yes, proposed workflow makes lot of sense. Migrating this workload on Kinesis Firehose would provide lot of benefits over current architecture like auto scaling, high availability, no administration, ease of use etc.

Firehose pricing is based on volume of data you ingest into Firehose plus charges for data format conversion to Parquet or ORC which is not the requirement here.

Refer to link https://aws.amazon.com/kinesis/data-firehose/pricing/

Pricing example#4 explains how to calculate pricing if Firehose loads data into Elasticsearch cluster. You need to collect volumetric in terms of size of record, count of records/second to be able to calculate monthly pricing of Firehose delivery stream and compare the same with existing solution.

답변함 4년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠