By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Besides the operational costs and hourly costs - why should i use AWS Timestream instead of OpenSearch? for metrics ingestions and querying

0

I have a use case where I ingest metrics into the system, those can be various types of metrics, and eventually tenants in the system will have dashboard to Visualize those metrics, and create reports. Users might be able to also create their own dashboards.

I'm debating between Timestream and OpenSearch, both can achieve IAM fine grain permissions (per TS table / elastic index), and i know there are operational costs (storage / scaling) for ES, but I have 2 concerns about TS which i'd love to discuss:

  • Timestream has maximum 128 dimensions, which means to have properties on my metrics, i might need to use generic d1, d2, d3 dimensions

  • queries are priced per 10MB minimum - This is the BIGGEST problem:

  • I can't create a dashboard with different queries, because if i'll visualize 20 metrics, i'll have to pay for 200MB of scan - which you multiply by 10,000 customers, every few hours - this will be horrible

  • I will have to to aggregate everything into one SQL call with UNION -

(Select ... from table) UNION (Select ... from table) UNION (select ... from table)

just to reduce query costs - this sounds super weird, and hard to do if there are different columns and results in each query.

  • I'll need to restrict the dashboard from realtime to every few hours, to reduce querying costs

in opensearch i don't have those.

Your thoughts?

Topics
DatabaseServerlessAnalytics
Tags
DatabaseAmazon OpenSearch ServiceAmazon Timestream
Language
English
ArielB
asked a year ago269 views
2 Answers
0

I already have a request in for AWS to add caching in Grafana for Timestream. We use Timestream with MultiMetric writes to help with this but the main reason we went with it over OpenSearch was that we have 2 people on our team and managing cross regional OpenSearch wasn't something I wanted to do. We've been pretty happy with it's scalability so far but I'm concerned over it only being a single region product.

profile picture
smithatlanta
answered a year ago
  • ArielB
    a year ago

    The problem is that i'm going to show dashboards to my customers.. it's not used internally, so mass dashboards means mass queries

0

Opensearch works well with lower data volumes and simpler queries. at higher data volumes, it’s not as well-suited for processing analytical queries compared to Timestream and it becomes difficult to formulate certain queries in the Opensearch query DSL

One big advantage is that timestream is completely serverless, for Opensearch, although you don’t have servers to manage directly, we do have to monitor the node health (which we can see in CW). they can experience issues and the cluster will be in an unhealthy state until the Opensearch support team is paged (they will not resolve it automatically). Timestream seems to be actually serverless so there’s no actual cluster for us to keep an eye on

If you have lots of real time aggregations and want to write them in SQL (and are expecting high cost), then Timestream will fit in your use case.

I found the following for you that lists other benefits of Timestream:

AWS
SUPPORT ENGINEER
suveshan_p
answered a year ago
  • ArielB
    a year ago

    Thank you for your detailed answer. For my use case, i don't expect high volumes of data, rather I expect large number of queries.

    How does "If you have lots of real time aggregations and want to write them in SQL (and are expecting high cost), then Timestream will fit in your use case." makes sense?

    Let's say i expose a dashboard to my customers, while each customer has a table (for tenant isolation purposes). Now, If i'm having 30 widgest, i'm going to pay 30*10MB worth of scan, where let's say the entire 30 queries only scanned few MB.

    won't i expect an outstanding read costs? how does this makes sense with grafana integration where every panel is a seperate query?

    how to battle this minimum 10MB query limitation?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content