Cost optimization tips for AWS IoT Workloads

IoT applications must balance the size and frequency of messages that are sent by end devices with the most efficient way that data should be stored, processed, and utilized by your IoT application. In this article, we will discuss tips to optimize the costs of your IoT workload.

1. Cost optimization process

Optimizing costs for your IoT workload should be considered an ongoing process and not a one-time event. The process entails identifying operational goals and metrics, measuring those items continually, and making small, reversible improvements over time. It is recommended that IoT deployments initially optimize data transfer based on the device constraints. Once a baseline is established it becomes possible to measure the impact of improvements.

To better enable measurement, ensure that your workload can group and organize resources by both technical and business entities. This may be achieved by leveraging resource tags and billing groups. Additionally, be sure to add tracking elements to messages and devices. Tracking elements could include product model, serial number, and a unique trace ID for each message.

The AWS Cost Management console provides the capabilities to visualize, understand, and manage your AWS costs and usage over time. With a tagging strategy in place, you are able to measure specific workloads or device types and understand exactly what is incurring costs. With AWS Budgets you are able to set notifications based on cost thresholds and be alerted when they are reached. If more granularity is desired, AWS Cost and Usage Reports (AWS CUR) contains the most comprehensive set of cost and usage data available. You can use Cost and Usage Reports to publish your AWS billing reports to an Amazon Simple Storage Service (Amazon S3) bucket that you own, download them and view in Excel, or access with other AWS services such as Amazon Athena and Amazon Quicksight.

2. Optimize message size and frequency

It is critical to optimize payload size from the device to the cloud. The two primary strategies for doing so are to aggregate and filter data before sending. Determine if it is necessary to send all data to the cloud or if you are able to summarize the data, remove duplicate values, truncate, or ignore certain categories in order to transmit less data overall.

When sending periodic messages to the AWS cloud is acceptable, your device can combine multiple smaller messages into batches for greater transmission efficiency. You will need to balance batch size against the delay of transmission and the risk of losing a message containing multiple batched events.

Tips for filtering and aggregating telemetry:

• Filter at the edge – Use local processing to reduce the quantity of messages, and the corresponding payload size. For example, consider whether you can safely ignore certain events entirely. In such cases, your device can filter out those events to avoid incurring the costs of transmitting and processing them. AWS IoT Greengrass is an edge-runtime that natively supports these operations.
• Aggregate at the edge – In situations where telemetry data must be sent, be sure to only include what is necessary as not all use cases require all the raw data. For example, you can leverage local processing to aggregate or average data values, and just send the summarized results back to the cloud instead.
• Prune message content – If you are able to reduce the number of decimal places and precision of values, or abbreviate certain text values to make them shorter, you can achieve a smaller message size.
• Send when needed – If you are publishing more messages than necessary, try only sending the data when there is an active customer or use case.
• Use compression – Use serialization frameworks to compress the message payload. This is best suited for use cases where IoT Rules do not need to directly inspect the payload of the message, as it will be unreadable in its encoded state.

3. Consider Basic Ingest

With Basic Ingest, you can securely send device data to the AWS services supported by AWS IoT rule actions, without incurring messaging costs. Basic Ingest optimizes data flow by removing the publish and subscribe message broker from the ingestion path, making it more cost efficient.

Using IoT Core as a message publish and subscribe message broker may still be the best option for large volume telemetry if there is more than one subscriber or immediate action required based on payload data.

4. Architect for cost optimization

When creating the architecture of your IoT workload it is important to understand the nature of the telemetry and data that is to be ingested. Use the AWS Pricing Calculator to model costs based on message size and frequency to determine which services are best for your ingestion needs.

In high-volume telemetry and video scenarios, it may be more cost effective to send your incoming device data directly to a buffering or streaming service like Amazon Kinesis Data Streams, or Amazon Kinesis Video Streams (KVS), while separately maintaining fleet command and control via AWS IoT Core.

For logs and binary files, it is often more efficient to put files directly to Amazon S3 (S3). With AWS IoT Core Credential Provider your devices are able to obtain short-lived tokens so that they can authorize direct calls to services such as S3 and KVS.

5, 6. Filter with AWS IoT Rules

AWS IoT Rules give your devices the ability to interact with AWS services. Rules are analyzed and actions are performed based on the MQTT topic stream.

Optimize the mapping of topics to rules and strive to have one topic to one rule. A high number of AWS IoT Rules billing actions could indicate inefficiencies in the rules and may require further analysis for optimization.

When your workload requires many rules and actions then also consider using AWS Lambda. In order to determine if this strategy will provide cost savings it is best to model it as every workload varies at what point it will see impactful savings.

7. Shadow and Registry operations

In AWS IoT, device shadow and registry operations are metered in 1 KB increments and billing is per million operations. The shadow stores the desired or actual state of each device and the registry is used to name and manage devices.

Cost optimization processes for device shadows and registry focus on managing how many operations are performed and the size of each operation. If your operation is cost sensitive to shadow and registry operations, you should look for ways to optimize.

Optimization strategies and tips for both the Registry and Shadow services:

• Use named shadows – Separate logical elements and reduce the size of updates.
• Aggregate shadow updates – Look for opportunities to put several reported fields together into one shadow message update instead of sending each reported change independently. Grouping shadow updates together reduces the overall cost of the shadow by consolidating updates to the service.
• Send only what is needed, when it is needed – For example, your device should only send its full device state if your firmware logic dictates that the full device state may be out of sync and would be best reconciled by sending all current settings. As individual state changes occur, the device should optimize the frequency it transmits those changes to the cloud.
• Immutable data – Use the AWS IoT device registry device attributes for immutable data such as a serial numbers.
• Minimize the frequency of reads and writes - Where possible, limit updates to device’s shadow document to reduce the total metered operations.
• Choose the right service – Avoid using shadow sas a guaranteed-delivery mechanism or for continuously fluctuating data. Consider MQTT Last Will and Testament (LWT) as a mitigation for the risk of loss of device communication instead of using shadows.
• Use the AWS Pricing calculator – Compare device shadow operations versus telemetry messages to better understand cost implications. Some services are better suited for constant queries and high volume message patterns.

8. Optimize for downstream services

Batch incoming data using IoT Core rules before sending downstream to other AWS Services. Once ingested, putting telemetry data into batches allows those batches to be processed or stored in bulk which can be more efficient. For example, you can batch telemetry message data with Amazon Kinesis Data Firehose before writing to Amazon S3 in bulk. Writing fewer, larger files of device telemetry data to S3 is more efficient than writing numerous smaller files.

For additional information on cost optimization for your IoT workload, please see the AWS IoT Lens Well-Architected cost optimization checklist.