How do I configure logging levels dynamically for specific resources in AWS IoT Core?

Short description

Note: This article relates to only V2 of AWS IoT Core logs.

You can use AWS IoT Core logs to dynamically set logging levels for specific resources. You can use a dynamic thing group to set a resource-specific logging level. You must register your clients as things and they must connect with the same client ID associated with the thing name. The dynamic thing group uses a fleet indexing query that continuously finds and adds things automatically to the group that meet the query conditions.

It's a best practice to configure default logging to a lower verbosity level and configure resource-specific logging to a higher verbosity level. Log verbosity levels include DISABLED (lowest), ERROR, WARN, INFO, and DEBUG (highest).

When creating or updating a dynamic thing group, things can be eligible to be in a dynamic thing group and not be added to it automatically. For more information, see Successful commands can log errors.

Important: Depending on your AWS IoT Core fleet size, turning on more verbose log levels can incur high costs and make troubleshooting more difficult. INFO or DEBUG should only be used as a temporary measure while troubleshooting. After troubleshooting has completed, logging levels should be set back to a less verbose setting.

Resolution

Prerequisite

Make sure that you have AWS Command Line Interface (AWS CLI) installed locally with IoT admin permission credentials. The default AWS Region for AWS CLI should point towards the targeted AWS Region. You must have clients connected to and interacting with your AWS IoT Core endpoints as registered IoT things.

Note: If you receive errors when running AWS CLI commands, make sure that you're using the most recent version of the AWS CLI.

Configuring automatic logging for specific resources

1.    Create an AWS IoT thing and add a thing attribute to it.

Note: You can add more thing attributes as needed. You can then change the dynamic thing group query to search for these.

2.    Turn on thing indexing.

3.    Run the DescribeIndex command to verify that your thing index has an ACTIVE indexStatus attribute:

aws iot describe-index --index-name "AWS_Things"

4.    Create a dynamic thing group using either the AWS IoT console or the AWS CLI.

AWS IoT console

1.    Turn on fleet indexing if you haven't already done so.

2.    Sign in to the AWS IoT console.

3.    In the navigation pane, choose All devices, and then choose Thing groups.

4.    Choose Create thing group.

5.    For Thing group type, choose Create dynamic thing group, and then choose Next.

6.    For Thing group name, enter logging_level_debug_dynamic. Then, choose Create thing group.
Note: Replace logging_level_debug_dynamic with your thing group name.

7.    Under Query, enter attributes.firmware:1, and then press Enter to apply the search query statement.
Note: Replace firmware:1 with the thing attribute and value that you want to use for the query.

8.    Choose Preview query to see all things that match the query criteria.

9.    Choose Create thing group.

AWS CLI

1.    Run the CreateDynamicThingGroup command to create a dynamic thing group:

Note: Dynamic thing group creation is not instantaneous and can take some time to complete. For more information, see Create a dynamic thing group.

aws iot create-dynamic-thing-group \
 --thing-group-name "logging_level_debug_dynamic" \
 --query-string "attributes.firmware:1"

Note: In the preceding example, replace firmware:1 with the attribute and value that you want to use for the query. Replace logging_level_debug_dynamic with your thing group name.

The output looks similar to the following message:

{
 "thingGroupName": "logging_level_debug_dynamic",
 "thingGroupArn": "arn:aws:iot:eu-west-1:123456789012:thinggroup/logging_level_debug_dynamic",
 "thingGroupId": "b8f96cc3-f36b-4da2-b0f8-e28f5a123456",
 "indexName": "AWS_Things",
 "queryString": "attributes.firmware:1",
 "queryVersion": "2017-09-30"
}

2.    Run the DescribeThingGroup command to verify that your dynamic thing group is ACTIVE:

aws iot describe-thing-group --thing-group-name "logging_level_debug_dynamic"

Note: In the preceding example, replace logging_level_debug_dynamic with your thing group name.

The output looks similar to the following message:

{
 "thingGroupName": "logging_level_debug_dynamic",
 "thingGroupId": "b8f96cc3-f36b-4da2-b0f8-e28f5a123456",
 "thingGroupArn": "arn:aws:iot:eu-west-1:123456789012:thinggroup/logging_level_debug_dynamic",
 "version": 1,
 "thingGroupProperties": {},
 "thingGroupMetadata": {
 "creationDate": "2022-11-30T12:37:19.980000+00:00"
 },
 "indexName": "AWS_Things",
 "queryString": "attributes.firmware:1",
 "queryVersion": "2017-09-30",
 "status": "ACTIVE"
}

3.    Run the SetV2LoggingLevel command to set the logging level for the resource. Setting the logging level can take up to ten minutes to complete.

aws iot set-v2-logging-level \
 --log-target targetType=THING_GROUP,targetName=logging_level_debug_dynamic \
 --log-level DEBUG

Note: In the preceding example, replace logging_level_debug_dynamic with your dynamic thing group name.

The output looks similar to the following message. The default logging is included in the output and includes the details based on your configurations. Other logging levels you have configured are also displayed.

{
"logTargetConfigurations": [{
"logTarget": {
"targetType": "DEFAULT"
},
"logLevel": "WARN"
},
{
"logTarget": {
"targetType": "THING_GROUP",
"targetName": "
logging_level_debug_dynamic "
},
"logLevel": "DEBUG"
}
]
}

Monitor generated logs

It's a best practice to monitor your IoT logs for issues or problems. You can use either the AWS CloudWatch Logs Console or the AWS CLI to monitor your AWS IoT Core logs. For more information, see the "Monitoring log entries" section of How do I best manage the logging levels of my AWS IoT logs in AWS IoT Core?

Related information

AWS IoT Device Management Pricing

How do I configure the default logging settings for AWS IoT Core?

How do I configure logging levels manually for specific resources in AWS IoT Core?