How to filter messages with a policy based on Thing Attributes in IoT Core

To filter MQTT traffic based on Thing Attributes in AWS IoT Core, you need to ensure your attributes are properly set up and your policy is correctly configured. Here's how to make it work:

First, for your use case of filtering messages based on location, you need to add the location attribute directly to the thing itself, not to the thing group or thing type. Thing policy variables like iot:Connection.Thing.Attributes[location] specifically reference attributes of the thing, not attributes of the thing group the thing belongs to.

Here's the correct approach:

1. Add the attribute "location" with value "Seattle" directly to your thing as a non-searchable attribute in the AWS IoT registry.

2. Make sure your device is connecting with a client ID that matches the thing name. This is crucial because the policy variables only work when the client ID of the MQTT connection matches the thing name in the registry.

3. Ensure the certificate used by your device is attached to the thing in the AWS IoT registry using the AttachThingPrincipal API.

4. Update your policy to match your region (you're using eu-west-2 in your connection string but us-east-1 in your policy) and account number.

Your policy looks correct in structure, but make sure:

• The region in the ARN matches your actual region (eu-west-2)
• The account number is correct
• The client ID in your mosquitto_sub command should match the thing name in the registry

The disconnection logs you're seeing with "CLIENT_ERROR" suggest that the policy evaluation is failing. This could be because:

• The client ID "iot" doesn't match a thing name in your registry
• The thing doesn't have the location attribute set
• The certificate isn't properly attached to the thing

For your mosquitto_sub command, try changing the client ID (-i parameter) to match exactly the name of your thing in the registry:

mosquitto_sub -h amplaceholder-ats.iot.eu-west-2.amazonaws.com -p 8883 --cafile rootCA.pem --cert cert.crt --key private.key -t "sensor/Seattle" -q 1 -i "YourThingNameHere"

Remember that thing attributes are only available when connecting over MQTT or MQTT over WebSocket protocols, and the client ID must match the thing name for the policy variables to resolve correctly.
Sources
Publish/Subscribe policy examples - AWS IoT Core
Thing policy variables - AWS IoT Core
Thing policy examples - AWS IoT Core

This is a great, practical constraint and a common real-world requirement. You want strict broker-level enforcement so that devices and apps in Seattle can only see Seattle topics, and devices can move between cities by changing group membership. The cleanest, scalable solution is to put authorization into a fast custom authorizer at connect time that issues minimal allow policies scoped to the current group membership. Here's an outline of a production-ready reference architecture, concrete implementation notes, test plan, and operational best practices. I have included links to AWS docs you will need.

Recommended architecture summary

Use Thing Groups to model cities. Example: Group Seattle, Group Madrid. Assign the thing to exactly one city group at a time.

Use topic namespaces per group. Convention: sensor/<city>/... for publishes and sensor/<city>/+ for subscribers. This keeps routing and telemetry simple.

Enforce broker authorization with a custom authorizer. On MQTT connect the custom authorizer validates the identity, queries group membership, and returns a short allow policy limited to the group topic ARNs. The authorizer is the single source of truth for what topics that connection may publish or subscribe to.

Use a fast cache to avoid excessive registry API calls. The authorizer Lambda looks up group membership in a DynamoDB table with TTL. Updates to group membership write through to that table so authorizer decisions are near-zero latency.

Manage device movement by updating group membership in the registry and the DynamoDB cache. On movement, update Thing Group membership via the IoT API and write the new mapping to DynamoDB. The next connection will receive the new policy and therefore new scope.

Why this works

IoT Core policy variable evaluation cannot use Thing Group attributes. A custom authorizer gives you control to evaluate registry state and issue a tailored allow policy.

Issuing a minimal allow policy at connect time enforces zero cross talk at the broker level. If a client tries to subscribe to sensor/Madrid, but its allow policy only has sensor/Seattle, the broker rejects the subscribe request.

Caching is necessary for scale and latency. The authorizer must be quick to keep connection latency low.

Components and flow (step by step)

Device boots and connects to AWS IoT endpoint using MQTT with client id that ties to the Thing.

AWS IoT calls your custom authorizer (Lambda) with the token or credentials.

Authorizer does these checks: • authenticate the token/cert • map client id or certificate to ThingName (or use client id == ThingName) • read group membership from DynamoDB cache. If cache miss, call ListThingGroupsForThing or DescribeThing, then populate cache • build and return a policy document that allows Publish/Subscribe/Receive only on arn:aws:iot:<region>:<acct>:topic/sensor/<city>* and arn:aws:iot:<region>:<acct>:topicfilter/sensor/<city>* and a minimal Connect permission to arn:aws:iot:<region>:<acct>:client/${iot:Connection.Thing.ThingName}

IoT broker enforces that allow policy. Any attempt to access another city fails at broker level.

Example policy fragment returned by authorizer

Return this JSON in the authorizer response (fill in region, account, city, thingname):

{ "principalId": "thing/SeattleSensor01", "policyDocuments": [ { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["iot:Connect"], "Resource": ["arn:aws:iot:us-east-1:123456789012:client/SeattleSensor01"] }, { "Effect": "Allow", "Action": ["iot:Publish"], "Resource": ["arn:aws:iot:us-east-1:123456789012:topic/sensor/Seattle/"] }, { "Effect": "Allow", "Action": ["iot:Subscribe"], "Resource": ["arn:aws:iot:us-east-1:123456789012:topicfilter/sensor/Seattle/"] }, { "Effect": "Allow", "Action": ["iot:Receive"], "Resource": ["arn:aws:iot:us-east-1:123456789012:topic/sensor/Seattle/*"] } ] } ] }

Implementation details and tips

Custom authorizers support MQTT and WebSocket connections. See docs here. https://docs.aws.amazon.com/iot/latest/developerguide/custom-authentication.html

Use the IoT API ListThingGroupsForThing only on cache misses. That API is straightforward: https://docs.aws.amazon.com/iot/latest/apireference/API_ListThingGroupsForThing.html

Keep DynamoDB item structure minimal: ThingName => GroupName, ExpiresAt. Update TTL when you change membership. Use conditional writes to avoid race conditions.

On group change, update the registry plus DynamoDB. You can implement that as a small admin Lambda triggered by an operator UI or by your provisioning pipeline.

Authorizer performance: enable provisioned concurrency for the Lambda if you expect rapid spikes and want consistent latency. Keep the Lambda runtime very small and avoid heavy SDK calls.

If devices can move while still connected, you must force reconnect or notify devices to reconnect so the authorizer can return a new policy. Broker policies are evaluated per connection, not continuously updated mid-connection.

For apps or operators, use Cognito or IAM based access and map their permissions to group topics in a similar way, so operators only subscribe to their city feed.

Testing and validation

Start with a static policy test. Hardcode “Seattle” in the authorizer and verify that subscribing to Madrid fails.

Test movement: change the ThingGroup membership in the registry, update DynamoDB, then force a reconnect and verify the broker allows new topics.

Load test the authorizer at expected connection rates. Validate cache hit ratio and cold start latency.

Security and operational controls

Authorizer must log decisions and reasoning to CloudWatch Logs for auditing. Record client id, thing name, returned group, and policy.

Use KMS to protect any tokens or credentials used by the authorizer.

Monitor and alert on cache miss rates and authorization latencies.

Alternatives and tradeoffs

Option: use X.509 certs with per-thing policies. That avoids authorizer complexity but forces you to attach a policy to each thing and you lose group dynamics without reattaching policies on every move. That is less operationally flexible at scale.

Option: custom authorizer is the most flexible and gives immediate group-driven control. It is the recommended approach when group-based runtime routing and strict broker enforcement are required.

Useful links

Custom authentication and authorizers in AWS IoT Core. https://docs.aws.amazon.com/iot/latest/developerguide/custom-authentication.html

ListThingGroupsForThing API. https://docs.aws.amazon.com/iot/latest/apireference/API_ListThingGroupsForThing.html

AWS IoT policy examples for pub/sub. https://docs.aws.amazon.com/iot/latest/developerguide/pub-sub-policy.html

Quick implementation checklist

Create Thing Groups per city.

Implement small admin flow to move Things between groups and write mapping to DynamoDB.

Build custom authorizer Lambda that: • authenticates the connection token or cert • reads mapping from DynamoDB or falls back to ListThingGroupsForThing • returns minimal allow policy scoped to city topics

Deploy authorizer and enable it in your IoT custom auth configuration.

Test connect/publish/subscribe behavior and force reconnect on group changes.

The next steps would be to produce a small reference repo that includes:

Authorizer Lambda code (Node or Python) with DynamoDB caching.

Example admin scripts for moving things between groups.

A test plan and sample mosquitto commands for verification.

Which language you prefer for the Lambda ?

That’s a great observation, and you’re absolutely right to question whether the built-in HTTP caching inside the Lambda authorizer could be enough on its own. In practice, that cache is ephemeral and tied to the container instance running your Lambda. Once AWS scales your authorizer out or the container is recycled, any in-memory cache is lost. For small workloads, it might appear sufficient, but it becomes inconsistent at scale or under bursty connection patterns.

For your use case, where devices move dynamically between city groups and authorization needs to reflect that change immediately, DynamoDB provides a reliable and cost-effective cache layer. It keeps the custom authorizer stateless, while still letting you make sub-millisecond group membership lookups. It also gives you auditability and control over expiry policies, which the transient Lambda cache can’t guarantee.

Here’s a proven pattern that scales well in production across large fleets:

1. Minimal DynamoDB Cache Schema

Partition key: thingName

Attributes: groupName, expiresAt, lastUpdated (optional for traceability)

TTL: Use the built-in DynamoDB Time to Live (TTL) feature so entries expire automatically after a defined window (for example, 5–10 minutes). This keeps the cache lightweight and self-cleaning.

Documentation: Using Time to Live (TTL) in Amazon DynamoDB : https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/TTL.html

2. Update and Refresh Strategy

Whenever a device moves from one city to another (for instance, from Seattle to Madrid):

Update the Thing Group membership using the IoT API (UpdateThingGroupsForThing or your provisioning pipeline).

Write the new mapping { thingName, groupName, expiresAt } into DynamoDB.

On the next connection, your custom authorizer reads from DynamoDB.

If the entry is valid, it builds a short-lived allow policy scoped to that city’s MQTT namespace.

If the entry is missing or expired, it calls ListThingGroupsForThing from the IoT Core registry to refresh the cache, then proceeds to issue the new scoped policy.

Documentation references: ListThingGroupsForThing API Reference : https://docs.aws.amazon.com/iot/latest/apireference/API_ListThingGroupsForThing.html

UpdateThingGroupsForThing API Reference : https://docs.aws.amazon.com/iot/latest/apireference/API_UpdateThingGroupsForThing.html

3. Performance and Reliability Considerations

Initialize your DynamoDB client outside the Lambda handler to reuse connections across invocations.

Enable Provisioned Concurrency for your Lambda authorizer to maintain consistent startup latency, especially during connection bursts.

Keep the authorizer Lambda small and efficient, focusing on caching and policy generation rather than registry lookups.

Documentation references: AWS Lambda Provisioned Concurrency : https://docs.aws.amazon.com/lambda/latest/dg/provisioned-concurrency.html

4. Validation and Testing Strategy

Start by hardcoding a group name (for example, “Seattle”) in your authorizer’s policy response to confirm the MQTT authorization flow works end-to-end.

Then introduce the DynamoDB lookup and verify that devices receive policies scoped to the correct city.

Finally, simulate a device relocation:

Move the Thing to another group (for example, Madrid).

Update the cache entry in DynamoDB.

Force a reconnect.

Confirm that the authorizer now issues a new policy scoped to the Madrid topic namespace.

5. Key Documentation References for Implementation

Here are the AWS resources you’ll want to keep handy while building this out:

Custom authentication and authorization in AWS IoT Core https://docs.aws.amazon.com/iot/latest/developerguide/custom-authentication.html

AWS IoT Core Policy Variables https://docs.aws.amazon.com/iot/latest/developerguide/iot-policy-variables.html

AWS IoT Core Policy Examples (Publish/Subscribe) https://docs.aws.amazon.com/iot/latest/developerguide/pub-sub-policy.html

AWS IoT Core API Reference (AttachThingPrincipal) https://docs.aws.amazon.com/iot/latest/apireference/API_AttachThingPrincipal.html

AWS IoT Core API Reference (ListThingGroupsForThing) https://docs.aws.amazon.com/iot/latest/apireference/API_ListThingGroupsForThing.html

AWS IoT Core API Reference (UpdateThingGroupsForThing) https://docs.aws.amazon.com/iot/latest/apireference/API_UpdateThingGroupsForThing.html

Using Time to Live (TTL) in Amazon DynamoDB https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/TTL.html

AWS Lambda Provisioned Concurrency https://docs.aws.amazon.com/lambda/latest/dg/provisioned-concurrency.html

6. Practical Implementation Tip

If you prefer to minimize DynamoDB dependencies, you could use a very short-lived in-memory cache inside the Lambda (for example, a Python dictionary or LRU cache). However, that will only help within a single Lambda container and will not persist across scale-out events. For production-grade consistency, the DynamoDB approach remains the best balance of simplicity, cost, and reliability.

The next steps would be to outline a working Python version of this custom authorizer that uses boto3, DynamoDB TTL, and a test harness for validating group-based MQTT topic filtering. It’s concise, fast, and easily extensible for production use.