I am relatively new to IoT and AWS, and I have a quick question regarding a project I am working on. I am using this documentation as a guideline: Build a Digital Twin with AWS IoT TwinMaker.
I am currently reading data from a DHT11 sensor connected to an Arduino board. I have data coming into Amazon timestream in the following format:
For the next step, I am writing an Amazon Lambda function to read this data from Amazon Timestream and assign it to a specific entity in AWS IoT TwinMaker. This is what my code currently looks like, which was acquired directly from the documentation mentioned above:
import logging
import json
import os
import boto3
from datetime import datetime
LOGGER = logging.getLogger()
LOGGER.setLevel(logging.INFO)
# Get db and table name from Env variables
DATABASE_NAME = os.environ['TIMESTREAM_DATABASE_NAME']
TABLE_NAME = os.environ['TIMESTREAM_TABLE_NAME']
# Python boto client for AWS Timestream
QUERY_CLIENT = boto3.client('timestream-query')
# Utility function: parses a timestream row into a python dict for more convenient field access
def parse_row(column_schema, timestream_row):
data = timestream_row['Data']
result = {}
for i in range(len(data)):
info = column_schema[i]
datum = data[i]
key, val = parse_datum(info, datum)
result[key] = val
return result
# Utility function: parses timestream datum entries into (key,value) tuples. Only ScalarTypes currently supported.
def parse_datum(info, datum):
if datum.get('NullValue', False):
return info['Name'], None
column_type = info['Type']
if 'ScalarType' in column_type:
return info['Name'], datum['ScalarValue']
else:
raise Exception(f"Unsupported columnType[{column_type}]")
# This function extracts the timestamp from a Timestream row and returns in ISO8601 basic format
def get_iso8601_timestamp(str):
# e.g. '2022-04-06 00:17:45.419000000' -> '2022-04-06T00:17:45.419000000Z'
return str.replace(' ', 'T') + 'Z'
# Main logic
def lambda_handler(event, context):
selected_property = event['selectedProperties'][0]
LOGGER.info("Selected property is %s", selected_property)
# 1. EXECUTE THE QUERY TO RETURN VALUES FROM DATABASE
query_string = f"SELECT measure_name, time, measure_value::double" \
f" FROM {DATABASE_NAME}.{TABLE_NAME} " \
f" WHERE time > from_iso8601_timestamp('{event['startTime']}')" \
f" AND time <= from_iso8601_timestamp('{event['endTime']}')" \
f" AND measure_name = '{selected_property}'" \
f" ORDER BY time ASC"
try:
query_page = QUERY_CLIENT.query(
QueryString = query_string
)
except Exception as err:
LOGGER.error("Exception while running query: %s", err)
raise err
# Query result structure: https://docs.aws.amazon.com/timestream/latest/developerguide/API_query_Query.html
next_token = None
if query_page.get('NextToken') is not None:
next_token = query_page['NextToken']
schema = query_page['ColumnInfo']
# 2. PARSE TIMESTREAM ROWS
result_rows = []
for row in query_page['Rows']:
row_parsed = parse_row(schema,row)
#LOGGER.info('row parsed: %s', row_parsed)
result_rows.append(row_parsed)
# 3. CONVERT THE QUERY RESULTS TO THE FORMAT TWINMAKER EXPECTS
# There must be one entityPropertyReference for Humidity OR one for Temperature
entity_property_reference_temp = {}
entity_property_reference_temp['componentName'] = 'newest_comp'
entity_property_reference_temp['propertyName'] = 'temperature'
entity_property_reference_temp['entityId'] = '2be0c298-decc-4fcb-b652-dd3e4ee11652'
entity_property_reference_hum = {}
entity_property_reference_hum['componentName'] = 'newest_comp'
entity_property_reference_hum['propertyName'] = 'humidity'
entity_property_reference_hum['entityId'] = '2be0c298-decc-4fcb-b652-dd3e4ee11652'
values_temp = []
values_hum = []
for result_row in result_rows:
ts = result_row['time']
measure_name = result_row['measure_name']
measure_value = result_row['measure_value::bigint']
time = get_iso8601_timestamp(ts)
value = { 'doubleValue' : str(measure_value) }
if measure_name == 'temperature':
values_temp.append({
'time': time,
'value': value
})
elif measure_name == 'humidity':
values_hum.append({
'time': time,
'value': value
})
# The final structure "propertyValues"
property_values = []
if(measure_name == 'temperature'):
property_values.append({
'entityPropertyReference': entity_property_reference_temp,
'values': values_temp
})
elif(measure_name == 'humidity'):
property_values.append({
'entityPropertyReference': entity_property_reference_hum,
'values': values_hum
})
LOGGER.info("property_values: %s", property_values)
# marshall propertyValues and nextToken into final response
return_obj = {
'propertyValues': property_values,
'nextToken': next_token
}
return return_obj
When I run this Lambda function, I am expecting data to be assigned to my entity in AWS IoT TwinMaker, which in turn should be plotting data in my Grafana dashboard. I have set up the rules and policies exactly as described in the tutorial. However, my Grafana dashboard returns the following error when I attempt to plot the data:
ConnectorFailureException: Lambda function arn:aws:lambda:us-east-1:002403874886:function:timestreamReader-test throws exception. Please check your lambda logs for detailed information.
When I check my lambda logs, this is the most recent log event message:
INIT_START Runtime Version: python:3.10.v9 Runtime Version ARN: arn:aws:lambda:us-east-1::runtime:e0fc6c1275348290e8d4bb434b30f57fffb3611531064e480993b0bcef3e62b9
[INFO] 2023-07-26T18:59:11.095Z Found credentials in environment variables.
START RequestId: 18b316b6-a9ae-41e6-916b-11f10255d1d8 Version: $LATEST
[INFO] 2023-07-26T18:59:11.222Z 18b316b6-a9ae-41e6-916b-11f10255d1d8 Selected property is temperature
[ERROR] 2023-07-26T18:59:11.690Z 18b316b6-a9ae-41e6-916b-11f10255d1d8 Exception while running query: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
[ERROR] ValidationException: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 100, in lambda_handler
raise err
File "/var/task/lambda_function.py", line 95, in lambda_handler
query_page = QUERY_CLIENT.query(
File "/var/runtime/botocore/client.py", line 530, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 960, in _make_api_call
raise error_class(parsed_response, operation_name)
END RequestId: 18b316b6-a9ae-41e6-916b-11f10255d1d8
REPORT RequestId: 18b316b6-a9ae-41e6-916b-11f10255d1d8 Duration: 510.76 ms Billed Duration: 511 ms Memory Size: 128 MB Max Memory Used: 68 MB Init Duration: 414.25 ms
START RequestId: 5a7256aa-4f77-49bd-8c6f-21a3303028bf Version: $LATEST
[INFO] 2023-07-26T19:00:43.923Z 5a7256aa-4f77-49bd-8c6f-21a3303028bf Selected property is temperature
[ERROR] 2023-07-26T19:00:44.071Z 5a7256aa-4f77-49bd-8c6f-21a3303028bf Exception while running query: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
[ERROR] ValidationException: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 100, in lambda_handler
raise err
File "/var/task/lambda_function.py", line 95, in lambda_handler
query_page = QUERY_CLIENT.query(
File "/var/runtime/botocore/client.py", line 530, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 960, in _make_api_call
raise error_class(parsed_response, operation_name)
END RequestId: 5a7256aa-4f77-49bd-8c6f-21a3303028bf
REPORT RequestId: 5a7256aa-4f77-49bd-8c6f-21a3303028bf Duration: 149.68 ms Billed Duration: 150 ms Memory Size: 128 MB Max Memory Used: 69 MB
START RequestId: fc46934c-bda1-424e-bd3e-8babcb46f76e Version: $LATEST
[INFO] 2023-07-26T19:01:10.257Z fc46934c-bda1-424e-bd3e-8babcb46f76e Selected property is temperature
[ERROR] 2023-07-26T19:01:10.410Z fc46934c-bda1-424e-bd3e-8babcb46f76e Exception while running query: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
[ERROR] ValidationException: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 100, in lambda_handler
raise err
File "/var/task/lambda_function.py", line 95, in lambda_handler
query_page = QUERY_CLIENT.query(
File "/var/runtime/botocore/client.py", line 530, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 960, in _make_api_call
raise error_class(parsed_response, operation_name)
END RequestId: fc46934c-bda1-424e-bd3e-8babcb46f76e
REPORT RequestId: fc46934c-bda1-424e-bd3e-8babcb46f76e Duration: 154.94 ms Billed Duration: 155 ms Memory Size: 128 MB Max Memory Used: 69 MB
START RequestId: 092f2e09-02a8-4a53-9e52-14f075bb1a26 Version: $LATEST
[INFO] 2023-07-26T19:02:19.209Z 092f2e09-02a8-4a53-9e52-14f075bb1a26 Selected property is temperature
[ERROR] 2023-07-26T19:02:19.372Z 092f2e09-02a8-4a53-9e52-14f075bb1a26 Exception while running query: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
[ERROR] ValidationException: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 100, in lambda_handler
raise err
File "/var/task/lambda_function.py", line 95, in lambda_handler
query_page = QUERY_CLIENT.query(
File "/var/runtime/botocore/client.py", line 530, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 960, in _make_api_call
raise error_class(parsed_response, operation_name)
END RequestId: 092f2e09-02a8-4a53-9e52-14f075bb1a26
REPORT RequestId: 092f2e09-02a8-4a53-9e52-14f075bb1a26 Duration: 182.68 ms Billed Duration: 183 ms Memory Size: 128 MB Max Memory Used: 69 MB
START RequestId: 0ffb24ef-b139-4042-a509-ca4b4cebd9a7 Version: $LATEST
[INFO] 2023-07-26T19:06:16.023Z 0ffb24ef-b139-4042-a509-ca4b4cebd9a7 Selected property is temperature
[ERROR] 2023-07-26T19:06:16.168Z 0ffb24ef-b139-4042-a509-ca4b4cebd9a7 Exception while running query: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
[ERROR] ValidationException: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 100, in lambda_handler
raise err
File "/var/task/lambda_function.py", line 95, in lambda_handler
query_page = QUERY_CLIENT.query(
File "/var/runtime/botocore/client.py", line 530, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 960, in _make_api_call
raise error_class(parsed_response, operation_name)
END RequestId: 0ffb24ef-b139-4042-a509-ca4b4cebd9a7
REPORT RequestId: 0ffb24ef-b139-4042-a509-ca4b4cebd9a7 Duration: 150.92 ms Billed Duration: 151 ms Memory Size: 128 MB Max Memory Used: 69 MB
START RequestId: b38cb572-3546-46ce-a859-31d844aff6fb Version: $LATEST
[INFO] 2023-07-26T19:07:30.725Z b38cb572-3546-46ce-a859-31d844aff6fb Selected property is temperature
[ERROR] 2023-07-26T19:07:30.856Z b38cb572-3546-46ce-a859-31d844aff6fb Exception while running query: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
[ERROR] ValidationException: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 100, in lambda_handler
raise err
File "/var/task/lambda_function.py", line 95, in lambda_handler
query_page = QUERY_CLIENT.query(
File "/var/runtime/botocore/client.py", line 530, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 960, in _make_api_call
raise error_class(parsed_response, operation_name)
END RequestId: b38cb572-3546-46ce-a859-31d844aff6fb
REPORT RequestId: b38cb572-3546-46ce-a859-31d844aff6fb Duration: 150.50 ms Billed Duration: 151 ms Memory Size: 128 MB Max Memory Used: 69 MB
Hi! I have correctly configured TIMESTREAM_DATABASE_NAME and TIMESTREAM_TABLE_NAME in my environmental variables.