Problem with Lambda function reading data from Amazon Timestream

0

I am relatively new to IoT and AWS, and I have a quick question regarding a project I am working on. I am using this documentation as a guideline: Build a Digital Twin with AWS IoT TwinMaker.

I am currently reading data from a DHT11 sensor connected to an Arduino board. I have data coming into Amazon timestream in the following format:

Amazon Timestream output

For the next step, I am writing an Amazon Lambda function to read this data from Amazon Timestream and assign it to a specific entity in AWS IoT TwinMaker. This is what my code currently looks like, which was acquired directly from the documentation mentioned above:

import logging
import json
import os
import boto3

from datetime import datetime

LOGGER = logging.getLogger()
LOGGER.setLevel(logging.INFO)

# Get db and table name from Env variables
DATABASE_NAME = os.environ['TIMESTREAM_DATABASE_NAME']
TABLE_NAME = os.environ['TIMESTREAM_TABLE_NAME']

# Python boto client for AWS Timestream
QUERY_CLIENT = boto3.client('timestream-query')

# Utility function: parses a timestream row into a python dict for more convenient field access
def parse_row(column_schema, timestream_row):
    data = timestream_row['Data']
    result = {}
    for i in range(len(data)):
        info = column_schema[i]
        datum = data[i]
        key, val = parse_datum(info, datum)
        result[key] = val
    return result

# Utility function: parses timestream datum entries into (key,value) tuples. Only ScalarTypes currently supported.
def parse_datum(info, datum):
    if datum.get('NullValue', False):
        return info['Name'], None
    column_type = info['Type']
    if 'ScalarType' in column_type:
        return info['Name'], datum['ScalarValue']
    else:
        raise Exception(f"Unsupported columnType[{column_type}]")

# This function extracts the timestamp from a Timestream row and returns in ISO8601 basic format
def get_iso8601_timestamp(str):
    #  e.g. '2022-04-06 00:17:45.419000000' -> '2022-04-06T00:17:45.419000000Z'
    return str.replace(' ', 'T') + 'Z'

# Main logic
def lambda_handler(event, context):
    selected_property = event['selectedProperties'][0]

    LOGGER.info("Selected property is %s", selected_property)

    # 1. EXECUTE THE QUERY TO RETURN VALUES FROM DATABASE
    query_string = f"SELECT measure_name, time, measure_value::double" \
        f" FROM {DATABASE_NAME}.{TABLE_NAME} " \
        f" WHERE time > from_iso8601_timestamp('{event['startTime']}')" \
        f" AND time <= from_iso8601_timestamp('{event['endTime']}')" \
        f" AND measure_name = '{selected_property}'" \
        f" ORDER BY time ASC"
            
    try:
        query_page = QUERY_CLIENT.query(
            QueryString = query_string
        )
    except Exception as err:
        LOGGER.error("Exception while running query: %s", err)
        raise err

    # Query result structure: https://docs.aws.amazon.com/timestream/latest/developerguide/API_query_Query.html

    next_token = None
    if query_page.get('NextToken') is not None:
       next_token = query_page['NextToken']
    schema = query_page['ColumnInfo']

    # 2. PARSE TIMESTREAM ROWS
    result_rows = []
    for row in query_page['Rows']:
        row_parsed = parse_row(schema,row)
        #LOGGER.info('row parsed: %s', row_parsed)
        result_rows.append(row_parsed)

    # 3. CONVERT THE QUERY RESULTS TO THE FORMAT TWINMAKER EXPECTS

    # There must be one entityPropertyReference for Humidity OR one for Temperature
    entity_property_reference_temp = {}
    entity_property_reference_temp['componentName'] = 'newest_comp'
    entity_property_reference_temp['propertyName'] = 'temperature'
    entity_property_reference_temp['entityId'] = '2be0c298-decc-4fcb-b652-dd3e4ee11652'


    entity_property_reference_hum = {}
    entity_property_reference_hum['componentName'] = 'newest_comp'
    entity_property_reference_hum['propertyName'] = 'humidity'
    entity_property_reference_hum['entityId'] = '2be0c298-decc-4fcb-b652-dd3e4ee11652'


    values_temp = []
    values_hum = []

    for result_row in result_rows:
        ts = result_row['time']
        measure_name = result_row['measure_name']
        measure_value = result_row['measure_value::bigint']

        time = get_iso8601_timestamp(ts)
        value = { 'doubleValue' : str(measure_value) }

        if measure_name == 'temperature':
            values_temp.append({
                'time': time,
                'value':  value
            })
        elif measure_name == 'humidity':
             values_hum.append({
                'time': time,
                'value':  value
            })

    # The final structure "propertyValues"
    property_values = []

    if(measure_name == 'temperature'):
        property_values.append({
            'entityPropertyReference': entity_property_reference_temp,
            'values': values_temp
        })
    elif(measure_name == 'humidity'):
        property_values.append({
            'entityPropertyReference': entity_property_reference_hum,
            'values': values_hum
        })
    LOGGER.info("property_values: %s", property_values)

    # marshall propertyValues and nextToken into final response
    return_obj = {
       'propertyValues': property_values,
       'nextToken': next_token
       }

    return return_obj

When I run this Lambda function, I am expecting data to be assigned to my entity in AWS IoT TwinMaker, which in turn should be plotting data in my Grafana dashboard. I have set up the rules and policies exactly as described in the tutorial. However, my Grafana dashboard returns the following error when I attempt to plot the data:

ConnectorFailureException: Lambda function arn:aws:lambda:us-east-1:002403874886:function:timestreamReader-test throws exception. Please check your lambda logs for detailed information.

When I check my lambda logs, this is the most recent log event message:

INIT_START Runtime Version: python:3.10.v9	Runtime Version ARN: arn:aws:lambda:us-east-1::runtime:e0fc6c1275348290e8d4bb434b30f57fffb3611531064e480993b0bcef3e62b9
[INFO]	2023-07-26T18:59:11.095Z		Found credentials in environment variables.
START RequestId: 18b316b6-a9ae-41e6-916b-11f10255d1d8 Version: $LATEST
[INFO]	2023-07-26T18:59:11.222Z	18b316b6-a9ae-41e6-916b-11f10255d1d8	Selected property is temperature
[ERROR]	2023-07-26T18:59:11.690Z	18b316b6-a9ae-41e6-916b-11f10255d1d8	Exception while running query: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
[ERROR] ValidationException: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 100, in lambda_handler
    raise err
  File "/var/task/lambda_function.py", line 95, in lambda_handler
    query_page = QUERY_CLIENT.query(
  File "/var/runtime/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)
END RequestId: 18b316b6-a9ae-41e6-916b-11f10255d1d8
REPORT RequestId: 18b316b6-a9ae-41e6-916b-11f10255d1d8	Duration: 510.76 ms	Billed Duration: 511 ms	Memory Size: 128 MB	Max Memory Used: 68 MB	Init Duration: 414.25 ms	
START RequestId: 5a7256aa-4f77-49bd-8c6f-21a3303028bf Version: $LATEST
[INFO]	2023-07-26T19:00:43.923Z	5a7256aa-4f77-49bd-8c6f-21a3303028bf	Selected property is temperature
[ERROR]	2023-07-26T19:00:44.071Z	5a7256aa-4f77-49bd-8c6f-21a3303028bf	Exception while running query: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
[ERROR] ValidationException: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 100, in lambda_handler
    raise err
  File "/var/task/lambda_function.py", line 95, in lambda_handler
    query_page = QUERY_CLIENT.query(
  File "/var/runtime/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)
END RequestId: 5a7256aa-4f77-49bd-8c6f-21a3303028bf
REPORT RequestId: 5a7256aa-4f77-49bd-8c6f-21a3303028bf	Duration: 149.68 ms	Billed Duration: 150 ms	Memory Size: 128 MB	Max Memory Used: 69 MB	
START RequestId: fc46934c-bda1-424e-bd3e-8babcb46f76e Version: $LATEST
[INFO]	2023-07-26T19:01:10.257Z	fc46934c-bda1-424e-bd3e-8babcb46f76e	Selected property is temperature
[ERROR]	2023-07-26T19:01:10.410Z	fc46934c-bda1-424e-bd3e-8babcb46f76e	Exception while running query: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
[ERROR] ValidationException: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 100, in lambda_handler
    raise err
  File "/var/task/lambda_function.py", line 95, in lambda_handler
    query_page = QUERY_CLIENT.query(
  File "/var/runtime/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)
END RequestId: fc46934c-bda1-424e-bd3e-8babcb46f76e
REPORT RequestId: fc46934c-bda1-424e-bd3e-8babcb46f76e	Duration: 154.94 ms	Billed Duration: 155 ms	Memory Size: 128 MB	Max Memory Used: 69 MB	
START RequestId: 092f2e09-02a8-4a53-9e52-14f075bb1a26 Version: $LATEST
[INFO]	2023-07-26T19:02:19.209Z	092f2e09-02a8-4a53-9e52-14f075bb1a26	Selected property is temperature
[ERROR]	2023-07-26T19:02:19.372Z	092f2e09-02a8-4a53-9e52-14f075bb1a26	Exception while running query: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
[ERROR] ValidationException: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 100, in lambda_handler
    raise err
  File "/var/task/lambda_function.py", line 95, in lambda_handler
    query_page = QUERY_CLIENT.query(
  File "/var/runtime/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)
END RequestId: 092f2e09-02a8-4a53-9e52-14f075bb1a26
REPORT RequestId: 092f2e09-02a8-4a53-9e52-14f075bb1a26	Duration: 182.68 ms	Billed Duration: 183 ms	Memory Size: 128 MB	Max Memory Used: 69 MB	
START RequestId: 0ffb24ef-b139-4042-a509-ca4b4cebd9a7 Version: $LATEST
[INFO]	2023-07-26T19:06:16.023Z	0ffb24ef-b139-4042-a509-ca4b4cebd9a7	Selected property is temperature
[ERROR]	2023-07-26T19:06:16.168Z	0ffb24ef-b139-4042-a509-ca4b4cebd9a7	Exception while running query: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
[ERROR] ValidationException: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 100, in lambda_handler
    raise err
  File "/var/task/lambda_function.py", line 95, in lambda_handler
    query_page = QUERY_CLIENT.query(
  File "/var/runtime/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)
END RequestId: 0ffb24ef-b139-4042-a509-ca4b4cebd9a7
REPORT RequestId: 0ffb24ef-b139-4042-a509-ca4b4cebd9a7	Duration: 150.92 ms	Billed Duration: 151 ms	Memory Size: 128 MB	Max Memory Used: 69 MB	
START RequestId: b38cb572-3546-46ce-a859-31d844aff6fb Version: $LATEST
[INFO]	2023-07-26T19:07:30.725Z	b38cb572-3546-46ce-a859-31d844aff6fb	Selected property is temperature
[ERROR]	2023-07-26T19:07:30.856Z	b38cb572-3546-46ce-a859-31d844aff6fb	Exception while running query: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
[ERROR] ValidationException: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 100, in lambda_handler
    raise err
  File "/var/task/lambda_function.py", line 95, in lambda_handler
    query_page = QUERY_CLIENT.query(
  File "/var/runtime/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)
END RequestId: b38cb572-3546-46ce-a859-31d844aff6fb
REPORT RequestId: b38cb572-3546-46ce-a859-31d844aff6fb	Duration: 150.50 ms	Billed Duration: 151 ms	Memory Size: 128 MB	Max Memory Used: 69 MB	
2 Answers
0
Accepted Answer

Hi. There seems to be a problem with your query_string. I note that your database name is TempHumidityDatabase-test and your table name is TempHumidityTable. However the blog post uses TempHumidityDatabase and TempHumidity. Have you correctly set TIMESTREAM_DATABASE_NAME and TIMESTREAM_TABLE_NAME?

profile pictureAWS
EXPERT
Greg_B
answered 9 months ago
  • Hi! I have correctly configured TIMESTREAM_DATABASE_NAME and TIMESTREAM_TABLE_NAME in my environmental variables.

0

If I consider your generated query to be roughly same as

SELECT measure_name, time, measure_value::double FROM {DATABASE_NAME}.{TABLE_NAME}  WHERE time > from_iso8601_timestamp('{event['startTime']}') AND time <= from_iso8601_timestamp('{event['endTime']}') AND measure_name = '{selected_property}' ORDER BY time ASC

The query syntax is invalid at line 1:75 comes out to be somewhere around database / table name . Are you properly enclosing database / table name with double quotes as you have "-" in the name?

The code needs correction at FROM {DATABASE_NAME}.{TABLE_NAME} to FROM "{DATABASE_NAME}"."{TABLE_NAME}"

AWS
answered 9 months ago
  • Thank you for your feedback. My function is still returning a syntax error in the same place (line 1:75) with this modification. Would you suggest changing the database/table name to remove the "-"?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions