Problem with Lambda function reading data from Amazon Timestream

0

I am relatively new to IoT and AWS, and I have a quick question regarding a project I am working on. I am using this documentation as a guideline: Build a Digital Twin with AWS IoT TwinMaker.

I am currently reading data from a DHT11 sensor connected to an Arduino board. I have data coming into Amazon timestream in the following format:

Amazon Timestream output

For the next step, I am writing an Amazon Lambda function to read this data from Amazon Timestream and assign it to a specific entity in AWS IoT TwinMaker. This is what my code currently looks like, which was acquired directly from the documentation mentioned above:

import logging
import json
import os
import boto3

from datetime import datetime

LOGGER = logging.getLogger()
LOGGER.setLevel(logging.INFO)

# Get db and table name from Env variables
DATABASE_NAME = os.environ['TIMESTREAM_DATABASE_NAME']
TABLE_NAME = os.environ['TIMESTREAM_TABLE_NAME']

# Python boto client for AWS Timestream
QUERY_CLIENT = boto3.client('timestream-query')

# Utility function: parses a timestream row into a python dict for more convenient field access
def parse_row(column_schema, timestream_row):
    data = timestream_row['Data']
    result = {}
    for i in range(len(data)):
        info = column_schema[i]
        datum = data[i]
        key, val = parse_datum(info, datum)
        result[key] = val
    return result

# Utility function: parses timestream datum entries into (key,value) tuples. Only ScalarTypes currently supported.
def parse_datum(info, datum):
    if datum.get('NullValue', False):
        return info['Name'], None
    column_type = info['Type']
    if 'ScalarType' in column_type:
        return info['Name'], datum['ScalarValue']
    else:
        raise Exception(f"Unsupported columnType[{column_type}]")

# This function extracts the timestamp from a Timestream row and returns in ISO8601 basic format
def get_iso8601_timestamp(str):
    #  e.g. '2022-04-06 00:17:45.419000000' -> '2022-04-06T00:17:45.419000000Z'
    return str.replace(' ', 'T') + 'Z'

# Main logic
def lambda_handler(event, context):
    selected_property = event['selectedProperties'][0]

    LOGGER.info("Selected property is %s", selected_property)

    # 1. EXECUTE THE QUERY TO RETURN VALUES FROM DATABASE
    query_string = f"SELECT measure_name, time, measure_value::double" \
        f" FROM {DATABASE_NAME}.{TABLE_NAME} " \
        f" WHERE time > from_iso8601_timestamp('{event['startTime']}')" \
        f" AND time <= from_iso8601_timestamp('{event['endTime']}')" \
        f" AND measure_name = '{selected_property}'" \
        f" ORDER BY time ASC"
            
    try:
        query_page = QUERY_CLIENT.query(
            QueryString = query_string
        )
    except Exception as err:
        LOGGER.error("Exception while running query: %s", err)
        raise err

    # Query result structure: https://docs.aws.amazon.com/timestream/latest/developerguide/API_query_Query.html

    next_token = None
    if query_page.get('NextToken') is not None:
       next_token = query_page['NextToken']
    schema = query_page['ColumnInfo']

    # 2. PARSE TIMESTREAM ROWS
    result_rows = []
    for row in query_page['Rows']:
        row_parsed = parse_row(schema,row)
        #LOGGER.info('row parsed: %s', row_parsed)
        result_rows.append(row_parsed)

    # 3. CONVERT THE QUERY RESULTS TO THE FORMAT TWINMAKER EXPECTS

    # There must be one entityPropertyReference for Humidity OR one for Temperature
    entity_property_reference_temp = {}
    entity_property_reference_temp['componentName'] = 'newest_comp'
    entity_property_reference_temp['propertyName'] = 'temperature'
    entity_property_reference_temp['entityId'] = '2be0c298-decc-4fcb-b652-dd3e4ee11652'


    entity_property_reference_hum = {}
    entity_property_reference_hum['componentName'] = 'newest_comp'
    entity_property_reference_hum['propertyName'] = 'humidity'
    entity_property_reference_hum['entityId'] = '2be0c298-decc-4fcb-b652-dd3e4ee11652'


    values_temp = []
    values_hum = []

    for result_row in result_rows:
        ts = result_row['time']
        measure_name = result_row['measure_name']
        measure_value = result_row['measure_value::bigint']

        time = get_iso8601_timestamp(ts)
        value = { 'doubleValue' : str(measure_value) }

        if measure_name == 'temperature':
            values_temp.append({
                'time': time,
                'value':  value
            })
        elif measure_name == 'humidity':
             values_hum.append({
                'time': time,
                'value':  value
            })

    # The final structure "propertyValues"
    property_values = []

    if(measure_name == 'temperature'):
        property_values.append({
            'entityPropertyReference': entity_property_reference_temp,
            'values': values_temp
        })
    elif(measure_name == 'humidity'):
        property_values.append({
            'entityPropertyReference': entity_property_reference_hum,
            'values': values_hum
        })
    LOGGER.info("property_values: %s", property_values)

    # marshall propertyValues and nextToken into final response
    return_obj = {
       'propertyValues': property_values,
       'nextToken': next_token
       }

    return return_obj

When I run this Lambda function, I am expecting data to be assigned to my entity in AWS IoT TwinMaker, which in turn should be plotting data in my Grafana dashboard. I have set up the rules and policies exactly as described in the tutorial. However, my Grafana dashboard returns the following error when I attempt to plot the data:

ConnectorFailureException: Lambda function arn:aws:lambda:us-east-1:002403874886:function:timestreamReader-test throws exception. Please check your lambda logs for detailed information.

When I check my lambda logs, this is the most recent log event message:

INIT_START Runtime Version: python:3.10.v9	Runtime Version ARN: arn:aws:lambda:us-east-1::runtime:e0fc6c1275348290e8d4bb434b30f57fffb3611531064e480993b0bcef3e62b9
[INFO]	2023-07-26T18:59:11.095Z		Found credentials in environment variables.
START RequestId: 18b316b6-a9ae-41e6-916b-11f10255d1d8 Version: $LATEST
[INFO]	2023-07-26T18:59:11.222Z	18b316b6-a9ae-41e6-916b-11f10255d1d8	Selected property is temperature
[ERROR]	2023-07-26T18:59:11.690Z	18b316b6-a9ae-41e6-916b-11f10255d1d8	Exception while running query: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
[ERROR] ValidationException: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 100, in lambda_handler
    raise err
  File "/var/task/lambda_function.py", line 95, in lambda_handler
    query_page = QUERY_CLIENT.query(
  File "/var/runtime/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)
END RequestId: 18b316b6-a9ae-41e6-916b-11f10255d1d8
REPORT RequestId: 18b316b6-a9ae-41e6-916b-11f10255d1d8	Duration: 510.76 ms	Billed Duration: 511 ms	Memory Size: 128 MB	Max Memory Used: 68 MB	Init Duration: 414.25 ms	
START RequestId: 5a7256aa-4f77-49bd-8c6f-21a3303028bf Version: $LATEST
[INFO]	2023-07-26T19:00:43.923Z	5a7256aa-4f77-49bd-8c6f-21a3303028bf	Selected property is temperature
[ERROR]	2023-07-26T19:00:44.071Z	5a7256aa-4f77-49bd-8c6f-21a3303028bf	Exception while running query: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
[ERROR] ValidationException: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 100, in lambda_handler
    raise err
  File "/var/task/lambda_function.py", line 95, in lambda_handler
    query_page = QUERY_CLIENT.query(
  File "/var/runtime/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)
END RequestId: 5a7256aa-4f77-49bd-8c6f-21a3303028bf
REPORT RequestId: 5a7256aa-4f77-49bd-8c6f-21a3303028bf	Duration: 149.68 ms	Billed Duration: 150 ms	Memory Size: 128 MB	Max Memory Used: 69 MB	
START RequestId: fc46934c-bda1-424e-bd3e-8babcb46f76e Version: $LATEST
[INFO]	2023-07-26T19:01:10.257Z	fc46934c-bda1-424e-bd3e-8babcb46f76e	Selected property is temperature
[ERROR]	2023-07-26T19:01:10.410Z	fc46934c-bda1-424e-bd3e-8babcb46f76e	Exception while running query: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
[ERROR] ValidationException: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 100, in lambda_handler
    raise err
  File "/var/task/lambda_function.py", line 95, in lambda_handler
    query_page = QUERY_CLIENT.query(
  File "/var/runtime/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)
END RequestId: fc46934c-bda1-424e-bd3e-8babcb46f76e
REPORT RequestId: fc46934c-bda1-424e-bd3e-8babcb46f76e	Duration: 154.94 ms	Billed Duration: 155 ms	Memory Size: 128 MB	Max Memory Used: 69 MB	
START RequestId: 092f2e09-02a8-4a53-9e52-14f075bb1a26 Version: $LATEST
[INFO]	2023-07-26T19:02:19.209Z	092f2e09-02a8-4a53-9e52-14f075bb1a26	Selected property is temperature
[ERROR]	2023-07-26T19:02:19.372Z	092f2e09-02a8-4a53-9e52-14f075bb1a26	Exception while running query: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
[ERROR] ValidationException: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 100, in lambda_handler
    raise err
  File "/var/task/lambda_function.py", line 95, in lambda_handler
    query_page = QUERY_CLIENT.query(
  File "/var/runtime/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)
END RequestId: 092f2e09-02a8-4a53-9e52-14f075bb1a26
REPORT RequestId: 092f2e09-02a8-4a53-9e52-14f075bb1a26	Duration: 182.68 ms	Billed Duration: 183 ms	Memory Size: 128 MB	Max Memory Used: 69 MB	
START RequestId: 0ffb24ef-b139-4042-a509-ca4b4cebd9a7 Version: $LATEST
[INFO]	2023-07-26T19:06:16.023Z	0ffb24ef-b139-4042-a509-ca4b4cebd9a7	Selected property is temperature
[ERROR]	2023-07-26T19:06:16.168Z	0ffb24ef-b139-4042-a509-ca4b4cebd9a7	Exception while running query: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
[ERROR] ValidationException: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 100, in lambda_handler
    raise err
  File "/var/task/lambda_function.py", line 95, in lambda_handler
    query_page = QUERY_CLIENT.query(
  File "/var/runtime/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)
END RequestId: 0ffb24ef-b139-4042-a509-ca4b4cebd9a7
REPORT RequestId: 0ffb24ef-b139-4042-a509-ca4b4cebd9a7	Duration: 150.92 ms	Billed Duration: 151 ms	Memory Size: 128 MB	Max Memory Used: 69 MB	
START RequestId: b38cb572-3546-46ce-a859-31d844aff6fb Version: $LATEST
[INFO]	2023-07-26T19:07:30.725Z	b38cb572-3546-46ce-a859-31d844aff6fb	Selected property is temperature
[ERROR]	2023-07-26T19:07:30.856Z	b38cb572-3546-46ce-a859-31d844aff6fb	Exception while running query: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
[ERROR] ValidationException: An error occurred (ValidationException) when calling the Query operation: The query syntax is invalid at line 1:75
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 100, in lambda_handler
    raise err
  File "/var/task/lambda_function.py", line 95, in lambda_handler
    query_page = QUERY_CLIENT.query(
  File "/var/runtime/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)
END RequestId: b38cb572-3546-46ce-a859-31d844aff6fb
REPORT RequestId: b38cb572-3546-46ce-a859-31d844aff6fb	Duration: 150.50 ms	Billed Duration: 151 ms	Memory Size: 128 MB	Max Memory Used: 69 MB	
2 Antworten
0
Akzeptierte Antwort

Hi. There seems to be a problem with your query_string. I note that your database name is TempHumidityDatabase-test and your table name is TempHumidityTable. However the blog post uses TempHumidityDatabase and TempHumidity. Have you correctly set TIMESTREAM_DATABASE_NAME and TIMESTREAM_TABLE_NAME?

profile pictureAWS
EXPERTE
Greg_B
beantwortet vor 9 Monaten
  • Hi! I have correctly configured TIMESTREAM_DATABASE_NAME and TIMESTREAM_TABLE_NAME in my environmental variables.

0

If I consider your generated query to be roughly same as

SELECT measure_name, time, measure_value::double FROM {DATABASE_NAME}.{TABLE_NAME}  WHERE time > from_iso8601_timestamp('{event['startTime']}') AND time <= from_iso8601_timestamp('{event['endTime']}') AND measure_name = '{selected_property}' ORDER BY time ASC

The query syntax is invalid at line 1:75 comes out to be somewhere around database / table name . Are you properly enclosing database / table name with double quotes as you have "-" in the name?

The code needs correction at FROM {DATABASE_NAME}.{TABLE_NAME} to FROM "{DATABASE_NAME}"."{TABLE_NAME}"

AWS
beantwortet vor 9 Monaten
  • Thank you for your feedback. My function is still returning a syntax error in the same place (line 1:75) with this modification. Would you suggest changing the database/table name to remove the "-"?

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen