Greengrass V2 S3 export problem

0

Hi,
I'm trying to develop a component that will be exporting media files from IoT device (nvidia jetson nano). I wanted to start with a simple example provided in this repo: https://github.com/aws-greengrass/aws-greengrass-stream-manager-sdk-python/blob/main/samples/stream_manager_s3.py
I basically copy/pasted this example with one change. I had to configure stream manager client to connect to different port that default one (8099) because 8088 is already taken. The change looks like this:
client = StreamManagerClient(port=8099)
Everything else looks the same as in the example. The issue I'm having is that the code is not able to read the status messages. I'm getting this in the logs:
2021-01-08T16:25:20.633Z [INFO] (pool-2-thread-35) media_exporter: shell-runner-start. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=STARTING, command=["python3 /home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/ex..."]}
2021-01-08T16:25:21.199Z [WARN] (Copier) media_exporter: stderr. INFO:root:Successfully appended S3 Task Definition to stream with sequence number 0. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.211Z [WARN] (Copier) media_exporter: stderr. ERROR:root:Exception while running. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.211Z [WARN] (Copier) media_exporter: stderr. Traceback (most recent call last):. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.211Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/exporter.py", line 89, in main. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.211Z [WARN] (Copier) media_exporter: stderr. status_stream_name, ReadMessagesOptions(min_message_count=1, read_timeout_millis=5000). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.211Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/streammanagerclient.py", line 460, in read_messages. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.211Z [WARN] (Copier) media_exporter: stderr. return UtilInternal.sync(self._read_messages(stream_name, options), loop=self.__loop). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.211Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/utilinternal.py", line 39, in sync. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. return asyncio.run_coroutine_threadsafe(coro, loop=loop).result(). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. File "/usr/lib/python3.7/concurrent/futures/_base.py", line 435, in result. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. return self.__get_result(). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. File "/usr/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. raise self._exception. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/streammanagerclient.py", line 415, in _read_messages. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. UtilInternal.raise_on_error_response(read_messages_response). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/utilinternal.py", line 202, in raise_on_error_response. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. raise NotEnoughMessagesException(response.error_message, response.status, response.request_id). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. stream_manager.exceptions.NotEnoughMessagesException: not enough messages to return before time out. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}

I understand the error (not enough messages to return before time out.) but why this is happening?
At the end the files are being exported but this read status part it's not working at all.

Here is the snippet of the code that creates the streams:

Try deleting the status stream (if it exists) so that we have a fresh start

    try:  
        client.delete_message_stream(stream_name=status_stream_name)  
    except ResourceNotFoundException:  
        pass  

    # Try deleting the stream (if it exists) so that we have a fresh start  
    try:  
        client.delete_message_stream(stream_name=stream_name)  
    except ResourceNotFoundException:  
        pass  


    exports = ExportDefinition(  
        s3_task_executor=\[  
            S3ExportTaskExecutorConfig(  
                identifier="S3TaskExecutor" + stream_name,  # Required  
                # Optional. Add an export status stream to add statuses for all S3 upload tasks.  
                status_config=StatusConfig(  
                    status_level=StatusLevel.INFO,  # Default is INFO level statuses.  
                    # Status Stream should be created before specifying in S3 Export Config.  
                    status_stream_name=status_stream_name,  
                ),  
            )  
        ]  
    )  
      
    # Create the Status Stream.  
    client.create_message_stream(  
        MessageStreamDefinition(name=status_stream_name, strategy_on_full=StrategyOnFull.OverwriteOldestData)  
    )  

    # Create the message stream with the S3 Export definition.  
    client.create_message_stream(  
        MessageStreamDefinition(  
            name=stream_name,   
            strategy_on_full=StrategyOnFull.OverwriteOldestData,   
            export_definition=exports  
        )  
    )  

    # Append a S3 Task definition and print the sequence number  
    s3_export_task_definition = S3ExportTaskDefinition(input_url=file_url, bucket=bucket_name, key=key_name)  
    logger.info(  
        "Successfully appended S3 Task Definition to stream with sequence number %d",  
        client.append_message(stream_name, Util.validate_and_serialize_to_json_bytes(s3_export_task_definition)),  
    )
asked 3 years ago794 views
13 Answers
0

Hi szymon888,

The error seems to be coming from: https://github.com/aws-greengrass/aws-greengrass-stream-manager-sdk-python/blob/main/samples/stream_manager_s3.py#L115

The way I see it is this error should be harmless. The status is not immediately appear in the status stream and hence NotEnoughMessagesException is expected. But the status should eventually appear.

So:

  1. Can you include the complete logs? I hope to see the log string "Successfully uploaded file at path" in the logs.
  2. Also to further debug, please include the GG V2 logs. You can find them here: /greengrass/v2/logs
answered 3 years ago
0

Hi,
Exactly. I know that the error comes from the
https://github.com/aws-greengrass/aws-greengrass-stream-manager-sdk-python/blob/main/samples/stream_manager_s3.py#L115
The exception is thrown and is caught by except StreamManagerException:
The problem is that the status never appear in the status stream.

Here are the logs:
2021-01-08T16:25:20.633Z [INFO] (pool-2-thread-35) media_exporter: shell-runner-start. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=STARTING, command=["python3 /home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/ex..."]}
2021-01-08T16:25:21.199Z [WARN] (Copier) media_exporter: stderr. INFO:root:Successfully appended S3 Task Definition to stream with sequence number 0. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.211Z [WARN] (Copier) media_exporter: stderr. ERROR:root:Exception while running. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.211Z [WARN] (Copier) media_exporter: stderr. Traceback (most recent call last):. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.211Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/exporter.py", line 89, in main. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.211Z [WARN] (Copier) media_exporter: stderr. status_stream_name, ReadMessagesOptions(min_message_count=1, read_timeout_millis=5000). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.211Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/streammanagerclient.py", line 460, in read_messages. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.211Z [WARN] (Copier) media_exporter: stderr. return UtilInternal.sync(self._read_messages(stream_name, options), loop=self.__loop). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.211Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/utilinternal.py", line 39, in sync. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. return asyncio.run_coroutine_threadsafe(coro, loop=loop).result(). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. File "/usr/lib/python3.7/concurrent/futures/_base.py", line 435, in result. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. return self.__get_result(). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. File "/usr/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. raise self._exception. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/streammanagerclient.py", line 415, in _read_messages. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. UtilInternal.raise_on_error_response(read_messages_response). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/utilinternal.py", line 202, in raise_on_error_response. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. raise NotEnoughMessagesException(response.error_message, response.status, response.request_id). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. stream_manager.exceptions.NotEnoughMessagesException: not enough messages to return before time out. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.218Z [WARN] (Copier) media_exporter: stderr. ERROR:root:Exception while running. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.218Z [WARN] (Copier) media_exporter: stderr. Traceback (most recent call last):. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.218Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/exporter.py", line 89, in main. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.218Z [WARN] (Copier) media_exporter: stderr. status_stream_name, ReadMessagesOptions(min_message_count=1, read_timeout_millis=5000). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.218Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/streammanagerclient.py", line 460, in read_messages. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.218Z [WARN] (Copier) media_exporter: stderr. return UtilInternal.sync(self._read_messages(stream_name, options), loop=self.__loop). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.219Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/utilinternal.py", line 39, in sync. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.219Z [WARN] (Copier) media_exporter: stderr. return asyncio.run_coroutine_threadsafe(coro, loop=loop).result(). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.219Z [WARN] (Copier) media_exporter: stderr. File "/usr/lib/python3.7/concurrent/futures/_base.py", line 435, in result. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.219Z [WARN] (Copier) media_exporter: stderr. return self.__get_result(). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.219Z [WARN] (Copier) media_exporter: stderr. File "/usr/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.219Z [WARN] (Copier) media_exporter: stderr. raise self._exception. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.219Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/streammanagerclient.py", line 415, in _read_messages. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.219Z [WARN] (Copier) media_exporter: stderr. UtilInternal.raise_on_error_response(read_messages_response). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.220Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/utilinternal.py", line 202, in raise_on_error_response. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.220Z [WARN] (Copier) media_exporter: stderr. raise NotEnoughMessagesException(response.error_message, response.status, response.request_id). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.220Z [WARN] (Copier) media_exporter: stderr. stream_manager.exceptions.NotEnoughMessagesException: not enough messages to return before time out. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.225Z [WARN] (Copier) media_exporter: stderr. ERROR:root:Exception while running. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.225Z [WARN] (Copier) media_exporter: stderr. Traceback (most recent call last):. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.225Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/exporter.py", line 89, in main. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.225Z [WARN] (Copier) media_exporter: stderr. status_stream_name, ReadMessagesOptions(min_message_count=1, read_timeout_millis=5000). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.225Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/streammanagerclient.py", line 460, in read_messages. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.225Z [WARN] (Copier) media_exporter: stderr. return UtilInternal.sync(self._read_messages(stream_name, options), loop=self.__loop). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.225Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/utilinternal.py", line 39, in sync. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.226Z [WARN] (Copier) media_exporter: stderr. return asyncio.run_coroutine_threadsafe(coro, loop=loop).result(). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.226Z [WARN] (Copier) media_exporter: stderr. File "/usr/lib/python3.7/concurrent/futures/_base.py", line 435, in result. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.226Z [WARN] (Copier) media_exporter: stderr. return self.__get_result(). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.226Z [WARN] (Copier) media_exporter: stderr. File "/usr/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.226Z [WARN] (Copier) media_exporter: stderr. raise self._exception. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.226Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/streammanagerclient.py", line 415, in _read_messages. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.226Z [WARN] (Copier) media_exporter: stderr. UtilInternal.raise_on_error_response(read_messages_response). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.226Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/utilinternal.py", line 202, in raise_on_error_response. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.226Z [WARN] (Copier) media_exporter: stderr. raise NotEnoughMessagesException(response.error_message, response.status, response.request_id). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.226Z [WARN] (Copier) media_exporter: stderr. stream_manager.exceptions.NotEnoughMessagesException: not enough messages to return before time out. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.233Z [WARN] (Copier) media_exporter: stderr. ERROR:root:Exception while running. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.233Z [WARN] (Copier) media_exporter: stderr. Traceback (most recent call last):. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.233Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/exporter.py", line 89, in main. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.233Z [WARN] (Copier) media_exporter: stderr. status_stream_name, ReadMessagesOptions(min_message_count=1, read_timeout_millis=5000). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.233Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/streammanagerclient.py", line 460, in read_messages. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.233Z [WARN] (Copier) media_exporter: stderr. return UtilInternal.sync(self._read_messages(stream_name, options), loop=self.__loop). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.233Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/utilinternal.py", line 39, in sync. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.233Z [WARN] (Copier) media_exporter: stderr. return asyncio.run_coroutine_threadsafe(coro, loop=loop).result(). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.234Z [WARN] (Copier) media_exporter: stderr. File "/usr/lib/python3.7/concurrent/futures/_base.py", line 435, in result. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.234Z [WARN] (Copier) media_exporter: stderr. return self.__get_result(). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.234Z [WARN] (Copier) media_exporter: stderr. File "/usr/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.234Z [WARN] (Copier) media_exporter: stderr. raise self._exception. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.234Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/streammanagerclient.py", line 415, in _read_messages. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.234Z [WARN] (Copier) media_exporter: stderr. UtilInternal.raise_on_error_response(read_messages_response). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.234Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/utilinternal.py", line 202, in raise_on_error_response. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.235Z [WARN] (Copier) media_exporter: stderr. raise NotEnoughMessagesException(response.error_message, response.status, response.request_id). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.235Z [WARN] (Copier) media_exporter: stderr. stream_manager.exceptions.NotEnoughMessagesException: not enough messages to return before time out. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.240Z [WARN] (Copier) media_exporter: stderr. INFO:root:Successfully uploaded file at path file:/home/recordings/35513fae-4e82-4282-b17d-2c95a952a1f4/2020-12-10/23-58.mp4 to S3.. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}

And a GGC logs from that time:
2021-01-08T16:25:12.607Z [INFO] (Serialized listener processor) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-config-change. Requesting restart for component. {configNode=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=FINISHED}
2021-01-08T16:25:12.607Z [INFO] (Serialized listener processor) com.aws.greengrass.lifecyclemanager.GenericExternalService: Waiting for the desired state list. {serviceName=media_exporter, currentState=FINISHED}
2021-01-08T16:25:12.607Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-report-state. {serviceName=media_exporter, currentState=FINISHED, newState=INSTALLED}
2021-01-08T16:25:12.607Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=media_exporter, currentState=FINISHED, newState=INSTALLED}
2021-01-08T16:25:12.608Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.config.Configuration: Configuration currently updating, will wait for the update to complete.. {}
2021-01-08T16:25:12.616Z [INFO] (Serialized listener processor) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-config-change. Requesting reinstallation for component. {configNode=services.media_exporter.version, serviceName=media_exporter, currentState=INSTALLED}
2021-01-08T16:25:20.588Z [INFO] (Serialized listener processor) com.aws.greengrass.componentmanager.GreengrassComponentServiceClientFactory: initialize-greengrass-client. {service-region=eu-central-1, service-endpoint=https://greengrass-ats.iot.eu-central-1.amazonaws.com:8443}
2021-01-08T16:25:20.622Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.config.Configuration: Config update finished.. {}
2021-01-08T16:25:20.623Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-report-state. {serviceName=media_exporter, currentState=INSTALLED, newState=NEW}
2021-01-08T16:25:20.623Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=media_exporter, currentState=INSTALLED, newState=NEW}
2021-01-08T16:25:20.623Z [INFO] (aws.greengrass.Cli-lifecycle) com.aws.greengrass.config.Configuration: Config update finished.. {}
2021-01-08T16:25:20.623Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-report-state. {serviceName=media_exporter, currentState=NEW, newState=INSTALLED}
2021-01-08T16:25:20.623Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=media_exporter, currentState=NEW, newState=INSTALLED}
2021-01-08T16:25:20.624Z [INFO] (pool-2-thread-35) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-awaiting-start. waiting for dependencies to start. {serviceName=media_exporter, currentState=INSTALLED}
2021-01-08T16:25:20.624Z [INFO] (pool-2-thread-35) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-starting. {serviceName=media_exporter, currentState=INSTALLED}
2021-01-08T16:25:20.624Z [INFO] (pool-2-thread-35) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-report-state. {serviceName=media_exporter, currentState=INSTALLED, newState=STARTING}
2021-01-08T16:25:20.625Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=media_exporter, currentState=INSTALLED, newState=STARTING}
2021-01-08T16:25:20.625Z [INFO] (pool-2-thread-34) com.aws.greengrass.cli.CLIService: Set up symlink to CLI binary. {binary=/home/greengrass/packages/artifacts-unarchived/aws.greengrass.Cli/2.0.3/aws.greengrass.cli.client/cliclient-1.0-SNAPSHOT/bin/greengrass-cli, link=/home/greengrass/bin/greengrass-cli, serviceName=aws.greengrass.Cli, currentState=NEW}
2021-01-08T16:25:20.625Z [INFO] (aws.greengrass.Cli-lifecycle) com.aws.greengrass.cli.CLIService: service-report-state. {serviceName=aws.greengrass.Cli, currentState=NEW, newState=INSTALLED}
2021-01-08T16:25:20.625Z [INFO] (aws.greengrass.Cli-lifecycle) com.aws.greengrass.cli.CLIService: service-set-state. {serviceName=aws.greengrass.Cli, currentState=NEW, newState=INSTALLED}
2021-01-08T16:25:20.626Z [INFO] (pool-2-thread-34) com.aws.greengrass.cli.CLIService: service-awaiting-start. waiting for dependencies to start. {serviceName=aws.greengrass.Cli, currentState=INSTALLED}
2021-01-08T16:25:20.626Z [INFO] (pool-2-thread-34) com.aws.greengrass.cli.CLIService: service-starting. {serviceName=aws.greengrass.Cli, currentState=INSTALLED}
2021-01-08T16:25:20.626Z [INFO] (pool-2-thread-34) com.aws.greengrass.cli.CLIService: service-report-state. {serviceName=aws.greengrass.Cli, currentState=INSTALLED, newState=STARTING}
2021-01-08T16:25:20.627Z [INFO] (aws.greengrass.Cli-lifecycle) com.aws.greengrass.cli.CLIService: service-set-state. {serviceName=aws.greengrass.Cli, currentState=INSTALLED, newState=STARTING}
2021-01-08T16:25:20.628Z [INFO] (pool-2-thread-34) com.aws.greengrass.cli.CLIService: Auth tokens have been revoked. {serviceName=aws.greengrass.Cli, currentState=STARTING}
2021-01-08T16:25:20.629Z [INFO] (pool-2-thread-34) com.aws.greengrass.cli.CLIService: service-report-state. {serviceName=aws.greengrass.Cli, currentState=STARTING, newState=RUNNING}
2021-01-08T16:25:20.629Z [INFO] (aws.greengrass.Cli-lifecycle) com.aws.greengrass.cli.CLIService: service-set-state. {serviceName=aws.greengrass.Cli, currentState=STARTING, newState=RUNNING}
2021-01-08T16:25:20.650Z [INFO] (pool-2-thread-35) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-report-state. {serviceName=media_exporter, currentState=STARTING, newState=RUNNING}
2021-01-08T16:25:20.650Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=media_exporter, currentState=STARTING, newState=RUNNING}
2021-01-08T16:25:21.488Z [INFO] (pool-2-thread-35) com.aws.greengrass.tes.CredentialRequestHandler: Received IAM credentials that will be cached until 2021-01-08T17:20:21Z. {iotCredentialsPath=/role-aliases/GreengrassV2TokenExchangeRoleAlias/credentials}
2021-01-08T16:25:21.624Z [INFO] (pool-2-thread-33) com.aws.greengrass.deployment.DeploymentConfigMerger: merge-config. Removing services. {service-to-remove=[]}
2021-01-08T16:25:21.624Z [INFO] (pool-2-thread-33) com.aws.greengrass.deployment.activator.DeploymentActivator: merge-config. All services updated. {deploymentId=arn:aws:greengrass:eu-central-1:494176211087:configuration:thinggroup/safeway-stg:7}
2021-01-08T16:25:21.624Z [INFO] (pool-2-thread-33) com.aws.greengrass.lifecyclemanager.UpdateSystemPolicyService: service-update-finish. {serviceName=UpdateSystemPolicyService, currentState=RUNNING}
2021-01-08T16:25:21.624Z [INFO] (pool-2-thread-32) com.aws.greengrass.deployment.DeploymentService: deployment-task-execution. Finished deployment task. {deploymentId=arn:aws:greengrass:eu-central-1:494176211087:configuration:thinggroup/safeway-stg:7, serviceName=DeploymentService, currentState=RUNNING}
2021-01-08T16:25:33.842Z [INFO] (pool-2-thread-14) com.aws.greengrass.deployment.DeploymentService: Current deployment finished. {DeploymentId=8e060f8e-4c58-494f-846e-c0d6801531a1, serviceName=DeploymentService, currentState=RUNNING}
2021-01-08T16:25:33.919Z [INFO] (pool-2-thread-14) com.aws.greengrass.deployment.IotJobsHelper: Updating status of persisted deployment. {Status=SUCCEEDED, StatusDetails={detailed-deployment-status=SUCCESSFUL}, ThingName=nano-core-V3, JobId=8e060f8e-4c58-494f-846e-c0d6801531a1}
2021-01-08T16:25:38.771Z [INFO] (Thread-10) com.aws.greengrass.deployment.IotJobsHelper: Job status update was accepted. {Status=SUCCEEDED, ThingName=nano-core-V3, JobId=8e060f8e-4c58-494f-846e-c0d6801531a1}
2021-01-08T16:25:38.885Z [INFO] (pool-2-thread-14) com.aws.greengrass.status.FleetStatusService: fss-status-update-published. Status update published to FSS. {serviceName=FleetStatusService, currentState=RUNNING}
2021-01-08T16:25:38.885Z [INFO] (pool-2-thread-14) com.aws.greengrass.deployment.DeploymentDirectoryManager: Persist link to last deployment. {link=/home/greengrass/deployments/previous-success}
2021-01-08T16:25:38.885Z [INFO] (pool-2-thread-14) com.aws.greengrass.deployment.DeploymentDirectoryManager: Clean up link to earlier deployment. {link=/home/greengrass/deployments/previous-success}
2021-01-08T16:25:39.843Z [INFO] (Thread-10) com.aws.greengrass.deployment.IotJobsHelper: Received empty jobs in notification . {ThingName=nano-core-V3}
2021-01-08T16:25:42.290Z [INFO] (Copier) com.aws.greengrass.lifecyclemanager.GenericExternalService: Run script exited. {exitCode=0, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:42.291Z [INFO] (Copier) com.aws.greengrass.lifecyclemanager.GenericExternalService: generic-service-stopping. Service finished running. {serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:42.291Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-report-state. {serviceName=media_exporter, currentState=RUNNING, newState=STOPPING}
2021-01-08T16:25:42.292Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=media_exporter, currentState=RUNNING, newState=STOPPING}
2021-01-08T16:25:42.292Z [INFO] (pool-2-thread-32) com.aws.greengrass.lifecyclemanager.GenericExternalService: Shutdown initiated. {serviceName=media_exporter, currentState=STOPPING}
2021-01-08T16:25:42.292Z [INFO] (pool-2-thread-32) com.aws.greengrass.lifecyclemanager.GenericExternalService: generic-service-shutdown. {serviceName=media_exporter, currentState=STOPPING}
2021-01-08T16:25:42.292Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-report-state. {serviceName=media_exporter, currentState=STOPPING, newState=FINISHED}
2021-01-08T16:25:42.293Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=media_exporter, currentState=STOPPING, newState=FINISHED}

answered 3 years ago
0

Hi szymon888,

It looks like the upload status is put into the status stream and read by your code successfully, as shown by the log line:
"
2021-01-08T16:25:41.240Z https://forums.aws.amazon.com/ (Copier) media_exporter: stderr. INFO:root:Successfully uploaded file at path file:/home/recordings/35513fae-4e82-4282-b17d-2c95a952a1f4/2020-12-10/23-58.mp4 to S3.. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
"

The error you saw is because it took many seconds to upload your file to s3 and while loop (https://github.com/aws-greengrass/aws-greengrass-stream-manager-sdk-python/blob/main/samples/stream_manager_s3.py#L90) needs to run multiple times.

Edited by: fengaws on Jan 11, 2021 10:10 AM

answered 3 years ago
0

Hi,
I understand that perfectly but I was expecting to have some other status while the file is being uploaded. There is a Status.InProgress and to be honest I was expecting to have InProgress while the file is being uploaded. Maybe I'm getting it wrong....

Here is a snippet of my code:
try:
messages_list = client.read_messages(
status_stream_name, ReadMessagesOptions(min_message_count=1, read_timeout_millis=1000)
)
for message in messages_list:
# Deserialize the status message first.
status_message = Util.deserialize_json_bytes_to_obj(message.payload, StatusMessage)

                if status_message.status == Status.Success:  
                    logger.info("Successfully uploaded file at path " _ file_url _ " to S3.")  
                    is_file_uploaded_to_s3 = True  
                elif status_message.status == Status.InProgress:  
                    logger.info(f'File is being uploaded!')  
                elif status_message.status == Status.Failure or status_message.status == Status.Canceled:  
                    logger.info(  
                        "Unable to upload file at path " _ file_url _ " to S3. Message: " + status_message.message  
                    )  
                    is_file_uploaded_to_s3 = True  
            time.sleep(5)  
        except NotEnoughMessagesException as nem:  
            logger.debug('No messages in the status queue for the task')  
        except StreamManagerException:  
            logger.exception("Exception while running")  

I took 30M file on purpose to validate the status change.

answered 3 years ago
0

Hi,

From the document: https://docs.aws.amazon.com/greengrass/latest/developerguide/stream-export-configurations.html#export-streams-to-s3

Specifically, set the status configuration property of the stream’s S3ExportTaskExecutorConfig export configuration. This tells stream manager to write status messages about the export tasks to the status stream. In the StatusConfig object, specify the name of the status stream and the level of verbosity. The following supported values range from least verbose (ERROR) to most verbose (TRACE). The default is INFO.

To get the IN PROGRESS status, you will have to set the status level to TRACE. Sorry this is not covered in the document.
I will get the documentation rectified.

Please try this out and let us know.

answered 3 years ago
0

Hi,
I checked the status_level=StatusLevel.TRACE and indeed I was able to get the message status InProgress but now the status check loop is infinite. The status 'Success' is never set by the stream manager. The message status is constantly InProgress. I can confirm that the file is successfully uploaded to S3 after 40sec but the status is not changed.
This is very strange. I thought that it's a simple check but it's a nightmare to use.
Is there a source code of StreamManager? (Not python sdk - StreamManager itself).

answered 3 years ago
0

Hi,
I apologies for the bug in the sample. The sample is not checkpointing the desired_start_sequence_number. Here is the corrected version. I will get the samples updated too. Thanks for catching this issue.

# Read the statuses from the export status stream  
        stop_checking = False  
        checkpoint = 0  
        while not stop_checking:  
            try:  
                messages_list = client.read_messages(  
                    status_stream_name, ReadMessagesOptions(desired_start_sequence_number=checkpoint, min_message_count=1, read_timeout_millis=1000)  
                )  
                for message in messages_list:  
                    # Deserialize the status message first.  
                    status_message = Util.deserialize_json_bytes_to_obj(message.payload, StatusMessage)  
  
                    # Check the status of the status message. If the status is "Success",  
                    # the file was successfully uploaded to S3.  
                    # If the status was either "Failure" or "Cancelled", the server was unable to upload the file to S3.  
                    # We will print the message for why the upload to S3 failed from the status message.  
                    # If the status was "InProgress", the status indicates that the server has started uploading  
                    # the S3 task.  
                    if status_message.status == Status.Success:  
                        logger.info("Successfully uploaded file at path " _ file_url _ " to S3.")  
                        stop_checking = True  
                    elif status_message.status == Status.InProgress:  
                        logger.info("File upload is in Progress.")  
                        checkpoint = message.sequence_number  
                    elif status_message.status == Status.Failure or status_message.status == Status.Canceled:  
                        logger.info(  
                            "Unable to upload file at path " _ file_url _ " to S3. Message: " + status_message.message  
                        )  
                        stop_checking = True  
                if not stop_checking:  
                    time.sleep(5)  
                    checkpoint += 1  
            except StreamManagerException:  
                logger.exception("Exception while running")  

Edited by: sushantAtAws on Jan 14, 2021 4:45 PM

answered 3 years ago
0

Hi,
Thanks for the example. I came up with an almost exactly the same approach with local 'start_sequence' incrementing when something has been succesfully delivered to s3.

answered 3 years ago
0

Hi szymon888,
I also use this stream manager SDK in in my GG V2.
in the stream_manager_s3.py, I need to define the key_name and file url, I have a little confuse in the guide. I see your log you already upload successfuly with file_url"file:/home/recordings/35513fae-4e82-4282-b17d-2c95a952a1f4/2020-12-10/23-58.mp4", how do you set your key_name in your code?
key_name = "SomeKey"
file_url = "file:/path/to/some/file.someExtension"

2021-01-08T16:25:41.240Z https://forums.aws.amazon.com/ (Copier) media_exporter: stderr. INFO:root:Successfully uploaded file at path file:/home/recordings/35513fae-4e82-4282-b17d-2c95a952a1f4/2020-12-10/23-58.mp4 to S3.**

answered 3 years ago
0

hi @sushantAtAws

I use the sample code to test, I create a file at path /home/ubuntu/data.txt.
and set the file_url = "file:/home/ubuntu/data.txt" in the code , but when run the code, see the errr

stream_manager.exceptions.InvalidRequestException: File does not exist at path /home/ubuntu/data.txt.

do you know why?

2021-02-04T10:02:43.626Z [WARN] (Copier) com.example.StreamManager: stderr. UtilInternal.raise_on_error_response(append_message_response). {scriptName=services.com.example.StreamManager.lifecycle.Run, serviceName=com.example.StreamManager, currentState=RUNNING}
2021-02-04T10:02:43.626Z [WARN] (Copier) com.example.StreamManager: stderr. File "/greengrass/v2/packages/artifacts/com.example.StreamManager/1.0.0/aws-greengrass-stream-manager-sdk-python/stream_manager/utilinternal.py", line 188, in raise_on_error_response. {scriptName=services.com.example.StreamManager.lifecycle.Run, serviceName=com.example.StreamManager, currentState=RUNNING}
2021-02-04T10:02:43.626Z [WARN] (Copier) com.example.StreamManager: stderr. raise InvalidRequestException(response.error_message, response.status, response.request_id). {scriptName=services.com.example.StreamManager.lifecycle.Run, serviceName=com.example.StreamManager, currentState=RUNNING}
2021-02-04T10:02:43.626Z [WARN] (Copier) com.example.StreamManager: stderr. stream_manager.exceptions.InvalidRequestException: File does not exist at path /home/ubuntu/data.txt.

answered 3 years ago
0

Hi,
I'm not sure If I get you right but here is what I did:
Firstly I'm preparing s3_key:
s3_key = f'{dt.year}/{dt.month}/{dt.day}/{self.hostname}/{filename}'
and then

Append a S3 Task definition and print the sequence number

        user_data = {  
            'camera_id': camera_id or 'none' # User data cannot be null/None that's why adding 'none'  
        }  
        s3_export_task_definition = S3ExportTaskDefinition(input_url=file_url, bucket=self.S3_BUCKET_NAME, key=s3_key, user_metadata=user_data)  
        seq_num = self._stream_manager_client.append_message(self.S3_STREAM_NAME, Util.validate_and_serialize_to_json_bytes(s3_export_task_definition))  

        logger.info(f'Successfully appended S3 Task Definition to stream with sequence number {seq_num}')
answered 3 years ago
0

Hello @kennyGao,
I'm experiencing the same issue on a EC2 Linux instance that's running greengrass and the sample example. Have you managed to find a solution?

fbianco
answered 3 years ago
0

fbianco, kennyGao,
I would request you to open a new forum question for Stream Manager is GG V2.
Please provide detailed logs and description of the components involved. Also please mention if you are using Lambdas in GG V2.

answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions