- Newest
- Most votes
- Most comments
Hi szymon888,
The error seems to be coming from: https://github.com/aws-greengrass/aws-greengrass-stream-manager-sdk-python/blob/main/samples/stream_manager_s3.py#L115
The way I see it is this error should be harmless. The status is not immediately appear in the status stream and hence NotEnoughMessagesException is expected. But the status should eventually appear.
So:
- Can you include the complete logs? I hope to see the log string "Successfully uploaded file at path" in the logs.
- Also to further debug, please include the GG V2 logs. You can find them here: /greengrass/v2/logs
Hi,
Exactly. I know that the error comes from the
https://github.com/aws-greengrass/aws-greengrass-stream-manager-sdk-python/blob/main/samples/stream_manager_s3.py#L115
The exception is thrown and is caught by except StreamManagerException:
The problem is that the status never appear in the status stream.
Here are the logs:
2021-01-08T16:25:20.633Z [INFO] (pool-2-thread-35) media_exporter: shell-runner-start. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=STARTING, command=["python3 /home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/ex..."]}
2021-01-08T16:25:21.199Z [WARN] (Copier) media_exporter: stderr. INFO:root:Successfully appended S3 Task Definition to stream with sequence number 0. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.211Z [WARN] (Copier) media_exporter: stderr. ERROR:root:Exception while running. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.211Z [WARN] (Copier) media_exporter: stderr. Traceback (most recent call last):. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.211Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/exporter.py", line 89, in main. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.211Z [WARN] (Copier) media_exporter: stderr. status_stream_name, ReadMessagesOptions(min_message_count=1, read_timeout_millis=5000). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.211Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/streammanagerclient.py", line 460, in read_messages. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.211Z [WARN] (Copier) media_exporter: stderr. return UtilInternal.sync(self._read_messages(stream_name, options), loop=self.__loop). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.211Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/utilinternal.py", line 39, in sync. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. return asyncio.run_coroutine_threadsafe(coro, loop=loop).result(). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. File "/usr/lib/python3.7/concurrent/futures/_base.py", line 435, in result. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. return self.__get_result(). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. File "/usr/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. raise self._exception. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/streammanagerclient.py", line 415, in _read_messages. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. UtilInternal.raise_on_error_response(read_messages_response). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/utilinternal.py", line 202, in raise_on_error_response. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. raise NotEnoughMessagesException(response.error_message, response.status, response.request_id). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:26.212Z [WARN] (Copier) media_exporter: stderr. stream_manager.exceptions.NotEnoughMessagesException: not enough messages to return before time out. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.218Z [WARN] (Copier) media_exporter: stderr. ERROR:root:Exception while running. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.218Z [WARN] (Copier) media_exporter: stderr. Traceback (most recent call last):. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.218Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/exporter.py", line 89, in main. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.218Z [WARN] (Copier) media_exporter: stderr. status_stream_name, ReadMessagesOptions(min_message_count=1, read_timeout_millis=5000). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.218Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/streammanagerclient.py", line 460, in read_messages. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.218Z [WARN] (Copier) media_exporter: stderr. return UtilInternal.sync(self._read_messages(stream_name, options), loop=self.__loop). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.219Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/utilinternal.py", line 39, in sync. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.219Z [WARN] (Copier) media_exporter: stderr. return asyncio.run_coroutine_threadsafe(coro, loop=loop).result(). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.219Z [WARN] (Copier) media_exporter: stderr. File "/usr/lib/python3.7/concurrent/futures/_base.py", line 435, in result. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.219Z [WARN] (Copier) media_exporter: stderr. return self.__get_result(). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.219Z [WARN] (Copier) media_exporter: stderr. File "/usr/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.219Z [WARN] (Copier) media_exporter: stderr. raise self._exception. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.219Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/streammanagerclient.py", line 415, in _read_messages. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.219Z [WARN] (Copier) media_exporter: stderr. UtilInternal.raise_on_error_response(read_messages_response). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.220Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/utilinternal.py", line 202, in raise_on_error_response. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.220Z [WARN] (Copier) media_exporter: stderr. raise NotEnoughMessagesException(response.error_message, response.status, response.request_id). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:31.220Z [WARN] (Copier) media_exporter: stderr. stream_manager.exceptions.NotEnoughMessagesException: not enough messages to return before time out. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.225Z [WARN] (Copier) media_exporter: stderr. ERROR:root:Exception while running. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.225Z [WARN] (Copier) media_exporter: stderr. Traceback (most recent call last):. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.225Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/exporter.py", line 89, in main. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.225Z [WARN] (Copier) media_exporter: stderr. status_stream_name, ReadMessagesOptions(min_message_count=1, read_timeout_millis=5000). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.225Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/streammanagerclient.py", line 460, in read_messages. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.225Z [WARN] (Copier) media_exporter: stderr. return UtilInternal.sync(self._read_messages(stream_name, options), loop=self.__loop). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.225Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/utilinternal.py", line 39, in sync. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.226Z [WARN] (Copier) media_exporter: stderr. return asyncio.run_coroutine_threadsafe(coro, loop=loop).result(). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.226Z [WARN] (Copier) media_exporter: stderr. File "/usr/lib/python3.7/concurrent/futures/_base.py", line 435, in result. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.226Z [WARN] (Copier) media_exporter: stderr. return self.__get_result(). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.226Z [WARN] (Copier) media_exporter: stderr. File "/usr/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.226Z [WARN] (Copier) media_exporter: stderr. raise self._exception. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.226Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/streammanagerclient.py", line 415, in _read_messages. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.226Z [WARN] (Copier) media_exporter: stderr. UtilInternal.raise_on_error_response(read_messages_response). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.226Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/utilinternal.py", line 202, in raise_on_error_response. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.226Z [WARN] (Copier) media_exporter: stderr. raise NotEnoughMessagesException(response.error_message, response.status, response.request_id). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:36.226Z [WARN] (Copier) media_exporter: stderr. stream_manager.exceptions.NotEnoughMessagesException: not enough messages to return before time out. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.233Z [WARN] (Copier) media_exporter: stderr. ERROR:root:Exception while running. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.233Z [WARN] (Copier) media_exporter: stderr. Traceback (most recent call last):. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.233Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/exporter.py", line 89, in main. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.233Z [WARN] (Copier) media_exporter: stderr. status_stream_name, ReadMessagesOptions(min_message_count=1, read_timeout_millis=5000). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.233Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/streammanagerclient.py", line 460, in read_messages. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.233Z [WARN] (Copier) media_exporter: stderr. return UtilInternal.sync(self._read_messages(stream_name, options), loop=self.__loop). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.233Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/utilinternal.py", line 39, in sync. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.233Z [WARN] (Copier) media_exporter: stderr. return asyncio.run_coroutine_threadsafe(coro, loop=loop).result(). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.234Z [WARN] (Copier) media_exporter: stderr. File "/usr/lib/python3.7/concurrent/futures/_base.py", line 435, in result. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.234Z [WARN] (Copier) media_exporter: stderr. return self.__get_result(). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.234Z [WARN] (Copier) media_exporter: stderr. File "/usr/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.234Z [WARN] (Copier) media_exporter: stderr. raise self._exception. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.234Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/streammanagerclient.py", line 415, in _read_messages. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.234Z [WARN] (Copier) media_exporter: stderr. UtilInternal.raise_on_error_response(read_messages_response). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.234Z [WARN] (Copier) media_exporter: stderr. File "/home/greengrass/packages/artifacts-unarchived/media_exporter/1.6.0/exporter/stream_manager/utilinternal.py", line 202, in raise_on_error_response. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.235Z [WARN] (Copier) media_exporter: stderr. raise NotEnoughMessagesException(response.error_message, response.status, response.request_id). {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.235Z [WARN] (Copier) media_exporter: stderr. stream_manager.exceptions.NotEnoughMessagesException: not enough messages to return before time out. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:41.240Z [WARN] (Copier) media_exporter: stderr. INFO:root:Successfully uploaded file at path file:/home/recordings/35513fae-4e82-4282-b17d-2c95a952a1f4/2020-12-10/23-58.mp4 to S3.. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
And a GGC logs from that time:
2021-01-08T16:25:12.607Z [INFO] (Serialized listener processor) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-config-change. Requesting restart for component. {configNode=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=FINISHED}
2021-01-08T16:25:12.607Z [INFO] (Serialized listener processor) com.aws.greengrass.lifecyclemanager.GenericExternalService: Waiting for the desired state list. {serviceName=media_exporter, currentState=FINISHED}
2021-01-08T16:25:12.607Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-report-state. {serviceName=media_exporter, currentState=FINISHED, newState=INSTALLED}
2021-01-08T16:25:12.607Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=media_exporter, currentState=FINISHED, newState=INSTALLED}
2021-01-08T16:25:12.608Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.config.Configuration: Configuration currently updating, will wait for the update to complete.. {}
2021-01-08T16:25:12.616Z [INFO] (Serialized listener processor) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-config-change. Requesting reinstallation for component. {configNode=services.media_exporter.version, serviceName=media_exporter, currentState=INSTALLED}
2021-01-08T16:25:20.588Z [INFO] (Serialized listener processor) com.aws.greengrass.componentmanager.GreengrassComponentServiceClientFactory: initialize-greengrass-client. {service-region=eu-central-1, service-endpoint=https://greengrass-ats.iot.eu-central-1.amazonaws.com:8443}
2021-01-08T16:25:20.622Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.config.Configuration: Config update finished.. {}
2021-01-08T16:25:20.623Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-report-state. {serviceName=media_exporter, currentState=INSTALLED, newState=NEW}
2021-01-08T16:25:20.623Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=media_exporter, currentState=INSTALLED, newState=NEW}
2021-01-08T16:25:20.623Z [INFO] (aws.greengrass.Cli-lifecycle) com.aws.greengrass.config.Configuration: Config update finished.. {}
2021-01-08T16:25:20.623Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-report-state. {serviceName=media_exporter, currentState=NEW, newState=INSTALLED}
2021-01-08T16:25:20.623Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=media_exporter, currentState=NEW, newState=INSTALLED}
2021-01-08T16:25:20.624Z [INFO] (pool-2-thread-35) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-awaiting-start. waiting for dependencies to start. {serviceName=media_exporter, currentState=INSTALLED}
2021-01-08T16:25:20.624Z [INFO] (pool-2-thread-35) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-starting. {serviceName=media_exporter, currentState=INSTALLED}
2021-01-08T16:25:20.624Z [INFO] (pool-2-thread-35) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-report-state. {serviceName=media_exporter, currentState=INSTALLED, newState=STARTING}
2021-01-08T16:25:20.625Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=media_exporter, currentState=INSTALLED, newState=STARTING}
2021-01-08T16:25:20.625Z [INFO] (pool-2-thread-34) com.aws.greengrass.cli.CLIService: Set up symlink to CLI binary. {binary=/home/greengrass/packages/artifacts-unarchived/aws.greengrass.Cli/2.0.3/aws.greengrass.cli.client/cliclient-1.0-SNAPSHOT/bin/greengrass-cli, link=/home/greengrass/bin/greengrass-cli, serviceName=aws.greengrass.Cli, currentState=NEW}
2021-01-08T16:25:20.625Z [INFO] (aws.greengrass.Cli-lifecycle) com.aws.greengrass.cli.CLIService: service-report-state. {serviceName=aws.greengrass.Cli, currentState=NEW, newState=INSTALLED}
2021-01-08T16:25:20.625Z [INFO] (aws.greengrass.Cli-lifecycle) com.aws.greengrass.cli.CLIService: service-set-state. {serviceName=aws.greengrass.Cli, currentState=NEW, newState=INSTALLED}
2021-01-08T16:25:20.626Z [INFO] (pool-2-thread-34) com.aws.greengrass.cli.CLIService: service-awaiting-start. waiting for dependencies to start. {serviceName=aws.greengrass.Cli, currentState=INSTALLED}
2021-01-08T16:25:20.626Z [INFO] (pool-2-thread-34) com.aws.greengrass.cli.CLIService: service-starting. {serviceName=aws.greengrass.Cli, currentState=INSTALLED}
2021-01-08T16:25:20.626Z [INFO] (pool-2-thread-34) com.aws.greengrass.cli.CLIService: service-report-state. {serviceName=aws.greengrass.Cli, currentState=INSTALLED, newState=STARTING}
2021-01-08T16:25:20.627Z [INFO] (aws.greengrass.Cli-lifecycle) com.aws.greengrass.cli.CLIService: service-set-state. {serviceName=aws.greengrass.Cli, currentState=INSTALLED, newState=STARTING}
2021-01-08T16:25:20.628Z [INFO] (pool-2-thread-34) com.aws.greengrass.cli.CLIService: Auth tokens have been revoked. {serviceName=aws.greengrass.Cli, currentState=STARTING}
2021-01-08T16:25:20.629Z [INFO] (pool-2-thread-34) com.aws.greengrass.cli.CLIService: service-report-state. {serviceName=aws.greengrass.Cli, currentState=STARTING, newState=RUNNING}
2021-01-08T16:25:20.629Z [INFO] (aws.greengrass.Cli-lifecycle) com.aws.greengrass.cli.CLIService: service-set-state. {serviceName=aws.greengrass.Cli, currentState=STARTING, newState=RUNNING}
2021-01-08T16:25:20.650Z [INFO] (pool-2-thread-35) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-report-state. {serviceName=media_exporter, currentState=STARTING, newState=RUNNING}
2021-01-08T16:25:20.650Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=media_exporter, currentState=STARTING, newState=RUNNING}
2021-01-08T16:25:21.488Z [INFO] (pool-2-thread-35) com.aws.greengrass.tes.CredentialRequestHandler: Received IAM credentials that will be cached until 2021-01-08T17:20:21Z. {iotCredentialsPath=/role-aliases/GreengrassV2TokenExchangeRoleAlias/credentials}
2021-01-08T16:25:21.624Z [INFO] (pool-2-thread-33) com.aws.greengrass.deployment.DeploymentConfigMerger: merge-config. Removing services. {service-to-remove=[]}
2021-01-08T16:25:21.624Z [INFO] (pool-2-thread-33) com.aws.greengrass.deployment.activator.DeploymentActivator: merge-config. All services updated. {deploymentId=arn:aws:greengrass:eu-central-1:494176211087:configuration:thinggroup/safeway-stg:7}
2021-01-08T16:25:21.624Z [INFO] (pool-2-thread-33) com.aws.greengrass.lifecyclemanager.UpdateSystemPolicyService: service-update-finish. {serviceName=UpdateSystemPolicyService, currentState=RUNNING}
2021-01-08T16:25:21.624Z [INFO] (pool-2-thread-32) com.aws.greengrass.deployment.DeploymentService: deployment-task-execution. Finished deployment task. {deploymentId=arn:aws:greengrass:eu-central-1:494176211087:configuration:thinggroup/safeway-stg:7, serviceName=DeploymentService, currentState=RUNNING}
2021-01-08T16:25:33.842Z [INFO] (pool-2-thread-14) com.aws.greengrass.deployment.DeploymentService: Current deployment finished. {DeploymentId=8e060f8e-4c58-494f-846e-c0d6801531a1, serviceName=DeploymentService, currentState=RUNNING}
2021-01-08T16:25:33.919Z [INFO] (pool-2-thread-14) com.aws.greengrass.deployment.IotJobsHelper: Updating status of persisted deployment. {Status=SUCCEEDED, StatusDetails={detailed-deployment-status=SUCCESSFUL}, ThingName=nano-core-V3, JobId=8e060f8e-4c58-494f-846e-c0d6801531a1}
2021-01-08T16:25:38.771Z [INFO] (Thread-10) com.aws.greengrass.deployment.IotJobsHelper: Job status update was accepted. {Status=SUCCEEDED, ThingName=nano-core-V3, JobId=8e060f8e-4c58-494f-846e-c0d6801531a1}
2021-01-08T16:25:38.885Z [INFO] (pool-2-thread-14) com.aws.greengrass.status.FleetStatusService: fss-status-update-published. Status update published to FSS. {serviceName=FleetStatusService, currentState=RUNNING}
2021-01-08T16:25:38.885Z [INFO] (pool-2-thread-14) com.aws.greengrass.deployment.DeploymentDirectoryManager: Persist link to last deployment. {link=/home/greengrass/deployments/previous-success}
2021-01-08T16:25:38.885Z [INFO] (pool-2-thread-14) com.aws.greengrass.deployment.DeploymentDirectoryManager: Clean up link to earlier deployment. {link=/home/greengrass/deployments/previous-success}
2021-01-08T16:25:39.843Z [INFO] (Thread-10) com.aws.greengrass.deployment.IotJobsHelper: Received empty jobs in notification . {ThingName=nano-core-V3}
2021-01-08T16:25:42.290Z [INFO] (Copier) com.aws.greengrass.lifecyclemanager.GenericExternalService: Run script exited. {exitCode=0, serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:42.291Z [INFO] (Copier) com.aws.greengrass.lifecyclemanager.GenericExternalService: generic-service-stopping. Service finished running. {serviceName=media_exporter, currentState=RUNNING}
2021-01-08T16:25:42.291Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-report-state. {serviceName=media_exporter, currentState=RUNNING, newState=STOPPING}
2021-01-08T16:25:42.292Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=media_exporter, currentState=RUNNING, newState=STOPPING}
2021-01-08T16:25:42.292Z [INFO] (pool-2-thread-32) com.aws.greengrass.lifecyclemanager.GenericExternalService: Shutdown initiated. {serviceName=media_exporter, currentState=STOPPING}
2021-01-08T16:25:42.292Z [INFO] (pool-2-thread-32) com.aws.greengrass.lifecyclemanager.GenericExternalService: generic-service-shutdown. {serviceName=media_exporter, currentState=STOPPING}
2021-01-08T16:25:42.292Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-report-state. {serviceName=media_exporter, currentState=STOPPING, newState=FINISHED}
2021-01-08T16:25:42.293Z [INFO] (media_exporter-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=media_exporter, currentState=STOPPING, newState=FINISHED}
Hi szymon888,
It looks like the upload status is put into the status stream and read by your code successfully, as shown by the log line:
"
2021-01-08T16:25:41.240Z https://forums.aws.amazon.com/ (Copier) media_exporter: stderr. INFO:root:Successfully uploaded file at path file:/home/recordings/35513fae-4e82-4282-b17d-2c95a952a1f4/2020-12-10/23-58.mp4 to S3.. {scriptName=services.media_exporter.lifecycle.Run, serviceName=media_exporter, currentState=RUNNING}
"
The error you saw is because it took many seconds to upload your file to s3 and while loop (https://github.com/aws-greengrass/aws-greengrass-stream-manager-sdk-python/blob/main/samples/stream_manager_s3.py#L90) needs to run multiple times.
Edited by: fengaws on Jan 11, 2021 10:10 AM
Hi,
I understand that perfectly but I was expecting to have some other status while the file is being uploaded. There is a Status.InProgress and to be honest I was expecting to have InProgress while the file is being uploaded. Maybe I'm getting it wrong....
Here is a snippet of my code:
try:
messages_list = client.read_messages(
status_stream_name, ReadMessagesOptions(min_message_count=1, read_timeout_millis=1000)
)
for message in messages_list:
# Deserialize the status message first.
status_message = Util.deserialize_json_bytes_to_obj(message.payload, StatusMessage)
if status_message.status == Status.Success:
logger.info("Successfully uploaded file at path " _ file_url _ " to S3.")
is_file_uploaded_to_s3 = True
elif status_message.status == Status.InProgress:
logger.info(f'File is being uploaded!')
elif status_message.status == Status.Failure or status_message.status == Status.Canceled:
logger.info(
"Unable to upload file at path " _ file_url _ " to S3. Message: " + status_message.message
)
is_file_uploaded_to_s3 = True
time.sleep(5)
except NotEnoughMessagesException as nem:
logger.debug('No messages in the status queue for the task')
except StreamManagerException:
logger.exception("Exception while running")
I took 30M file on purpose to validate the status change.
Hi,
From the document: https://docs.aws.amazon.com/greengrass/latest/developerguide/stream-export-configurations.html#export-streams-to-s3
Specifically, set the status configuration property of the stream’s S3ExportTaskExecutorConfig export configuration. This tells stream manager to write status messages about the export tasks to the status stream. In the StatusConfig object, specify the name of the status stream and the level of verbosity. The following supported values range from least verbose (ERROR) to most verbose (TRACE). The default is INFO.
To get the IN PROGRESS status, you will have to set the status level to TRACE. Sorry this is not covered in the document.
I will get the documentation rectified.
Please try this out and let us know.
Hi,
I checked the status_level=StatusLevel.TRACE and indeed I was able to get the message status InProgress but now the status check loop is infinite. The status 'Success' is never set by the stream manager. The message status is constantly InProgress. I can confirm that the file is successfully uploaded to S3 after 40sec but the status is not changed.
This is very strange. I thought that it's a simple check but it's a nightmare to use.
Is there a source code of StreamManager? (Not python sdk - StreamManager itself).
Hi,
I apologies for the bug in the sample. The sample is not checkpointing the desired_start_sequence_number
. Here is the corrected version. I will get the samples updated too. Thanks for catching this issue.
# Read the statuses from the export status stream
stop_checking = False
checkpoint = 0
while not stop_checking:
try:
messages_list = client.read_messages(
status_stream_name, ReadMessagesOptions(desired_start_sequence_number=checkpoint, min_message_count=1, read_timeout_millis=1000)
)
for message in messages_list:
# Deserialize the status message first.
status_message = Util.deserialize_json_bytes_to_obj(message.payload, StatusMessage)
# Check the status of the status message. If the status is "Success",
# the file was successfully uploaded to S3.
# If the status was either "Failure" or "Cancelled", the server was unable to upload the file to S3.
# We will print the message for why the upload to S3 failed from the status message.
# If the status was "InProgress", the status indicates that the server has started uploading
# the S3 task.
if status_message.status == Status.Success:
logger.info("Successfully uploaded file at path " _ file_url _ " to S3.")
stop_checking = True
elif status_message.status == Status.InProgress:
logger.info("File upload is in Progress.")
checkpoint = message.sequence_number
elif status_message.status == Status.Failure or status_message.status == Status.Canceled:
logger.info(
"Unable to upload file at path " _ file_url _ " to S3. Message: " + status_message.message
)
stop_checking = True
if not stop_checking:
time.sleep(5)
checkpoint += 1
except StreamManagerException:
logger.exception("Exception while running")
Edited by: sushantAtAws on Jan 14, 2021 4:45 PM
Hi,
Thanks for the example. I came up with an almost exactly the same approach with local 'start_sequence' incrementing when something has been succesfully delivered to s3.
Hi szymon888,
I also use this stream manager SDK in in my GG V2.
in the stream_manager_s3.py, I need to define the key_name and file url, I have a little confuse in the guide. I see your log you already upload successfuly with file_url"file:/home/recordings/35513fae-4e82-4282-b17d-2c95a952a1f4/2020-12-10/23-58.mp4", how do you set your key_name in your code?
key_name = "SomeKey"
file_url = "file:/path/to/some/file.someExtension"
2021-01-08T16:25:41.240Z https://forums.aws.amazon.com/ (Copier) media_exporter: stderr. INFO:root:Successfully uploaded file at path file:/home/recordings/35513fae-4e82-4282-b17d-2c95a952a1f4/2020-12-10/23-58.mp4 to S3.**
hi @sushantAtAws
I use the sample code to test, I create a file at path /home/ubuntu/data.txt.
and set the file_url = "file:/home/ubuntu/data.txt" in the code , but when run the code, see the errr
stream_manager.exceptions.InvalidRequestException: File does not exist at path /home/ubuntu/data.txt.
do you know why?
2021-02-04T10:02:43.626Z [WARN] (Copier) com.example.StreamManager: stderr. UtilInternal.raise_on_error_response(append_message_response). {scriptName=services.com.example.StreamManager.lifecycle.Run, serviceName=com.example.StreamManager, currentState=RUNNING}
2021-02-04T10:02:43.626Z [WARN] (Copier) com.example.StreamManager: stderr. File "/greengrass/v2/packages/artifacts/com.example.StreamManager/1.0.0/aws-greengrass-stream-manager-sdk-python/stream_manager/utilinternal.py", line 188, in raise_on_error_response. {scriptName=services.com.example.StreamManager.lifecycle.Run, serviceName=com.example.StreamManager, currentState=RUNNING}
2021-02-04T10:02:43.626Z [WARN] (Copier) com.example.StreamManager: stderr. raise InvalidRequestException(response.error_message, response.status, response.request_id). {scriptName=services.com.example.StreamManager.lifecycle.Run, serviceName=com.example.StreamManager, currentState=RUNNING}
2021-02-04T10:02:43.626Z [WARN] (Copier) com.example.StreamManager: stderr. stream_manager.exceptions.InvalidRequestException: File does not exist at path /home/ubuntu/data.txt.
Hi,
I'm not sure If I get you right but here is what I did:
Firstly I'm preparing s3_key:
s3_key = f'{dt.year}/{dt.month}/{dt.day}/{self.hostname}/{filename}'
and then
Append a S3 Task definition and print the sequence number
user_data = {
'camera_id': camera_id or 'none' # User data cannot be null/None that's why adding 'none'
}
s3_export_task_definition = S3ExportTaskDefinition(input_url=file_url, bucket=self.S3_BUCKET_NAME, key=s3_key, user_metadata=user_data)
seq_num = self._stream_manager_client.append_message(self.S3_STREAM_NAME, Util.validate_and_serialize_to_json_bytes(s3_export_task_definition))
logger.info(f'Successfully appended S3 Task Definition to stream with sequence number {seq_num}')
Hello @kennyGao,
I'm experiencing the same issue on a EC2 Linux instance that's running greengrass and the sample example. Have you managed to find a solution?
fbianco, kennyGao,
I would request you to open a new forum question for Stream Manager is GG V2.
Please provide detailed logs and description of the components involved. Also please mention if you are using Lambdas in GG V2.
Relevant content
- Accepted Answerasked a year ago
- asked a year ago
- asked 3 years ago
- Accepted Answerasked a year ago
- AWS OFFICIALUpdated a month ago
- AWS OFFICIALUpdated 7 months ago
- AWS OFFICIALUpdated 4 months ago
- AWS OFFICIALUpdated 10 months ago