Forwarding log messages to log file using step function

Question

Hi there,

We are migrating from datapipeline to step function, and on datapipeline I am able to forward the log messages for individual steps to a log file, but this doesn't work on step function. Am I missing something in the configuration?

On datapipeline I have
/home/hadoop/general/ProfileData.sh $1  > >(tee -a /home/hadoop/myLogFile.log) -- Worked

On step function I have 
      "Args.$": "States.Array('bash', '/home/hadoop/general/ProfileData.sh', $.argument1, ' > >(tee -a /home/hadoop/myLogFile.log)' " --Not works

Answer

What integration are you calling from Step Functions?

Usually, in distributed systems like this, it is best to send the logs outside the host that generated them because you may not have access to it later. In AWS, we send logs to CloudWatch Logs.

You can then create a CloudWatch Logs subscription that will send the logs to a file in S3.

Forwarding log messages to log file using step function

Relevant content