DeviceFarm: How to record a test failure?

Hello rjscratton, thank you for providing details about the issue you're experiencing with Appium and NodeJS tests in a custom test environment on AWS Device Farm. I understand that your tests are being reported as passed even when they fail. Let's address this issue step by step:

Identifying the problem: It appears that Device Farm is not correctly interpreting the exit code of your Node.js process when running tests in a custom environment.

Verifying the issue: To confirm this behavior, you can add some logging to your test script to output the exit code. For example:

    process.on('exit', (code) => {
      console.log(`Exiting with code: ${code}`);
    });

This will help you verify that your script is indeed exiting with a non-zero code when tests fail.

Potential solutions:

1. Explicit exit code: Instead of relying on the process exit code, you can explicitly call process.exit() with the appropriate code at the end of your tests. For example:

if (testsPassed) {
      console.log('Tests passed');
      process.exit(0);
    } else {
      console.log('Tests failed');
      process.exit(1);
    }

2. Use a wrapper script: Create a shell script that runs your Node.js test and checks its exit code. For example:

    #!/bin/bash
    node index.js $DEVICEFARM_APP_PATH
    exit $?

3. Make sure to make this script executable (chmod +x test_wrapper.sh) and update your Device Farm test spec to use this wrapper script instead of calling Node directly.

4. Error output redirection: Ensure that your test script is outputting errors to stderr. Node.js typically does this by default, but you can force it by using console.error()for failure messages.

5. Updating the test spec: In your Device Farm test spec YAML file, make sure you're using the correct command to run your tests. For example:

tests:
      - ./test_wrapper.sh

Or if using the Node.js script directly:

tests:
      - node index.js $DEVICEFARM_APP_PATH

Verifying the solution: After implementing one of these solutions, run your tests again in the Device Farm custom environment. Check the test results and logs to confirm that failures are now being correctly reported.

Additional considerations:

1. Ensure your Node.js version in the custom environment matches your local development environment.
2. Check that all necessary dependencies are installed in the custom environment.
3. Review the Device Farm logs for any unexpected errors or warnings that might be affecting test execution or reporting.

If you continue to experience issues after trying these solutions, I recommend reaching out to AWS Support for further assistance. They can provide more specific guidance based on your exact setup and test configuration.

Additional Resources:

https://docs.aws.amazon.com/codecatalyst/latest/userguide/test.error-handling.html

https://docs.aws.amazon.com/devicefarm/latest/developerguide/custom-test-environment-migration.html

https://docs.aws.amazon.com/devicefarm/latest/developerguide/test-environments.html

Best of luck and thank you for using AWS!

Brian