Passer au contenu

Debugging Early Crashes in Nitro Enclaves (No Logs Before Death)

0

We have a process that crashes IMMEDIATELY upon start in a Nitro Enclave (within 2 seconds), producing ZERO log output.

The same binary works perfectly on the host EC2 instance.

Questions:

  1. When a process crashes before writing to stdout/stderr, is there ANY way to see:

    • Kernel messages about the crash (dmesg inside enclave)?
    • Signal that killed the process (SIGILL, SIGSEGV, etc.)?
    • Core dumps or crash dumps?
  2. Does debug mode (nitro-cli run-enclave --debug-mode) expose kernel-level crash information that we're missing?

  3. Is there a way to get MORE verbose logging from:

    • The enclave kernel initialization?
    • The Nitro hypervisor?
    • Process startup failures?
  4. Can we enable kernel crash dumps or coredumps inside enclaves?

  5. Are there any tools similar to strace/ltrace that work in enclaves?

Current situation:

  • nitro-cli console shows our init script output, then process dies silently
  • No entries in any log files (/var/log/*, application logs)
  • Makes it impossible to distinguish SIGILL vs OOM vs other failure modes

demandé il y a un an352 vues

3 réponses
0
Réponse acceptée

Ok, figured it out. We had to capture crash signals and debug early failures with a crash detection wrapper. The solution was to wrap the process to capture exit codes. Hope this saves someone hours of work!

#!/bin/bash
# crash-detect-wrapper
echo "[CRASH-DETECT] Starting: $@" | tee /dev/console
set +e
"$@" &
PID=$!
wait $PID
EXIT_CODE=$?
if [ $EXIT_CODE -gt 128 ]; then
    EXIT_SIGNAL=$((EXIT_CODE - 128))
    echo "[CRASH-DETECT] Killed by signal $EXIT_SIGNAL" | tee /dev/console
    case $EXIT_SIGNAL in
        4)  echo "SIGILL (Illegal Instruction) - CPU feature mismatch" ;;
        6)  echo "SIGABRT (Abort)" ;;
        11) echo "SIGSEGV (Segmentation Fault)" ;;
    esac
fi
dmesg | tail -20 | grep -i "out of memory\|illegal instruction" | tee /dev/console
exit $EXIT_CODE

Use: crash-detect-wrapper /path/to/your/binary --args

Output shows in nitro-cli console - you'll see the signal number!

Note: Requires --debug-mode to see console output. Also test your binary on the host first (outside enclave) to quickly rule out non-enclave issues.

répondu il y a un an

AWS
EXPERT

vérifié il y a un an

0

Hello.

Why not check the instance logs from the management console?
These logs output information that can be used for troubleshooting kernel issues and other problems.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/troubleshoot-unreachable-instance.html#instance-console-console-output

EXPERT

répondu il y a un an

AWS
EXPERT

vérifié il y a un an

0

Thanks - I checked the instance console and it shows the enclave starting up fine, but it doesn't show what happens inside the enclave when it crashes.

The instance console is the host's view of things of course, but I need to see inside the enclave itself to catch the crash, which is why I'm using nitro-cli console.

Do you know if SVE instructions are supported in Nitro Enclaves on Graviton3? My binary works on the host but crashes in the enclave with SIGILL, suggesting different CPU features are available.

répondu il y a un an

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.