Debugging Early Crashes in Nitro Enclaves (No Logs Before Death)

Question

We have a process that crashes IMMEDIATELY upon start in a Nitro Enclave 
(within 2 seconds), producing ZERO log output.

The same binary works perfectly on the host EC2 instance.

Questions:
1. When a process crashes before writing to stdout/stderr, is there ANY way 
   to see:
   - Kernel messages about the crash (dmesg inside enclave)?
   - Signal that killed the process (SIGILL, SIGSEGV, etc.)?
   - Core dumps or crash dumps?
   
2. Does debug mode (nitro-cli run-enclave --debug-mode) expose kernel-level 
   crash information that we're missing?
   
3. Is there a way to get MORE verbose logging from:
   - The enclave kernel initialization?
   - The Nitro hypervisor?
   - Process startup failures?
   
4. Can we enable kernel crash dumps or coredumps inside enclaves?

5. Are there any tools similar to strace/ltrace that work in enclaves?

Current situation: 
- nitro-cli console shows our init script output, then process dies silently
- No entries in any log files (/var/log/*, application logs)
- Makes it impossible to distinguish SIGILL vs OOM vs other failure modes

TDL · Accepted Answer

Ok, figured it out.  We had to capture crash signals and debug early failures with a crash detection wrapper.  The solution was to wrap the process to capture exit codes. Hope this saves someone hours of work!

```
#!/bin/bash
# crash-detect-wrapper
echo "[CRASH-DETECT] Starting: $@" | tee /dev/console
set +e
"$@" &
PID=$!
wait $PID
EXIT_CODE=$?
if [ $EXIT_CODE -gt 128 ]; then
    EXIT_SIGNAL=$((EXIT_CODE - 128))
    echo "[CRASH-DETECT] Killed by signal $EXIT_SIGNAL" | tee /dev/console
    case $EXIT_SIGNAL in
        4)  echo "SIGILL (Illegal Instruction) - CPU feature mismatch" ;;
        6)  echo "SIGABRT (Abort)" ;;
        11) echo "SIGSEGV (Segmentation Fault)" ;;
    esac
fi
dmesg | tail -20 | grep -i "out of memory\|illegal instruction" | tee /dev/console
exit $EXIT_CODE
```

Use: crash-detect-wrapper /path/to/your/binary --args

Output shows in nitro-cli console - you'll see the signal number!

**Note:** Requires `--debug-mode` to see console output. Also test your binary 
on the host first (outside enclave) to quickly rule out non-enclave issues.

Riku_Kobayashi · Answer

Hello.

Why not check the instance logs from the management console?  
These logs output information that can be used for troubleshooting kernel issues and other problems.  
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/troubleshoot-unreachable-instance.html#instance-console-console-output

TDL · Answer

Thanks - I checked the instance console and it shows the enclave starting up fine, 
but it doesn't show what happens inside the enclave when it crashes.

The instance console is the host's view of things of course, but I need to see inside the 
enclave itself to catch the crash, which is why I'm using nitro-cli console.

Do you know if SVE instructions are supported in Nitro Enclaves on Graviton3? 
My binary works on the host but crashes in the enclave with SIGILL, suggesting 
different CPU features are available.

Debugging Early Crashes in Nitro Enclaves (No Logs Before Death)

Add your answer

Relevant content