- Newest
- Most votes
- Most comments
Hello Rudolfs, Based on your description, this appears to be a specific issue with macOS Sonoma 14.7.4 instances on mac2.metal and the disk arbitration system. Let's break down the issue into individual sections in an attempt to find root cause:
Potential Root Causes:
There might be a race condition between ec2-macos-init's UnmountLocalSSD step and macOS Sonoma's disk arbitration system
The disk state information might not be properly cleared during the AMI creation process
Sonoma might have stricter disk arbitration policies that are conflicting with the EC2 initialization process
To troubleshoot:
Check the ec2-macos-init logs:
sudo cat /var/log/ec2-macos-init.log
Examine the disk setup:
diskutil list diskutil info disk0
Try manually resetting the disk arbitration daemon:
sudo killall diskarbitrationd
As a workaround, you might want to:
Add a delay before the UnmountLocalSSD step in init.toml
Create a new AMI after performing a clean shutdown with all disk operations completed
Consider filing an issue with AWS support as this might be a broader compatibility issue with Sonoma
The symptoms you're describing suggest a potential issue with the disk unmounting process:
High CPU usage from diskarbitrationd (100% on one core)
Continuous loop of unmount attempts and rejections
Issue persists across snapshots and new instances
Only affects Sonoma 14.7.4, not Ventura
The UnmountLocalSSD step in ec2-macos-init is designed to handle the initial unmounting of local SSDs during instance initialization. Here's what I suggest to troubleshoot and potentially resolve this:
First, check the current disk layout and mount status:
diskutil list diskutil info /dev/disk0 mount
Check the ec2-macos-init logs:
sudo cat /var/log/ec2-macos-init.log
Try manually running the disk arbitration debug command:
sudo log show --predicate 'subsystem == "com.apple.diskarbitrationd"' --last 1h
You can try to reset the disk arbitration daemon:
sudo launchctl unload /System/Library/LaunchDaemons/com.apple.diskarbitrationd.plist sudo launchctl load /System/Library/LaunchDaemons/com.apple.diskarbitrationd.plist
If this is indeed related to the UnmountLocalSSD step, you could try:
Create a custom AMI with a modified init.toml that comments out the UnmountLocalSSD step to test if this is the root cause.
Check if there are any stuck mount points:
lsof | grep /Volumes
Examine the system.log for any related errors:
sudo log show --predicate 'eventMessage CONTAINS "disk"' --last 30m
I recommend opening a support case with AWS as this might be a specific issue with the Sonoma 14.7.4 AMI that needs to be addressed at the platform level. Include:
The AMI ID you're using
Logs from diskarbitrationd
The ec2-macos-init logs
The results of the disk layout commands above
This could also be a timing issue with the initialization sequence in Sonoma 14.7.4 or a change in how macOS handles disk unmounting in this version.
Hopefully this provides some direction towards resolving this issue.
Thank you for using AWS! Brian
Relevant content
- asked 4 months ago
- asked 3 years ago
Thanks for the answer, yeah we just went with commenting out the unmounting step since that was the easiest way forward.