- Newest
- Most votes
- Most comments
I run into this several times a day at this point. I suspect there's some corner case in my configuration that triggers this bug but given the non-descript log and the lib not being open source I can't troubleshoot.
The stack trace references what looks like a private gem. It looks like the eb is calling some script during prebuild that expects an argument "$1" and it doesn't get it (so it's nil) but attempts to concat it to a string. After calling .to_s on $1 to fix, I uncovered a race condition on "failed_key." This block is entered so failed_key must initially be truthy, yet when it's referenced in the block it's nil as well. I locally patched the gem on my instances to check if failed_key is truthy before using it like so:
case failure
when failed_key && @@cfn_cmd_failed_regex
name = prebuild ? "command #{$1}" : "container_command #{$1}"
ret = ExternalInvocationError.new(
reason: ebext_file_name ? "#{name} in #{ebext_file_name} failed." : "EBExtension #{name} failed.",
output: failed_key && 'Command ' + $1.to_s == failed_key.name ? failed_key.output : '',
exit_code: !failed_key || failed_key.exit_code.nil? ? 1 : failed_key.exit_code
)
It'd be lovely if I didn't have to do this. The only ebextensions I have are 2 yum installs (via "packages:"), a "sources:", "Resources" to setup s3 auth, and a "files" section for 2 files. None of those extensions error though. I also have some nginx conf but this seems like something in the yaml stuff.
The issue was memory was running out during some internal eb task as I was on micro instances but it was getting reported in this weird way. This also explains why rebooting would resolve the issue. I was able to find this in the cfn logs.
Relevant content
- asked 3 years ago
