EB Ruby migration to AMI 2

1

We are in the Migration from "Puma with Ruby 2.6 running on 64bit Amazon Linux" to the new AMI2 Images for the Version "Ruby 2.6 running on 64bit Amazon Linux 2". We were able to Update all related configs but it seems that Puma wont start after an successful deployment. Our Problem is, we do not get any useful ouptut from the Logs except this one:

[8886] /opt/rubies/ruby-2.6.9/lib/ruby/site_ruby/2.6.0/bundler/runtime.rb:309:in `check_for_activated_spec!'
[8890] + Gemfile in context: /var/app/current/Gemfile
[8888] ! Unable to start worker
[8888] /opt/rubies/ruby-2.6.9/lib/ruby/site_ruby/2.6.0/bundler/runtime.rb:309:in `check_for_activated_spec!'
[8892] + Gemfile in context: /var/app/current/Gemfile
[8890] ! Unable to start worker
[8890] /opt/rubies/ruby-2.6.9/lib/ruby/site_ruby/2.6.0/bundler/runtime.rb:309:in `check_for_activated_spec!'
[8890] Early termination of worker
[8894] + Gemfile in context: /var/app/current/Gemfile
[8892] ! Unable to start worker
[8892] /opt/rubies/ruby-2.6.9/lib/ruby/site_ruby/2.6.0/bundler/runtime.rb:309:in `check_for_activated_spec!'
[8907] + Gemfile in context: /var/app/current/Gemfile
[18563] === puma shutdown: 2022-03-18 13:10:25 +0100 ===
[18563] - Goodbye!
[18563] - Gracefully shutting down workers...
[8894] Early termination of worker
[8907] Early termination of worker

Is there any proper way to figure out why it fails there?

  • I'm experiencing the same issue, also with a migration of a Ruby 2.7.x / Puma 3.x app to Amazon Linux 2.

    I had hoped to get more debugging info by shutting down the web service, and running Puma in an SSH session. Instead, the server came up successfully. So, my best guess is that it has to do with how the systemd service is configured or launched. Any insights?

asked 2 years ago574 views
1 Answer
0

I was able to resolve a similar issue, which as it turns out has been discussed elsewhere.

The heart of the issue is: Elastic Beanstalk instances have a pool of system gems, and also install the app's bundle of gems in /var/app/current/vendor/. The default Procfile lets gems from these two sets be intermingled, which can result in version inconsistencies for loaded gems, and crashes.

The crashes only happen in Puma worker processes, and the details don't appear in any ordinary EB logs, which made this particularly tricky to debug. You can get more debugging info about issues with Puma workers by modifying the Puma config file to run in single process mode.

sudo nano /opt/elasticbeanstalk/config/private/pumaconf.rb

directory '/var/app/current'
# Commented default configuration:
# threads 8, 32
# workers %x(grep -c processor /proc/cpuinfo)
# Temporary debugging configuration:
threads 1, 1
workers 0
bind 'unix:///var/run/puma/my_app.sock'
stdout_redirect '/var/log/puma/puma.log', '/var/log/puma/puma.log', true

Restart the web service to have these settings take effect: sudo systemctl restart web.service

Now the full exception will appear in Puma's logs, follow the web service's logs with: sudo journalctl -u web.service -b -f

There are two ways you could choose to resolve this problem, both with tradeoffs.

One option is to fix the version of the mismatched gem (in this case, nio4r) in your Gemfile to match the version installed in EB (see this version with gem list). The drawback to this solution is that the crash will reappear if the gem is updated when AWS updates the Ruby platform (or, if the gem version later changes in your Gemfile).

A second option is to exclude the system gems from execution, and require that only the bundled gems be used. This could still result in problems after a future AWS platform update, but only if the gems in your bundle are incompatible with the EB platform in some way. This seems like the more stable solution, and it's the one I opted for.

At this writing, the default Ruby Procfile is:

web: puma -C /opt/elasticbeanstalk/config/private/pumaconf.rb

To implement the second option, add a Procfile to the root of your app with bundle exec instead:

web: bundle exec puma -C /opt/elasticbeanstalk/config/private/pumaconf.rb

(If you modified pumaconf.rb, don't forget to reverse the debugging changes and restart the service!)

answered 2 years ago
  • Hey Patrick,

    Many Thanks for the Reply, i wasn't able to tackle it again since now, i will try that this Month, i'll let you know how it will go ;)

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions