Wednesday, April 3, 2013

Using Unicorn Worker Killer to help reduce queue backlog on Heroku with Unicorn

In light of the update from Heroku CTO Adam Wiggins and having been part of the efforts to test the larger Dynos with Unicorn on Heroku I felt it necessary to share Unicorn Worker Killer; a tool we have found indispensable.

Based on configurable thresholds for memory and number of requests received it will kill off a Unicorn Worker. For those of you on Heroku this is valuable in that it then returns all the requests back to the Heroku random routing. While not an exact correlation between too many requests and memory size it does help keep individual workers from becoming too overloaded.

Add this to your Gemfile:

gem 'unicorn-worker-killer'

The gem suggests that you configure your `config.ru` file for the thresholds. This can be cumbersome on Heroku if you need to test out different settings.

Thankfully you can also control the thresholds via environment variables. In config.ru do:
max_request_min =  ENV['MAX_REQUEST_MIN'].to_i || 3072
max_request_max =  ENV['MAX_REQUEST_MAX'].to_i || 4096

# Max requests per worker
use Unicorn::WorkerKiller::MaxRequests, max_request_min, max_request_max

oom_min = ((ENV['OOM_MIN'].to_i || 192) * (1024**2))
oom_max = ((ENV['OOM_MAX'].to_i || 256) * (1024**2))

# Max memory size (RSS) per worker
use Unicorn::WorkerKiller::Oom, oom_min, oom_max


The run this on the heroku command line:
heroku config:add OOM_MAX=256 memory_limit_min =192 MAX_REQUEST_MIN=3072 MAX_REQUEST_MAX=4096 -a unicon-ttm-sandbox
That will add in the variables needed to control the thresholds. The example shows the defaults though we found dropping the OOM_MAX to 216 worked best

8 comments:

  1. Thanks for the write up.

    Have you tried Puma in place of Unicorn and seen any difference in instance load?

    ReplyDelete
    Replies
    1. We did see lower instance loads with Puma. Unfortunately to get the full power of Puma you need to use JRuby or Rubinius to have the multi-threaded capability kick in. There are some issues with each that we need to iron out before we can make that switch.

      Delete
  2. When you say OOM_MAX to 216 worked best, how many Unicorn workers do you have? 2?

    ReplyDelete
    Replies
    1. We had two but were able to go to five when Heroku allowed us to double our Dyno memory

      Delete
  3. I ran into a minor issue, when using the config.ru as defined above. As written, the file only works when the OOM_MAX and OOM_MIN environment variables are set.

    I made the following change so that the file also works with the default values, when the environment values are not set:

    oom_min = ( Integer(ENV['OOM_MIN']|| 192) * (1024**2))
    oom_max = ( Integer(ENV['OOM_MAX'] || 256) * (1024**2))

    ReplyDelete
  4. I used to get error.

    uninitialized constant Unicorn::WorkerKiller (NameError)

    Any hint ?

    ReplyDelete
  5. I used 'unicorn-worker-killer' gem with the some additional modificationfrom here for ruby GC http://blog.newrelic.com/2013/05/28/unicorn-rawk-kick-gc-out-of-the-band/

    But after the following instruction both there and deployed to production server. My application performance degrade gradually like

    1. App server response time goes from 350 ms avg to 1100 ms
    2. Page loading time goes from 6s avg to 13s

    Also my heroku combination is:

    1. 6 Web dyno with 1 gb memory
    2. 1 woker dyno with 1x speed.
    3. unicron worker process is 3
    4. my db connection is 40 and set db pool 2 at heroku.

    Please help me about how i optimize page loading time and app server time.

    Any idea?

    ReplyDelete