Running Sidekiq on Heroku w/ Unicorn

Alex Egg,

This article describes how to run a sidekiq worker on heroku w/o paying for an additional heroku worker dyno.

First of all, I think heroku is doing a great service for the community by serving sites for free w/ one dyno. I know a lot of people who start on this free-tier and then migrate up to a paid tier. However, if you’re not ready to make that leap yet, there is still a way to have a sidekiq working by using Unicorn.

Unicorn is a webserver written in ruby which gives us the flexibility to hook into it’s event callbacks. A unicorn instance takes a config file:

worker_processes (ENV["WEB_CONCURRENCY"] || 3).to_i
timeout (ENV["WEB_TIMEOUT"] || 5).to_i
preload_app true

run_sidekiq_in_this_thread = ENV["RACK_ENV"]=="development" ? false : true
@sidekiq_pid = nil

before_fork do |server, worker|

  Signal.trap 'TERM' do
    puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
    Process.kill 'QUIT', Process.pid
  end

  defined?(ActiveRecord::Base) and
  ActiveRecord::Base.connection.disconnect!

  if run_sidekiq_in_this_thread
    p "starting Sidekiq w/ unicorn"
    @sidekiq_pid ||= spawn("bundle exec sidekiq")
    Rails.logger.info('Spawned sidekiq [email protected]_pid}')
  end
end


after_fork do |server, worker|

  Signal.trap 'TERM' do
    puts 'Unicorn worker intercepting TERM and doing nothing. Wait for master to sent QUIT'
  end

  if defined? ActiveRecord::Base
    config = ActiveRecord::Base.configurations[Rails.env] ||
      Rails.application.config.database_configuration[Rails.env]
    config["reaping_frequency"] = (ENV["DB_REAPING_FREQUENCY"] || 10).to_i
    config["pool"] = (ENV["DB_POOL"] || 2).to_i
    ActiveRecord::Base.establish_connection(config)
  end
end

Focus on the if run_sidekiq_in_this_thread block – because Unicorn lets us run arbitrary code in it’s fork callback, we can launch our sidekiq rake task along with our rails app.

So in this example we are running 3 rails processes and 1 sidekiq process in the same heroku dyno. The alternative is to pay ~$35/month for your delayed job that runs < %1 of the time.

Another alternative is to use a proper heroku worker dyno but adaptively scale up and down (turn on turn off) when you task needs to run. I’ll have an app that does this with good success, which I’ll document later.


Permalink: running-sidekiq-on-heroku-w-unicorn

Tags:

Last edited by Alex Egg, 2016-10-05 19:09:33
View Revision History