Delayed Jobs

Published about 6 years ago

---

Sometimes when you have a project, you'll want to run really long scripts and have them daemonized and running in the background. This allows your server to scale and not hang a http request and cause long latency requests on your box.

First Step

Let's install a few gems that will help us do exactly this. I chose to use delay jobs gem over sidekiq, because I felt it would be better for my usage of it. Keep in mind that active job was added in Rails 4.2, so if you are running a version less than that, it will not work.

gem 'delayed_job_active_record'
gem 'daemons'

Then run...

bundle install

Setting Up The Gem

We need to generate a migration using rails generator and then migrate it to the database where our jobs will be enqueued.

rails generate delayed_job:active_record
rails db:migrate

Application Settings

In your application.rb, we need to add a few config lines. Remember Rail's naming convention when creating files in your lib directory. The file name must be the class in lower camel case.

# config/application.rb
module AppName
  class Application < Rails::Application
    # autoload and expose lib/modules folder for additional classes and modules
    config.autoload_paths << Rails.root.join('lib/modules/')

    # or eager load it
    # config.eager_load_paths << Rails.root.join("lib")

    # tell rails to use delayed_job instead of the default one
    config.active_job.queue_adapter = :delayed_job
  end
end

Delayed Job Initializer

Now we will set default values for delayed job in its own initializer.

# config/initializers/delayed_job.rb
Delayed::Worker.destroy_failed_jobs = false
# defines the check interval to see if there are new jobs in seconds
Delayed::Worker.sleep_delay = 30
Delayed::Worker.max_attempts = 3
Delayed::Worker.max_run_time = 5.minutes
Delayed::Worker.read_ahead = 10
Delayed::Worker.default_queue_name = 'default'
Delayed::Worker.delay_jobs = !Rails.env.test?
Delayed::Worker.raise_signal_exceptions = :term
Delayed::Worker.logger = Logger.new(File.join(Rails.root, 'log', 'delayed_job.log'))

Creating a Job

To create a job, we just need to run the command below to auto create our class for what we need.

rails generate job FormBot

This will generate the below code.

# app/jobs/form_bot_job.rb
class FormBot < ActiveJob::Base
  queue_as :default

  def perform(*args)
    # Call a class's or module's method in your lib directory or
    # write you own code you want the job to do.
    Rails.logger.debug "Botting current arguments with FormBot: #{args.inspect}"
  end
end

If you'd like to test this job, use the following command from the terminal.

rails runner "FormBot.perform_later('Fill Form')" 

You can also call this job anywhere in your controller to initiate it.

# (*args) => one or more parameters you want to pass in 
FormBot.perform_later(*args)

# You can specify the priority and at what time it should run from when it's queued.
# Keep in mind if your sleep interval is every thirty seconds, this would get run within
# that time frame, and not always 10 seconds from when it's created.
FormBot.delay({:priority => 0, :run_at => 10.seconds.from_now}).perform_now

This will put a row in our job table to be queued and then ran by the following worker.

Running The Worker

To run our worker, we need to spin up another shell tab and start a rake task.

rake jobs:work

After running this command, it will open a binary that runs and will continuously check for new jobs and will output information as it goes.

Heroku

If you need to add this to Heroku, follow the steps here to add the worker to your proc file. Here's a guide by Heroku on pushing it all.

Databases

Workers can put quite a strain on your database because of how often it will hit it in your sleep delay interval. To help avoid this, it is recommended to use a Redis for the queue caching.