Fly Machine Workers

This application uses Fly Machines as a scale-to-zero back-end for Active Jobs. When a background job gets kicked off, Fly provisions a Fly Machine that's a copy of this Rails app, runs the job, and powers down the machine forever. That means you're not paying for servers that run sidekiq and mostly sit around idle until they're given a background job. Those cost savings also mean your background works can scale up to a ridiculously large number, then scale back to 0 when nothing happens. Huzah!

Security

To mitigate this risk, you will need to create a separate Fly account to run Machines. If this account gets compromised, you should be OK with the damage an adversary could inflict. It still wouldn't be pretty, but at least they can't destroy all the resources of the primary account.

Be careful with this approach!

Step-by-step

Now that we understand the security implications of this lil' plan, lets get going!

Get the name of the running Rails app

This application runs under Fly's first generation of hosting infrastructure with the application name rails-machine-workers.

  def self.env_app_name
    ENV["FLY_APP_NAME"]
  end

We grab this from the FLY_APP_NAME environment variable that's injected into the container by Fly when it's launched. If you're running this from a development environment, you'll need to set this to the name of a provisioned Fly app.

Get the image name of the running app

Here's the image reference for the currently running application: registry.fly.io/rails-machine-workers@sha256:602a10257d8720b33ec32d6a8233e4326c3430f4ef0eb0934ff65320f2aec595.

    # Doocker image of the currently running application
    def image_ref
      machine_image_ref || query_image_ref
    end

    private

    # Newer containers are injecting this variable, which
    # means we can skip the lookup.
    def machine_image_ref
      ENV["FLY_IMAGE_REF"]
    end

    # Some environments don't have the `FLY_IMAGE_REF` variable
    # so we have to look it up via an internal, undocumented, GraphQL
    # that if you try using it, it will one day break.
    def query_image_ref
      api.graphql(%{
        query {
          app(name: "#{name}") {
            currentRelease {
              imageRef
            }
          }
        }
      }).dig("data", "app", "currentRelease", "imageRef")
    end
  end

We're going to use this to launch machines, but before we do that we have to create an Application for workers to run.

Create a Machines Fly app to run the Machines

Machines that are deployed to Fly need to belong to an application, so create one by running:

$ fly apps create rails-machine-workers-workers --machines

We'll use this to boot Machines for our Rails jobs!

Run a Machine

Now we have everything in place needed to run a Machine. Let's try it from the console.

$ Fly.app.machine.fork init: { cmd: %w[sleep 30] }

This command "forks" your existing application by booting a machine with the image_ref and ENV vars.

module Fly
  class Machine

    def initialize(app:)
      @api = app.api
      @app = app
    end

    def image_ref
      @app.image_ref
    end

    # If we're running the application from a Nomad, we need to create a
    # new Fly application that works with machines.
    def worker_app_name
      "#{@app.name}-workers"
    end

    # Runs a Fly machine. When the machine is done running, that's it! You
    # don't need to worry about deleting it later.
    def run(name: nil, region: nil, config: {})
      @api.post "/v1/apps/#{worker_app_name}/machines", {
        name: name,
        region: region,
        config: config
      }
    end

    def fork(**config)
      # Merge the currently running environment into the configuration of the
      # machine that we're going to launch. This includes secrets, so make sure
      # you're running your code in the machine and you trust the image. If you
      # don't trust the runtime, say for a CI environment, you'd leave the `ENV`
      # vars out, so you should probably use the `run` method above.
      config.fetch(:env, {}).merge! ENV
      # We're going to boot the image of the currently running application.
      config[:image] = image_ref
      run config: config
    end
  end
end

Now we can see a list of currently running machines.

rails-machine-workers [main] → fly m ls -a rails-machine-workers-workers
e148e446bd6989  wandering-water-7858    started sjc     rails-machine-workers:  fdaa:0:a177:a7b:b2e2:c8a6:3277:2          2022-09-21T21:51:50Z  2022-09-21T21:51:50Z
06e82956fe2e87  green-breeze-2572       started sjc     rails-machine-workers:  fdaa:0:a177:a7b:a160:c3d3:cbca:2          2022-09-21T20:45:29Z  2022-09-21T20:45:46Z
9e784935c6d383  dawn-silence-5439       started sjc     rails-machine-workers:  fdaa:0:a177:a7b:a15f:9ef5:232a:2          2022-09-21T20:45:27Z  2022-09-21T20:45:30Z
d5683004be048e  long-sun-7558           started sjc     rails-machine-workers:  fdaa:0:a177:a7b:2295:8b8f:c55a:2          2022-09-21T20:50:12Z  2022-09-21T20:50:15Z
e148e444c26489  wispy-moon-9832         started sjc     rails-machine-workers:  fdaa:0:a177:a7b:b385:41c3:238d:2          2022-09-21T20:44:56Z  2022-09-21T20:44:58Z
9e784996fd7183  ancient-breeze-5519     started sjc     rails-machine-workers:  fdaa:0:a177:a7b:b2e2:4ba5:afc6:2          2022-09-21T20:50:25Z  2022-09-21T20:50:28Z
06e82997b6d987  lively-water-3206       started sjc     rails-machine-workers:  fdaa:0:a177:a7b:b389:6b3:a716:2           2022-09-21T20:46:25Z  2022-09-21T20:46:27Z

Don't forget to include the app name of the workers, not the currently running application.

ActiveJob Adapter

Well that was fun kicking off a task from the Rails console! Since we love monolith and ActiveJob, let's set this up so that when we enqueue a job, a machine is forked and the job is performed.

# frozen_string_literal: true

module ActiveJob
  module QueueAdapters
    # == Fly Machine adapter for Active Job
    #
    # Boot VMs in 500ms, run a Rails background job, and shuts it down when it's all done.
    #
    #   Rails.application.config.active_job.queue_adapter = :fly_machine
    class FlyMachineAdapter
      def enqueue(job) # :nodoc:
        Fly.app.machine.fork init: {
          cmd: [ "/app/bin/rails", "runner", "ActiveJob::Base.deserialize(#{job.serialize}).run"]
        }
      end

      def enqueue_at(*) # :nodoc:
        raise NotImplementedError, "Does not yet support queueing background jobs in the future"
      end
    end
  end
end

It's a pretty simple adapter. As you can see all it does it deserializes the job and sets up a `rails runner` command that deserializes it and runs it.

Additional resources

Links to documentation that can help better explain some of the APIs and tools that get this working.

• Machines Docs
• Machines Announcement
• Remote IDE Machines