HireFire - The Heroku Worker Manager (opens in new tab)

(github.com)

44 pointsmeskyanichi15y ago18 comments

18 comments

14 comments · 7 top-level

KarlFreeman15y ago· 2 in thread

Very handy. Was looking at using http://www.simpleworker.com/ as an alternative to the uncost effective Heroku DJ's but hopefully HireFire will sway me back to using Heroku.

meskyanichiOP15y ago

Hey. I was actually also using SimpleWorker prior to HireFire, but I find that setting it up is a bit hard / takes long since you have to set isolated code / credentials / etc to their servers in order to process background jobs. Also, the jobs aren't guaranteed to run immediately after they are queued, may take up to 10 minutes on the cheapest plan. When deployed to Heroku it's easy since Heroku will just use the same compiled slug it uses for Dyno's, no additional code/configuration required, it just works. I also found that on SimpleWorker a job that took only 3 seconds on Heroku, would run for 30-90 seconds on SimpleWorker (no idea why).

That said. There are some advantages when using SimpleWorker such as the web interface, scheduling recurring jobs, and other useful options. Especially when you aren't running on Heroku, and memory is an issue if you have a small VPS. In that case, offloading background jobs to SimpleWorker is ideal. But I find that HireFire will make Heroku Workers cheaper than SimpleWorker due to the fact that jobs appear to take (a lot) longer to finish on SimpleWorker than on Heroku (probably due to the start up time? I'm not sure why).

KarlFreeman15y ago

Couldn't agree more. The setup time of SimpleWorker is alot ( especially when dealing with specific gems ) and the fact that for DB work you have to create a separate non concurrent connection to the DB per job makes those times you talk about a reality. Ok, you could use the 'expensive jobs' but Heroku's DJ's win hands down in all tests I've done. Plus who doesn't like the handle_asynchronously :function.

1 more reply

freerobby15y ago· 2 in thread

Nice work, glad to see heroku API gems getting some love.

I wrote a similar gem back in January and have had good luck with it: https://github.com/freerobby/middle_management

It's fully configurable via environment variables, so you can make changes without redeploying. It also lets you specify a minimum number of workers (even if there are zero jobs) so that you don't have to wait for workers to spin up if you don't want to.

meskyanichiOP15y ago

Thanks. I like your approach of not having to deploy to change the min/max value of workers. Though it's hard to implement (I think) if you're working with a worker/ratio configuration like HireFire. Currently HireFire also supports the Mongoid ODM. I'm looking in to implementing the Resque worker as well so people can use HireFire with Resque with Redis on Heroku as well.

Thanks for your input!

freerobby15y ago

If I could fork your code and get it working with environment variables, would you be interested in pulling that in? (Not sure if you made a deliberate decision to avoid that).

MiddleManagement works great for my needs, but if these libraries serve the same use case(s), I'd rather converge towards one library than have things spread out.

One other thing you may want to add is a retry loop for the heroku API calls. Every now and then, the heroku gem will throw a RestClient::ServiceUnavailable: 503 Service Unavailable exception. In the worst case, I'm not sure if this is best handled in the gem or at the application layer; however it's definitely worth retrying a couple times if you can avoid the blowup completely. I got some feedback from the Boston.rb group on handling this if you're interested: http://groups.google.com/group/boston-rubygroup/browse_threa...

1 more reply

nestlequ1k15y ago· 1 in thread

Looks great. Hoping to see resque support on it soon.

One thing this really shows the need for is a real api on heroku. Having to hardcode your username/pass into env variables is pretty ghetto. Hopefully heroku devs are watching and listening. Oauth plz??

meskyanichiOP15y ago

Thanks! Looking at Resque and will probably be implementing it fairly soon.

Ha! And yeah, the ENV variables. :)

gerrit15y ago· 1 in thread

For the low-traffic scenario there's also an older, much simpler solution: https://github.com/mtravers/cheepnis

meskyanichiOP15y ago

Interesting and light-weight approach. Though, does it support other ORM's or ODM's like Mongoid? Currently HireFire supports ActiveRecord and Mongoid with Delayed Job. Planning to add Resque with Redis support fairly soon.

anulman15y ago

I was looking for something like this months ago. Thanks for building, and I'm looking forward to (at least trying to) contribute!

mrdigsus15y ago

Looks extremely cool. Can't wait to kick the tires.

ddemchuk15y ago· 1 in thread

Does anyone know of gems like this that will work on standard VPS servers? I'm launching something that completely depends on Delayed Job for processing data and would love an interface to manage queues and number of workers spun up at a given time. Also, visually watching the queue (a la resque) would be amazing.

The best I've found so far is: http://trevorturk.com/2011/01/21/delayed_job_admin-a-simple-...

meskyanichiOP15y ago

Hey. The gem actually has a local environment option which makes it work on a VPS similar to how it works on Heroku. You could try that and see how it works.

https://github.com/meskyanichi/hirefire/blob/develop/lib/hir...

Also, as far as I know that DJ Admin Interface is the only one for DJ that currently exists. If you want to use Resque, I'll be implementing Resque for HireFire soon.

j / k navigate · click thread line to collapse

18 comments

14 comments · 7 top-level

KarlFreeman15y ago· 2 in thread

Very handy. Was looking at using http://www.simpleworker.com/ as an alternative to the uncost effective Heroku DJ's but hopefully HireFire will sway me back to using Heroku.

meskyanichiOP15y ago

KarlFreeman15y ago

1 more reply

freerobby15y ago· 2 in thread

Nice work, glad to see heroku API gems getting some love.

I wrote a similar gem back in January and have had good luck with it: https://github.com/freerobby/middle_management

meskyanichiOP15y ago

Thanks for your input!

freerobby15y ago

If I could fork your code and get it working with environment variables, would you be interested in pulling that in? (Not sure if you made a deliberate decision to avoid that).

MiddleManagement works great for my needs, but if these libraries serve the same use case(s), I'd rather converge towards one library than have things spread out.

1 more reply

nestlequ1k15y ago· 1 in thread

Looks great. Hoping to see resque support on it soon.

meskyanichiOP15y ago

Thanks! Looking at Resque and will probably be implementing it fairly soon.

Ha! And yeah, the ENV variables. :)

gerrit15y ago· 1 in thread

For the low-traffic scenario there's also an older, much simpler solution: https://github.com/mtravers/cheepnis

meskyanichiOP15y ago

anulman15y ago

I was looking for something like this months ago. Thanks for building, and I'm looking forward to (at least trying to) contribute!

mrdigsus15y ago

Looks extremely cool. Can't wait to kick the tires.

ddemchuk15y ago· 1 in thread

The best I've found so far is: http://trevorturk.com/2011/01/21/delayed_job_admin-a-simple-...

meskyanichiOP15y ago

Hey. The gem actually has a local environment option which makes it work on a VPS similar to how it works on Heroku. You could try that and see how it works.

https://github.com/meskyanichi/hirefire/blob/develop/lib/hir...

Also, as far as I know that DJ Admin Interface is the only one for DJ that currently exists. If you want to use Resque, I'll be implementing Resque for HireFire soon.

j / k navigate · click thread line to collapse