REST Hooks - Stop the polling madness (opens in new tab)

Am I the only one who has trouble sifting through all this marketing? Even "read more" link leads to an article where half of the text promises what will be covered and another half talks about some subscriptions and how they are managed (they might have been mentioned in the first half, I agree - and I also read text only twice).

If someone has a better understanding of how this works, I would appreciate a TL;DR version for programmers (which are probably the target audience?). Thanks!

> REST Hooks itself is not a specification, it is a collection of patterns that treat webhooks like subscriptions.

Bummer. No matter if you like it or not, a collection of patterns with a name _is_ a specification, just possibly a poorly defined one. See the confusion in this thread. If it were a link to a spec, nobody would be confused.

> Skip the pedantic arguments about standards and implementation details

This reads to me as "everyone is going to have a slightly incompatible implementation. One library wont be good enough, I'll need to write a new one per site that uses this."

Furthermore, what about PubSubHubbub?

Finally, polling is great: Ive found few situations where it doesn't work well, people just tend to only do the most basic of implementations and blame polling. See http://roy.gbiv.com/untangled/2008/paper-tigers-and-hidden-d... for a really interesting example.

EDIT: numbers were real, added a supporting link

This isn't impressive. [MQTT][1] is a much better protocol, and an open standard. It's a pub/sub that runs over TCP with variants that run over UDP & TLS. There's only a 2+ byte overhead, so it's very light. It's ideal for mobile devices because it has a QoS system built-in, so the developer doesn't have to think through the problem of missing messages where cell service is sub-optimal. MQTT is even resource-based, so it fits well with REST. I wish more people would start using MQTT instead of these crazy hacks like REST Hooks.

[1]: http://mqtt.org/

If for some reason you can't or don't want to implement webhooks, at least make sure you the GET endpoint for any object has a query param that supports fetching the most recently updated or created objects and supports pagination.

It sounds trivial, but you'd be surprised how many APIs don't support one or both of those features. When you're writing an API it might seem unnecessary to start (after all, who could ever have 1000s of <object>?), but if someone ends up polling your API frequently, having those two features can reduce a lot of unnecessary load for both you and the poller. And, of course, make sure you have an index on the created and/or updated dimensions.

That said, webhooks are terrific. Few things to consider when implementing them:

- Think carefully about the payload you send to the webhook. It's usually a good idea to send some related objects/data because many times when someone gets a webhook payload, that'll trigger calls to your API to get related information you could've reasonably sent them in the initial payload.

- You'll likely want to some way to keep track of errors so if an endpoint starts returning 404s or 500s you have a way to programmatically discard it after X failed attempts.

- In your docs, give sample, "real world" payloads developers can test against. It saves times over creating a RequestBin, pushing there, copying, cURLing, etc. (Remember, you can't set up a webhook to localhost.)

- A nice to have is some sort of retry capability with an exponential back-off. Servers go offline and if they get pushed webhook data then, those messages are lost. You could say, "tough, it's the consumer's responsibility," but if having all the data is important, most people will resort to polling. (Somewhat related, you'd be surprised how often the APIs of some larger SaaS companies are "offline" -- e.g. returning 503 --, so these things do happen.)

It is awfully difficult to get a technical TL;DR version from a website, whose target audience is programmers. I think explaining what this does concisely and technically, preferably on the front page with code examples etc would be really useful.

We also wrote a quick drop-in Django application [0] (that plugs right into TastyPie if you have an API built on it) . We also have a sample CRM with a live API running [1]. Our friends at NetEngine wrote a sample Rails app [2], and our very own James wrote an awesome Node.js Sails app [3].

[0] https://github.com/zapier/django-rest-hooks

[1] http://demo.resthooks.org/

[2] https://github.com/net-engine/resthooks

[3] https://github.com/zapier/node-resthooksdemo

Oh neat! We're launching resthooks.org today with several partners to promote an existing but underused real-time API pattern.

Here are more resources:

Intro: https://zapier.com/engineering/introducing-resthooksorg/

Why: http://resthooks.org/#why

Demo: http://demo.resthooks.org/

I hate to be THAT guy but someone was going to say it so it may as well be me :)

This is a collection of patterns, right? Well why not make this really REST and rather than list a bunch of URL templates provide REL types for each of these like so:

REL subscriptions-list -> GET subscription-create -> POST subscription-read -> GET subscription-update -> PUT subscription-destroy -> DELETE

Now it doesn't matter what the URL structure is, I can pull the <link> elements from the page and be TOLD the URL, rather than follow a URL template. That way, this doesnt rely on out of band knowledge i.e this web page and its (poor) description.

Has anyone use PubSubHubbub ( http://en.wikipedia.org/wiki/PubSubHubbub ) for this pattern in the past? For straight subscriptions, it's always been compelling to think of the use cases being covered by PSHB and RSS.

This sounds like webhooks, but they have a set payload, so that the server that is sent the webhook just calls the rest endpoint it is given in the payload. Interesting idea, but it does sound like a regular webhook to me. That flow is the same way I use the Stripe webhooks for instance. I use the event id and make a call to Stripe directly, so that I can make sure the event is real.

I don't really understand what this is. Is this an attempt at a formalization of webhooks + JSON?

The way the info is presented bugs me. The first "case study" is Zapier. Later, in the "Learn how..." set of links, Zapier is listed as one of many. Look to the footer, and the Github link, and it's of course evident it's a Zapier project.

This project (er hmm, "initiative") is core to Zapier's business. If every service out there had a hook atop their service, it'd make things a lot easier for Zapier. That's cool. What bugs me is the feeling that Zapier's branding in the whole thing is less than transparent.

It seems that open source projects are getting more and more marketing driven, and the way this "initiative" is packaged is a sign of things to come.

Curious if anybody can answer a question for me. I've recently started building some apps with MeteorJS, and am having loads of fun with it. At first I'd been a bit suspicious of its magic pixie dust -- seemed too good to be true -- but so far it's done everything it promised, and has allowed me to focus on developing the parts of the app I care about. After checking briefly to see that my data syncing was genuinely low-latency and low-bandwith (which under every test I've subjected it to, it appears to be), I've pretty much forgotten about this level of engineering and have focused on bigger-picture stuff.

Which brings me to the question: should I care about something like REST Hooks? Is it a mistake to become complacent and assume that the lower-level infrastructure will Just Work? Or can MeteorJS (and presumably other high-level frameworks) be trusted to handle this kind of stuff in a way that makes it safe for me to forget about it?

Just curious what the esteemed HN denizens think about this, as I'm sure there are some strong and reasonably well-informed opinions out there...

The docs talk about "event types" like contact.update which do not exist in HTTP and require knowing the semantics of a particular group of resources. To truly be RESTful, I would want "please notify me of any PUT, PATCH, or DELETE to this list of URLs, or POST creating a URL that matches this pattern." And if you're going to include a copy of the resource (I don't know what else a payload would be) you need the media type the subscriber expects as if you were doing content negotiation for a GET request from them. Or maybe the payload could just be the current result of a HEAD request so the subscriber can invalidate their caches, and they can send the appropriate GET request(s) when and if they care to.

There should be some standard link relation like <link rel="subscription"> so clients have some hint that this is available and where to request it.

I'd also want some way to manage the freshness/load tradeoff, like "please notify me within one hour but no more than every five minutes".

Doesn't this assume that the API will only be used from a module living under a URL?

Most REST apis these days seem to be consumed form client side.

In addition to webhooks, using a websocket connection to listen for streaming updates is a great way to eliminate polling.

Give a man a hammer, everything looks like a nail. REST has become a bit of a hammer. The reason a lot of RESTful designed software I've seen has such poor performance is because REST is used like a hammer... everywhere! And whole bunch of places it's not needed and it's not ideal.

I hate to be THAT guy but someone was going to say it so it may as well be me :)

This is a collection of patterns, right? Well why not make this really REST and rather than list a bunch of URL templates provide REL types for each of these like so:

REL subscriptions-list

Are there patterns for client side JavaScript to be notified of changes to REST resources? I don't believe this particular pattern applies if I understand correctly as it needs a URL to notify.

I don't agree with this part:

> In other words, if everyone implemented REST Hooks, server load for both the sender and receiver could be reduced by 66x.

No, the number of requests could be reduced by a factor of 66. I'm not saying that's not impressive, I'm saying that the polling requests that ended up resulting in no action are cheaper than actionable requests, so, server load will go down by much less than a factor of 66x. The amount of work is the same, just busywork is less.

While the paradigm of having a webhook makes more sense, I personally hate working with them. I've yet to see a graceful way of handling local development that doesn't involve localtunnel (or some alternative) and constantly having to un-register and re-register endpoints every time localtunnel gives you a new hostname. I understand the point and would prefer working with them if there was just a good way to handle local development.

Powerful idea, should be made a specification. Simple enough too - what they describe is almost trivial to implement in /* insert your web framework of choice */. You wouldn't even need a library. Handle your HTTP verbs, store subscriber state, and call them out on events. It's nice.

Here's another (work in progress) spec for managing webhook/HTTP callback subscriptions: https://github.com/progrium/http-subscriptions/blob/master/S...

Why is this a thing? Your API interfaces with a message queue; your end client uses all the libraries and patterns built up over the years to consume from the queue. That doesn't sound all startup-y like "REST Hooks", but the pattern has been around for a long time.

I just want to point out that having read that entire page, I still have absolutely no idea what this is, how it works, etc.

I have a solution for polling. Its called websockets. Even more, there's a well-supported library called socket.io that transparently handles it for all browsers.

After reading through all the comments here I still can't figure out what this is. Sorry.

Isn't this like SUP?

http://blog.friendfeed.com/2008/08/simple-update-protocol-fe...

FWIW http://barelyenough.org/blog/2009/05/push-or-pull/ case closed

> On average, 98.5% of polls are wasted

Erm. Is there a source for that?

TL;DR - REST Hooks is an API pattern where the API provides an endpoint from which webhooks can be programmatically setup and removed.

This sounds pretty much like the polyfill Socket.IO/SockJS do to create a socket atop polling-based transports.

I have no idea what this is.

Someone should show this to Facebook.... Just saying...