> This way if you query Elasticsearch to find all API errors in the last hour, it can find the answer by looking at a single index, increasing efficiency.
This is a really good way to know if third party APIs are having problems. Staying up to date with all those APIs they support must take up significant amount of engineering effort. Many APIs are just second-class citizens for their product owners. Bugs are introduced, changes are made without announcements, and even if there are announcements when you're dealing with so many different APIs it's hard work keeping track of them all and making changes in your app to keep it running especially when APIs are turned off, or schema changes are happening. This seems to be the hard problem IFTTT is solving, integrating into APIs.
I'd shy away from starting a project that involves so many other companies APIs just because of how hard of a problem that is to manage, but IFTTT is doing a great job here.
As Jonathan mentioned, we made this decision around 9 months back and at that time Kinesis wasn't as mature and had less flexibility around retention period etc.
Kafka is very reliable (as I had seen it handling billions of events a day at LinkedIn) and has a huge open-source community around it. At IFTTT, we always prefer to use and contribute to open source ( http://engineering.ifttt.com/oss/2015/07/23/open-source/ ).
Is this a typo? Did they mean 'a few'?
We don't do it for the production database because we don't need it in realtime.
Can a moderator such as dang fix this please?
Medium's republishing API has already become a problem for original sources on HN.
p.s. Comments in the threads are an unreliable way to reach us. There are too many for us to see them all. The reliable way is to email hn@ycombinator.com.
