Google Cloud outage brings down Layer (opens in new tab)

I automatically skip any product where I have to speak to a human at any time

Agreed, that's where I close the browser tab. I also wouldn't shop at a store where merchandise wasn't priced.

I wonder if companies trying to make money should invest their time in talking to you.

Especially when they seem to be referencing only a single region. Multi-region deployments is the most basic protection against outages when using IaaS.

Blake from Layer here: I've reviewed the updates from last night and I don't feel like the tone was out of line. We were simply trying to provide our customers with complete transparency about where the issue was and where we were in restoring service.

With that said, we do feel that Google came up in short in their responses to us over the course of the issue. We pay handsomely on a support contract to get off-hours responses and issue escalations. The responses we received were hand-wavy and vague, leaving us without sufficient data to make decisions. We have raised these concerns with our Google representative and will be working with them to tighten our partnership going forward.

We take full responsibility for this event and are working to cover the exposure. Building a system and business with resource constraints and complex distributed technologies is a long game of managing risk and trade-offs. We're human and we make bad calls along the way. We are very sorry and violated our commitments to our customers and their users. The entire Layer engineering team is head down right now working to make it right.

That was my first thought as well. Why did they need to start migrating customers to another AZ? I hope their customer's started asking that as well. The title should be "Poor Design Choices Brings Down Layer"

I imagine if one were a customer of this SaaS, it's on the customer to ask what availability to expect.

Clearly this vendor thought that their savings on their IaaS bill outweighed any operational or reputational risk they'd suffer from an outage at a lower layer (pun unintended).

I agree. If you're going to use abstracted infrastructure but you don't understand basic distributed architecture you shouldn't really be blaming your cloud provider.

Agreed, and to reply in the context of a critical comment someone further up noted (about never wanting to deal with any SaaS provider that required talking to a human) ... these are the kinds of reasons -- asking for architecture details or regulatory/security audits -- you NEED to be able to talk to humans, especially if you're operating in a regulated industry yourself or you're trying to sign an enterprise agreement with far-reaching consequences.

Hey, Blake from Layer here: we regularly undergo architecture and deployment reviews with our customers. We are fully transparent with the current deployment configuration and timelines for revisions.

Last night we lost a race to evolve our architecture and deployment ahead of a zone level issue that affected our total operations. We are working on it in earnest but there is very real complexity in operating a widely distributed real-time system.

I like how Algolia does that, https://www.algolia.com/infra (their blogposts and presentations go into much more detail)

Currently thinking of creating a similar page for getstream.io, at the moment we always explain it during sales/onboarding calls. (we replicate our data to 3 different instances across multiple AZs)

Hey, Blake from Layer here: We were not at all trying to finger-point our issues at Google, only provide our customers with up to the minute transparency on where we were with the availability issue. We have received direct feedback from our customers that they do value detailed responses even when the news isn't great.

I take full responsibility for the issues here and the team is working to remediate the exposure as quickly as possible.

I'm not sure I agree. Customers like to know why it doesn't work. If it was a physical machine, they would have said something like "the disks are broken and we are replacing them". But it is cloud and they said "Google persistent disks are currently unavailable and they are fixing it".

Blake from Layer here: I'll definitely share your feedback with the product and marketing teams. Here's my version of a summary for a developer-centric audience:

Layer provides a comprehensive platform for adding rich messaging experiences inside other products. You can think of our offering as similar to iMessage or Facebook Messenger as a library / platform. We provide native SDKs on iOS and Android that provide a high-level development experience for implementing messaging. The SDK abstracts away all the low level details of implementing a great messaging system on mobile such as content synchronization and managing a persistent connection while still providing the developer with full control over the user experience. We also offer an open source UI toolkit called Atlas that provides a reference UI implementation on iOS and Android.

In addition to our mobile offering, we also provide a Javascript SDK for browser clients as well as raw REST and WebSocket APIs for other platforms. There is also a rich set of integration APIs in the form of backend to backend REST APIs and Webhooks for tracking events within the system.

The platform is fully managed and offered as a service. Historically our availability has been very strong, but last night exposed an achilles heal and we are working quickly to remediate the issues exposed.

If you just go to layer.com, the first text you see on the page does a pretty good job of spelling out what it is. At least, it did for me. It's also more up-to-date than that comment, it would seem.

CometChat works seamlessly on web, mobile & desktop! Your users can be on any platform and communicate with each other.

Check out the demo here: https://www.cometchat.com/demo

Use https://matrix.org if you don't want to depend on the infrastructure deccisions of a third party.

PubNub. We have fantastic uptime and do over a trillion transactions a month flawlessly. We also do some cool blogging too. (my job :)

http://www.pubnub.com/

Firebase is a great platform for building chat functionality

Twilio's chat SDKs https://www.twilio.com/ip-messaging

You can try cometchat.com which is a self hosted. So no such issue.