Response for this incident went by the book, as described in Brent's talk above. Incident Management programs like these ensure that incidents can be resolved while also minimizing stress and chaos for engineers and other responders.
PagerDuty has a good Incident Responder and Incident Commander training courses, if you are interested in setting up a program similar to Slack's:
- https://response.pagerduty.com/training/courses/incident_res...
- https://response.pagerduty.com/training/incident_commander/
Fun fact: Brent Chapman is also known for creating the `majordomo` mailing list manager from the early 90s
usually, keep your head cool and focus on the problem at hand.
This is also the reason why engineering/operations should be seperate from customer communications. The people who are fixing the issue should not also be the ones doing the communication with the outside world.
Talk about being on center stage..
It would be awesome to have essentially the same experience as using Slack, but also a server running on our lab's hardware that we can reboot/debug ourselves.
We aren't even particularly dependent on slack, but the convenience of a few basic integrations like google calendar make moving to Discord or anything else a non starter.
The integrations, and chat itself needs an open protocol, otherwise it's just going to be more and more of slack in the future.
I think I'll probably try again once most people have switched to a server written in Go or Rust.
no thanks. I would never install zoom on any of my computers voluntarily.
As it stands, while I'm not fully blocked from being productive, I am blocked for a lot of work that needs to get done.
Coupled with COVID work-from-home constraints, there's no practical way right now to get code reviewed for merge into the main codebase 'round these parts.
Do you tie core biz/dev workflows to Github (or any other service)?
However, the iPhone version is working just fine.
They are responding on twitter that there's something up: https://twitter.com/SlackHQ/status/1496133311558737923
* I want to find someone by name => ctrl+K
* I want to search with sane keywords => search, "from:me" "to:<channel name>" work.
* I want to remember someone I talked to recently => people you chat with show up in the list on the left
* I want to keep com channels organized as the ground changes => easily rename channels, favorites UI works well, channel grouping works right.
* I want to give a public emotional response to someone's statement => reactions
* I want to continue discussion of a point someone made, which may not be relevant to everyone in the channel => threads
* I want to edit a typo I just made => hit up on keyboard
It's got that quality Factorio has, where you can let yourself imagine it is the ideal product, and start expecting features you need to be there, rather than not bothering to explore because you think they won't be.
It is AMAZINGLY good at solving actual user stories around communication. I have lots of respect for their PMs.
EDIT: See comments below for instructions on re-enabling Markdown mode.
̶T̶h̶e̶ ̶o̶n̶l̶y̶ ̶d̶o̶w̶n̶s̶i̶d̶e̶ ̶i̶s̶ ̶t̶h̶a̶t̶ ̶m̶o̶d̶e̶ ̶d̶o̶e̶s̶n̶'̶t̶ ̶s̶u̶p̶p̶o̶r̶t̶ ̶m̶a̶r̶k̶d̶o̶w̶n̶ ̶s̶t̶y̶l̶e̶ ̶U̶R̶L̶s̶ ̶l̶i̶k̶e̶:̶ ̶ ̶[̶e̶x̶a̶m̶p̶l̶e̶ ̶s̶i̶t̶e̶!̶]̶(̶h̶t̶t̶p̶s̶:̶/̶/̶e̶x̶a̶m̶p̶l̶e̶.̶c̶o̶m̶)̶ Never mind, it looks like Slack has fixed this!
I have yet to hear a good argument for not having a little button to enable raw Markdown mode.
Format messages with markup
The text formatting toolbar won’t show in the composer.* I used to be able to type a message containing "@channel" or "@here" and just hit Enter without looking, and 100% of the time it would be parsed correctly. Now it fails to detect the "@channel" or "@here" often enough that I have to look to see if it failed, then go back and retype that part of the message until it lights up. Once a week or so, I see a message posted by someone else that contains "@channel" in black, and they expected the message to notify everyone but it didn't.
* Recently Slack pushed everyone to switch from usernames to full names with spaces, and eliminated the entire concept of unique usernames. There were many annoying consequences to this, one of which is that when I type "@name", now I have to look to see if it highlighted, or look to interact with the drop-down menu, instead of just typing and knowing it will work.
* Searching also got worse for the same reason. When I type "@name" in the search bar, it NEVER lights up. For 90% of searches this means I can no longer type in a search query and just press Enter. I always have to look through the drop-down menu and either mouse-click or press down-arrow repeatedly to get to the thing I want.
That's your Slack settings my dude, blame your admins.
WTH? Why not?
> A configuration change inadvertently lead to a sudden increase in activity on our database infrastructure. Due to this increased activity, the affected databases failed to serve incoming requests to connect to Slack. We introduced tighter rate limits on connection requests to reduce the load on the system. This meant that some people could not access Slack at all, but also that Slack would continue working for those who were already connected.
> Once the system had stabilized, we began lifting these rate limits to enable more connections to Slack. However, we moved too quickly and the increased activity affected the system again. We reinstated the rate limits and redirected some traffic to the database replicas to relieve the demand on our primary databases.
I wonder if their bottleneck is at vitess vttablet or at mysql.
The degraded availability thing is because I run multiple servers for latency, normally an IRC outage is super clear, except with netsplits where IRC servers themselves unlink from each other. (I average about 3 of those a year and they last about 30s roughly).
It's not video though, so that doesn't answer your question thoroughly.
It's pretty confusing.
I guess it was down so they could do this crappy update.
I'm putting my money on an expired certificate in certain client app versions.
Every single person on my team and at my company.
Something’s gone awry and we’re having trouble loading your workspace.
Sorry we can’t be more specific – this is one of those cases where we don’t know what’s gone wrong either. A restart of Slack might help, and you can always contact us.
In all seriousness, since I'm sure a lot of people don't know this, use SHIFT-CTRL-r. That instructs the browser to reload everything, ignoring cached resources. This can occasionally "restart" a web page that even a reload or loading the URL in the URL bar again is not sufficient for. Slack is one of those things I've seen it work on.
However, SHIFT-CTRL-r isn't enough now.
most people IME use the slack app, in which case "restart" makes sense.
It's a really refreshing feeling to know, that channel of communication is not available for anyone in my company.
it's like a veil has been lifted off my eyes