3a is identifying all systems that
might have been touched and determining what logs (if any) are even available for them. 3b is identifying who the administrators are, contacting them, and getting them all to furnish logs. Sometimes they have weird requirements like only being able to query a backend by IP or username, so you have to precompile lists to provide to them. Time spent on this back-and-forth is time
not spent on investigation.
Sometimes you get deep into one thread, then you discover intruders came from a new IP you didn't notice the first time around. So you start the process over for each new discovery. Early on in investigations, every new discovery presents more unknowns, but this trend reverses itself as you work through it.
Attackers are incentivized to make things as confusing and opaque as possible. Maybe you recover a malware sample-- reverse-engineering deliberate obfuscation efforts takes time. Sometimes it's written in languages like Go that you don't have experts on-hand to understand (ChatGPT might be a game-changer here though; have not tried yet). Like ticks, sometimes the part you find is only the part you see (or a diversion) and the rest of it is still festering beneath the surface.
Attackers use pools of IPs, and spread recon/exploitation across them so the same IP that found a way in isn't necessarily the IP that exfiltrated anything. Same with implicated user accounts. Attribution takes time and is error-prone. You have to maintain opsec through the whole thing; everyone outside of your team will talk and sometimes the threat was internal all along. Sometimes the threat was incompetence that involved employees will try to divert attention from. Sometimes it's sabotage resulting from a bad performance review. Usually the threat is external, but they may still have agents running on internal systems that we don't know to look for.
When shit's going down, everybody wants answers within an hour of discovery, which is unrealistic. Like Twitter during a natural disaster or terrorist attack, it usually takes most of the first day to divine what the actual problem is from all the speculation and hysteria. If I do my job right, it does take months, but cutting every corner and being sloppy I can run a passable investigation in a matter of weeks. With the exception of SolarWinds, executive interest usually drops off quickly after the first week.
It's like someone set off a bomb, and your job is to account for the integrity of every single thing hit by every single piece of shrapnel. Done right, it takes time.
