Much respect and defineing the word professional for many.
The fact that these mechanisms are available is the reason I use such a system.
Also, if you consider any problems like this happening to a closed source vendor, you may never know it's happened. And don't tell me they don't do it as I've worked for a couple of companies that felt that burying security fuck ups was acceptable practice. It's why I don't work for them any more.
- They have to find it out first.
- Then they've to be willing to disclose the incident
- Even if, you still trust the source of the packages, the developers, etc. There's a zillion bugs that look like "just an error" which can also be "just a backdoor".
For these reasons, running mac, checking modified files, etc is ALWAYS good practice (that you seem to follow, don't get me wrong - but that's pretty rare)
"We unfortunately cannot guarantee the integrity of any packages available for installation between 19th September 2012 and 11th November 2012, or of any ports compiled from trees obtained via any means other than through svn.freebsd.org or one of its mirrors. Although we have no evidence to suggest any tampering took place and believe such interference is unlikely, we have to recommend you consider reinstalling any machine from scratch, using trusted sources."
But on a compromised machine you can't trust anybody, not even the kernel. Assuming the worst, the attacker could have gained root privileges and modified the kernel or the base tools like ls and grep. You also can't trust the log files if they're not stored off-site. The modified kernel or ls could hide the attacker's traces from you.
Thus, the only possibility to really make sure nothing is hidden from you is to (power off the machine and) attach its hard disks to a trusted computer where they're mounted and investigated.
If you power off, you can always mount the hard drive read-only and do a forensic analysis.
If you disconnect, you can still try to examine the current memory content and do a forensic analysis.
Of course what you really want is a memory dump via a trusted channel while the CPU is halted (hardware hypervisor or something like that) and then immediately power down. This is usually not supported on COTS hardware, so you have to choose the strategy that will erase the least evidence (power off, disconnect, suspend to disk, VM snapshot, whatever) depending on what you suspect the attack to be.
For all you know, the trojan might uninstall itself or otherwise delete all evidence after a short while without network connectivity to a control center ("a dead man's switch").
Or, it patches the kernel so that it is "invisible" from the inside.
What you want to do is take a snapshot of the memory (from outside the kernel, if you can) and then power down.
I assume if they had root though, they could theoretically install a rootkit in the MBR. Did you SHA256 verify the MBR too? :)
They're most definitely not amateurs, see the back catalogue for examples of 'how to do it right'
Please use passwords for your keys and allow key access only to a small set of known IP addresses.
Also do share other security techniques you're using besides the ones above.
It's much easier with git or hg, even in the single-master-repo model, even if all developers are using the CVS gateway so they don't have local copies of commit history. All such a project would need in order to protect against compromise (ignoring physical redundancy) is a backup git or hg repo, kept offline except for periodic pulls. As long as there are no warnings about upstream rebases, and you trust all visible commits, then everything's fine[1].
With SVN, as long as there's no built-in cryptographically secure way to connect one snapshot to the next when backing up SVN repos, determining the existence of a backdoor requires comparing entire backup SVN repos against the live main repo. Even if sufficient backups exist, the process is slow and requires taking the main SVN repo offline if there's any chance of a root compromise.
[1] Caveat 1: in any RCS, there could be backdoors masquerading as legitimate-looking commits if the attacker had commit access. Caveat 2: The security is subject to the cryptographic security of SHA-1.
In this FreeBSD case, if they're certain the svn repo wasn't compromised, all they have to do is validate the checkout on the two compromised machines against the svn repo.
The BSD's tend to be ultra-conservative in regard to version control systems - for example OpenBSD is still on CVS and developed their own version: http://www.opencvs.org
