Disk space is preposterously cheap, and even high-volume, Internet-scale text logs are small and low-bandwidth[1].
The problem isn't that disks fill up, but, rather, that logs are ever written somewhere that, if full, affects anything but the logging. The solution of a separate filesystem for logs has been around perhaps longer than the OP himself but only makes sense for a few, standalone servers.
Otherwise, the solution of remote syslog has been around for almost as long. If your logs are critical, you could even use two (for twice the price). Even the days of questionable reliability and small message size[2] limits are pretty long gone with syslog-ng.
This kind of thing, otherwise tedious minutae, is second nature to sysadmins. Hire one.
[1] It's the disk bandwidth or throughput which is still expensive
[2] That's on the order of 500 characters, old text pager lengths, not the newfangled 140 character sms/twitter cheapness. Hey, you kids get off my lawn!
Not to mention it's pretty much automatic these days. It's not like it's something you've got to go out of your way to configure.
I disagree, since, otherwise, there's nothing rotary. If one has effectively infinite disk space, nothing ever gets "rotated out," for example.
Certainly, log segmentation provides benefits beyond being a prerequisite to rotation, but, as you point out, it's a common default configuration. An inbuilt feature of syslog-ng is log file naming based on date, obviating any post-processing.
(the points re. backup and monitoring do still apply, but at least it's only for your app and not the whole server)
Chef vs Puppet is like Emacs vs Vi. I use Puppet because it was the only game in town when I started, but I know a bunch of the Chef guys (hi Adam!) and they know whats up. The tools have different approaches but are rapidly converging on the same feature set, pick a simple role (say a webserver) and do the entire config in both tools and you'll know almost instantly which one matches your mindset better. I like to think Chef is a bit closer to the shell, things happen in order and you can rely on that, whereas Puppet is a bit more abstract and doesn't require you to micro-manage it as much. The golden truth is somewhere in the middle of this approach, Puppet for example just added some features to make ordering more consistent since people have really struggled with that in the past.
Chef is pure Ruby, so if you know Ruby you'll have a nice head-start. Puppet has the Puppet DSL, which if you are a sys-admin jives well with config style syntaxes. Puppet just released a pure Ruby DSL as well but it doesn't have 100% feature parity with the Puppet Language.
Provisioning and Bootstrapping machines is still a black art. I use a ruby script that sets some environment variables then launches a first run of Puppet, after that Puppet takes over and manages. There is no silver bullet (yet) for getting a machine into your Puppet/Chef cluster, though there are a lot of innovative approaches that look very promising. Depending on your environment (EC2 vs say Slicehost) I'm sure you can find a blog post that will get you going in the right direction. Here is mine: http://blog.onehub.com/posts/coordinating-the-onehub-cloud . This is already a bit out of date, we're migrating from iClassify to the officially blessed Puppet Dashboard (http://www.puppetlabs.com/puppet/related-projects/dashboard/) we just need to put some more time in to finish.
Testing sucks. You will need to come up with some repeatable environment you can run your configuration against. I use VMWare and just continuously restore to a snapshot. There has been some good development here, especially with Vagrant (http://vagrantup.com/). This lets you write a shell script (or rake task in my case) that boots an instance, runs the config and then pauses for you to inspect. From here you can push another button and 'retry' the whole thing or destroy the VM and start over. It is annoying to get this going and you'll be tempted to just start on one of your machines but that is a bad idea. This tool is going to run as root, and after the first day it is going to run with minimal supervision. You need to be sure its doing what you expect.
At the risk of appearing to build a strawman, I'm purposefully taking this quote slightly out of context, because this strikes me as a far more general belief, which is used to justify these CM systems as a solution. Herein lies the danger of circular reasoning or self-imposed problem.
Since I'm an open-minded sysadmin, I always keep an eye on the likes of puppet[1], but I have continued to reject them. Much of it has to do with philosophy: use what I can, that's tried and true. Very nearly all the right pieces for provisioning and configuration management already exist[2].
Make sure your machine is 100% described by your tool.
This, sadly, smacks of perfectionism, which is known as the enemy of Good Enough, yet I agree that these tools demand it.
The vast majority of this kind of description is already done by the OS via the package system and init/upstart[3]. To duplicate this kind of description with a separate tool is, to me, incomprehensible.
What's more, for at least the past 5 years, brand name server hardware[4] has had, in the BMC, without any special add-on cards or option keys, enough IPMI-over-LAN support that one can, over the same physical ethernet as one of the regular network interfaces, set the next boot to be PXE and trigger a reset. From that point, a fully functioning server can be up in 5-10 minutes[5].
With those kinds of provisioning times, why would I want to bother with something that requires the extra step of "black art" bootstrapping[6]? At most extreme, to make a configuration change on a running system, I'd just need to trigger the installation of a new package version on the relevant systems.
The best part of such a scheme is that I don't need to make any further customization choices, like puppet vs. chef. All the infrastructure I need (DHCP, DNS, TFTP, kickstart or debian-installer, local mirror/repo) is a Good Idea to have anyway, and it's all standard. I would expect any moderately experienced sysadmin to be able to debug all those pieces, without learning a DSL or a particular system's quirks. One also benefits from years of evolution of such tools, including "free" redundancy and pre-existing plugins for monitoring.
The only thing that's left is some kind of higher-level templating, which can be added as a wrapper around all of the standard things. So far, the only tool I've found that doesn't want to take over everything all at once, and works fine with incremental takeover/integration of the underlying tools) is Cobbler [7].
Not all problems can be solved with (custom) software.
[1] It was this month's BayLISA topic.
[2] Growing up with parents in the semiconductor industry, my exposures to Unix (and VMS, TOPS, and VM/CMS, none of which "stuck") and Lego were around the same time, so there's a deep-seated analogy there.
[3] which are, of course, configured by files which can be contained in packages, so, really, just the package system.
[4] that is, any rackmount server which can be ordered with a cable management arm. That there is such a differntiating factor belies the notion of "commodity" hardware. I find it to be merely a euphemism for "lowest common denominator" hardware.
[5] I've observed this scale easily, with no slowdown, to 30 clients against one sub-$1k boot/repo server.
[6] "Because we use cloud providers" is a weak answer, since, besides being a self-imposed problem with other unique issues, it gets remarkably expensive beyond a few dozen (if that) instances.
[7] When I last dove into it a couple years ago, it was clearly focused on kickstart and the Redhat/Fedora world, with Debian/Ubuntu barely an afterthought.
* Conceptually clean and very powerful
* Quick and easy to write new cookbooks from scratch, but the public repo of cookbooks is more useful for code examples than day-to-day use
* Documentation (the wiki) is patchy but improving
* The webui could do with some love (buggy, plus some useful tasks like executing a chef run are missing) and the CLI, knife, is a bit odd and bloated with stuff that belongs elsewhere
* The chef server stack (Merb, Solr, Couchdb) needs babysitting
* Bootstrapping is still a pain (ain't it always?)
* There are some inconsistencies and a bit of churn in the API
The Ruby DSL has some interesting quirks. Recipes are compiled in one phase and executed in a later phase. So just to make a simple log entry:
ruby_block do
block do
Chef::Log.info("File created!")
end
end
But overall it's an effective tool for an essential task.
I found chef to have a bit of a learning curve, especially when running your own chef server. I actually gave up using the server after a while and manage my servers (between 5 and 10, depending on what I'm working on, plus a bunch of one/two-off servers for client work) using chef-solo. Now that they offer a hosted server service, I might start using that, but I'm pretty content with chef-solo, roles and a minimal json config file on each server.
Chef has been a bit of a moving target during its development, so it's been hard to keep up with. This goes both for chef itself (although, presumably the API will be more stable after it hits 1.0) as well as the cookbooks. I kind of dread pulling in the changes from main cookbook repo because it's tough to know whether some change to a cookbook will negatively affect your config.
It doesn't necessarily save time up front; it takes more time to write a bug-free cookbook than do an apt-get install and edit some configs. But over the long run it makes it much easier to understand your server config. Rather than having long-forgotten edits to hidden config files, you have a solid record of exactly what your server config is.
http://github.com/crafterm/sprinkle
Might be worth checking out if your needs aren't too complex and you want to get up and running quickly with something simple.
If you've only got a few servers, or if you've got a very heterogeneous environment.. then they might not be worth it. If you've got a lot of servers, config management tools may save your life.
The tools do keep improving though, and Puppet and Chef are miles ahead of earlier tools and even every commercial tool I've yet seen. But this is still an area with a lot of opportunity for innovation.