A lot of the programming that I've been exposed to has been scripting/analysis that does not necessarily need structure like, e.g. an app would. Starting with some raw data, I typically end up with a separate directory tree for each phase of preprocessing and quality control, then separate trees for each hypothesis or subset of data. It all ends up as one unholy mess of partially copied scripts, and redundant data. Could anyone recommend either a set of guidelines, or a project which showcases a sane way to layout a pipeline or analysis? Anything using R, python/numpy, or whatever language/framework you deem appropriate.
Organizing a project that can handle different data, store results and parameters of many different runs, etc, can make or break an analysis. I've searched around a bit online, but perhaps I'm not hitting the relevant terms.
