If you want to be sure to keep FactoryGirl out of unit tests but still have the benefit of using it in higher-level tests, you can use a different test/spec helper. I tend to do "spec_helper_lite", which sets up the bare minimum for unit tests (no FactoryGirl), and "spec_helper", which includes FactoryGirl for higher-level tests.
I say "higher-level" tests to refer to integration, system, and acceptance tests, because I don't have an opinion of where to draw the line for FactoryGirl. Is there a reason you would consider FactoryGirl for acceptance tests, but not integration or system tests? Just curious, because I might learn something.
