IMO, DHH (and others) argue too hard against core OO principles (like SRP and LoD) that have the possibility of helping out when the domain model becomes more complex. "Just use Rails" is fine when spinning up, but as the app grows, it becomes increasingly less pleasant. Coupling increases in the name of less code.
Ideas like hexagonal architecture look nice, but I'm left wondering how to implement them when the framework insists on infecting core domain classes (AR::Base).
I say "higher-level" tests to refer to integration, system, and acceptance tests, because I don't have an opinion of where to draw the line for FactoryGirl. Is there a reason you would consider FactoryGirl for acceptance tests, but not integration or system tests? Just curious, because I might learn something.