“Conservatism” is a tent with a lot of different groups, most of whom dislike FDR for somewhat different reasons. There’s small government conservatives like you mention. There are legal conservatives who are animated more by being sticklers for the rules than opposition to government social programs per se. There’s also lots of folks, maybe the biggest group, who like the New Deal but resent FDR judicial appointees for their tendency to ignore clear Constitutional limits on the federal government one hand, while on the other hand reading in lots of restrictions on state moral and social regulation that aren’t so apparent in the Constitutional text or history.