As someone who also maintains a few (much, much smaller) OSS projects on GitHub, I really understand the 'no' mentality. It's often much harder to say no, but usually I try to put it in a positive way (yes, this is a worthwhile idea, yes, it looks like it could help in this situation, but no, I won't be merging it because I don't think most of the project's users would benefit from its inclusion).