There are very few stable orbits close to the lunar surface. Basically a couple of polar orbits with very specific parameters, and that's it.

The rest get so perturbed by gravitational anomalies that they fall out of orbit after a few months or years--faster than low Earth orbit where there is atmospheric drag!

https://www.lpi.usra.edu/lunar/documents/NTRS/collection2/NA...

Yeah, the moon's gravitational field is quite lumpy compared to Earth's, due to its smaller size and the way it is believed to have formed. Plus you can orbit much closer to the surface due to the lack of atmosphere, so the lumps are steeper.