Yeah. There is. Trolling vs. the intended exercise of this "negative right"? : "Its the difference between using a feather and using a chicken." We know it when we see it.
I expect if given the choice, they'd be intelligent enough to choose the side of progress, rather than of hindrance.
I suggest that only the inventor has the right not to work out their invention but any buyer of the patent afterwards should be forced to build that invention.
No single behavior makes a patent holder a troll, but a combination of behaviors do.
Good: A patent holder partners with licensees to produce their invention. Bad: A patent holder waits until somebody produces their invention to sue them.
Good: A patent holder tries to get their invention produced. Bad: A patent holder takes no steps to get their invention produced.
Good: A patent holder sues for infringement that does not discourage going to court. Bad: A patent holder threatens to sue under conditions that make going to court pointless.
Good: A patent holder is the inventor. Suspicious: A patent holder bought the patent from the inventor.
Good: A patent holder takes the necessary steps to make their patent known. Bad: A patent holder exploits the system to hide their patent as much as possible (submarine patents)
Good: A patent represents a suitably complex/worthy invention with a clear and narrow definition. Bad: A patent is overly broad and claims a trivial invention.
A good patent/patent holder will never fulfill all bad criteria listed above. However a patent troll will always fill all the bad criteria for obvious reasons.