I think that's fair. After all, poach has literal meanings that apply to acts that are clearly wrong or illegal. In fact, if I look at the dictionary, you see words like trespass, steal, etc.
You do hear poach used colloquially or even somewhat jokingly when it's not intended to imply anything nefarious but the word does imply something underhanded.
