Are these trolls with an existing patent you're considering infringing?
Or are these trolls gremlins who will somehow come out of the woodwork after you get your own thing going and try to patent it?
Patent trolls survive by making money. If you don't have any money (and aren't making any), they probably won't bother with you. If you are making money, they will want something you can afford or raise, because otherwise, they won't make money that way either.
In many ways, fighting with a patent troll is a sign of success more than anything else. It means you have the money to fight over. Don't be afraid of success.