That's not it. If it's made by a plant it's more likely your body has enzymes and pathways to metabolize the item (the chemical structure is "familiar" or "compatible" with yours if you will). This means it doesn't build up and cause low toxicity over time. i.e. it's pretty obvious if it's poisonous.

Not so with lab created stuff - your body may not have the ability to do anything with it, so it either sits and causes slow damage, or gets excreted very slowly, and can cause hard to detect damage.

For example acetone is not very toxic because the body can metabolize it.