Someone crafty can watch your screen and wait until you type in a sensitive password (like root or 1Password) and decide to take over at that point.
The best part? The person with physical presence can lock the remote user out (you): https://www.youtube.com/watch?v=wbLYKEQk_mM
I reported this to Apple more than 90 days ago. They said it was intended behaviour.
I cannot remote into my machines safely since I discovered this.
Without exaggeration, my wife one day messaged me and said, "your computer is moving," because she could see I was logged into my iMac at home from the office and using it.
Another day a colleague was working late at the office and I had logged into my office machine from home. He messaged me saying, "Did you leave your computer on on purpose?" He knows I always lock my computer because I give him heck for not locking his.
What did Apple Security Research recommend when they closed out the issue? They said I should use Apple Remote Desktop. That app is on the Mac App Store and averages 2.1/5. It retails for $99.99.
Maybe I'm off here. Did you expect someone next to your machine to see everything you do and be able to take control when you remote into it?