I disagree entirely. It's an upside. They get to benefit from PKI without even understanding anything. A person's address gets them the actual person.
ToxMe requires trusting the ToxMe identity provider, and is an obvious point of attack. And we'd no doubt see fake addresses that resemble other peoples, and other such nonsense.
There's minimising the inconvenience (with ideas like the QR code feature they have), and there's plain giving up security for minimal gained convenience, which we should just avoid.