You also can directly download APKs from Google Play using Aurora Store and compare them to the standalone APK in theory, though both points of verification are against the same entity so it only rules out MITM on a CDN etc.
Problem is, who has time to do this for every single update? How many would even do it for the initial install? Most technical sysadmins don't even verify ssh host fingerprints unless automated CA infra does it for them.
Even if someone does do this religiously, in practice I suspect they will put off valuable security patches until they can manually verify every new binary corresponds with the published source code to rule out supply chain attacks etc.
If two totally independent entities compiled and published signed binaries and their hashes matched (when signatures are stripped) then there is some automated consensus there are currently no obvious supply chain attacks in play to protect users at large who don't have the time or experience to compile and verify against the published apk by hand or manually compare fingerprints. F-droid could keep the Signal Foundation honest if they let them but instead they say "trust us, or compile your own binaries" as if no middle ground exists.
Meanwhile I can hand my wife a phone with F-Droid and Matrix and know she can update reasonably safely without any manual key verification steps by me or her. Even when the signing key of matrix.org on Google Play gets compromised the blast radius does not extend to F-droid.
The more reputable independent package managers building, signing, and distributing protocol compatible binaries the better. Makes it impractical for even a sophisticated adversary to gain control. Also lets users to have the freedom to choose an easy automated install)update path for apps that respects their privacy by not requiring proprietary Google services.
