Does it actually matter where you are, or is that just an implementation detail?

One interpretation is that Google is changing the URL bar from "where" to "who", which may be the more relevant information for most users. Signed exchanges are an interesting way to achieve that.