I agree that full mesh network is bad for scaling, but IRC's directed graph—manually configured even—is bad for availability. There is some talk about making the routing better, but I guess it's not in near future..
In Matrix it could be a problem, but practically it's not a problem even in the largest room #matrix:matrix.org with probably thousands of servers. But it makes joining that room quite slow and resource intensive.
Of course there is no IRC network with thousands of servers, because accepted hosts are well curated by the rules of the IRC network whereas any Matrix server can join the global federation. Granted, you don't need to join a chat network with an IRC server, but then neither does a Matrix server.
