I think you're right that that was his point. It wouldn't necessarily come out of their budgets, though, it could be done with debt financing on the project itself, repaid purely from the revenues from leasing access to the network to the customer facing ISPs.
I also don't think it would necessarily be used by a small subset of the constituents - it seems as though it could potentially be much cheaper than alternative high-speed arrangements, since the customer facing companies would actually have some competition. Once the initial construction debt was paid down, the govt. could likely reduce the access lease cost, which would be passed on by the competitive ISPs, further enabling access by poorer residents.
I think it would be great for the long-term economic prospects of those areas as it broadens access and makes heavy uses like remote work more viable.
Plus, a good internet connection is a much cheaper form of entertainment than many of the alternatives, including cable, so it could actually save many people money.