As somebody doing this now I am absolutely out of the loop on certain discussions but while it's often challenging, it can actually be helpful in certain cases. Many of my teammates end up being roped into meetings and ad-hoc tasks as backup but ultimately provide little or no input and just waste their time. Meanwhile, I'm protected from a lot of the distracting, low-value work and am left to focus on things I'm leading and anything important enough to be communicated to the group as a broadcast or call to action. Additionally, my group has become so accustomed to working in a way that accommodates remote work that the colocated team members have started working from home more often, encouraging further adjustment to support remote work. That being said, I don't think my arrangement would have been successful without having my manager and a couple of colleagues acting as my advocates on the ground.
Good management will also avoid including people unnecessarily and encourage a culture where people can depart when it's obvious they're not needed. I tried making an early exit from meetings more acceptable with humor. But ultimately everyone has to buy into the idea.