It should be. If your company has a lax attitude to WFH or is even fully remote, then it may even be better than going to the office. I know some extremely high-performance companies that swear by WFH. Like Coinbase, GitLab and a few others out there. You'll hardly have issues in companies like that.
If however your company is a bit more traditional, then you run a risk that, psychologically, your manager think more about the people he sees at the office in person, and less about the people he only sees through video chat. This will hinder your career progress significantly.
In general, a good rule of thumb is: the more your career growth, or potential growth matters to you, the less "wiggle room" you have, and the more you will simply have to accept certain terms and conditions. For example, if you think it is unacceptable for you to attend the office, and your prospective employer wants you at the office at least a few days a week, then it is obvious that you will not be able to choose them, and you'll have to go with another employer that won't place that big of an emphasis on office work. To this end, it is obvious to conclude that your career growth is going to be more limited the more constraints you throw into the equation.
