That would be absurd for someone to do, but, if someone wants to, well, freedom of association?
Note: I am not strongly committed to this idea I am expressing at the time of this writing. I'm not even sure I am at all committed to it. I do think its a relevant position that seems worth considering, if nothing else.
Two freedoms seem to be conflicting here. Person A's freedom to do legal things in their free time and person B's freedom to not like the other person's choices. I can't stop person B from being prejudiced. However, I argue that person B should not have the right to control person A's legal activities.
In this case, doctor A could drink the glass of wine and decline patient B as his client. Patient B would pay a higher price on average to express his preference, and doctor A would receive less business to express his preference.
I imagine a pothead doctor could even use the fact that he's a pothead to market to other potheads.