Of course -- you can release software with a license that requires the recipient to abstain from wearing pink shirts if you want[0].
That license would be non-FSF-Free and wouldn't meet the OSI definition of Open Source, though, as both groups see the benefit of allowing anyone to use the software. The OSI lists the criteria[1] as "No Discrimination Against Persons or Groups" and "No Discrimination Against Fields of Endeavor".
Therefore, while possible, it might not be particularly useful to do that. You also need to consider the implications of combining your software with software released under a differently-permissive license -- you could quickly get into a position where your software can't be distributed at all without violating at least one of the licenses that apply.
Also note that I Am Not A Lawyer and I'm sure there's a jurisdiction somewhere which would read your license in a different way to that which is intended, or would strike out parts of it altogether, ending up with either a more permissive or a less permissive license than you'd intended.
[0]: I am currently wearing a pink shirt. Please don't sue me.
[1]: http://opensource.org/osd