The cost depends on the technology. More modern processes are dramatically more expensive (don't even think about something like 28nm). If you're looking at something ancient like 130nm, 0.35um, etc then it's going to be something like $1-2k per mm^2. Normally there's a minimum cost of a few mm^2. Expect to get a few tens of chips back. Transistor count has no effect, they couldn't care less what you put inside your part of the wafer, so long as it follows the design rules (for example minimum % coverage on each layer to prevent it collapsing vertically), but obviously if you need more transistors you might need more area.
Yes it's 100% possible to do an open-source design, qflow has been used to make sub-circuits of ASICs, but it's going to be extremely difficult. There are lots of things missing which you'd have to take from the fabs PDK or design yourself, for example open source I/O pads (sounds boring, but actually lots of work with ESD, etc). Combined with huge missing feature-sets in the open source tools, like extraction of designs back to SPICE circuits with parasitics and complete DRC checking, you're not going to have a fun time.
