What wasn't viable?
They had full access to the 64 bit GPRs. There wasn't anything technically stopping game code from accessing the 64 bit address space by reinterpreting a 64 bit int as a pointer (except that nothing was mapped there).
It's only the pointers that were 32 bit, and that was nothing more than a compiler modification (like the linux x32 ABI).
They did it to minimise memory space/bandwidth. With only 512 MB of memory, it made zero sense to waste the full 8 bytes per pointer. The savings quickly add up for pointer heavy structures.
I remember this being a pain point for early PS3 homebrew. Stock gcc was missing the compiler modifications, and you had a choice between compiling 32 bit code (which couldn't use the 64bit GPRs) or wasting bandwidth on 64 bit pointers (with a bunch of hacky adapter code for dealing 32 bit pointers from Sony libraries)
That allowed both a CPU and an advanced GPU to be on the same die.
They also wisely sold Global Foundries, and were able to scale with TSMC.
At that time AMD wasn't in the custom CPU business, AMD64 was a new unproven ISA, and x86 based CPUs of that time were notoriously hot for a console. These were also some of the reasons why Microsoft moved away from the Pentium III it had used in the original Xbox.
The PS3 was launched in 2006 but the hardware design was decided years earlier to provide a reference platform for the software.
Even they could use off-the-shelf SKU it wouldn't be viable - neither one had one that fits in power envelope (not that it helped xbox...)
The PSX's CPU is semi-custom. The core is a reasonably stock R3000 CPU, but the MMU is slightly modified and they attached a custom GTE coprocessor.... I guess you can debate if attaching a co-processor counts as custom or not (but then the ps4/xbone/ps5/xbs use unmodified AMD jaguar/zen2 cores)
IMO, the N64's CPU counts as off-the-shelf... however the requirements of the N64 (especially cost requirements) might have slightly leaked into the design of the R4300i. But the N64's RSP is a custom CPU, a from scratch MIPS design that doesn't share DNA with anything else.
But the Dreamcast's CPU is actually the result of a joint venture between Hitachi and Sega. There are actually two variants of the SH4, the SH4 and SH4a. The Dreamcast uses the SH4a (despite half the documentation on the internet saying it uses the SH4), which adds a 4-way SIMD unit that's absolutely essential for processing vertices.
We don't know how much influence Sega's needs had over the whole SH4 design, but the SIMD unit is absolutely there for the Dreamcast, I'm pretty sure it's the first 4-way floating point SIMD on the market. The fact that both the SH4/SH4a were then sold to everyone else, doesn't mean they were off the shelf.
Really, the original Xbox using an off-the-shelf CPU is an outlier (technically it's a custom SKU, but really it's just a binned die with half the cache disabled).
Do you trust the new line of CPUs that just launched that year?
