They only became more open out of necessity.
”After building it for himself and showing it at the Club, he and Steve Jobs gave out schematics (technical designs) for the computer to interested club members and even helped some of them build and test out copies”
Also: the Apple 2 shipped with full schematics and an assembly listing of its ROMs.
The ‘phone book’ edition of Inside Macintosh didn’t document all hardware, but I wouldn’t call it ‘closed’. It explains how OS calls are implemented, details on the floppy disk interface, etc.
What I remember are the LC and Quadra with NuBus, AppleTalk, QuickDraw, QuickDraw 3D, Sound Manager...
Hardly open in what regards using standards common to other platforms.
Having proper technical manuals was common to all companies during those days.
NuBus was an open standard, the ISA bus of PCs wasn’t. Neither were the Atari or Amiga buses.
As to AppleTalk: they could have chosen something else for networking, but I don’t see what they should have used instead of AppleTalk for networking. Ethernet existed, but used thick coax cable (twisted pair is from 1984, standardized in 1986. See https://en.wikipedia.org/wiki/Ethernet_over_twisted_pair); token ring is of about the same time, but also proprietary, and wasn’t ready in time for the launch of the Macintosh (https://en.wikipedia.org/wiki/AppleTalk#History). TCP/IP existed, but it wasn’t clear that it would win the race, and it was overkill for the intended uses of AppleTalk.
The various operating system APIs aren’t less or more proprietary than what OS/2, Windows, Atari’s TOS, AmigaOS, and the various Unixes offered at the time (OpenGL wasn’t available yet; X Windows would have crawled on Mac hardware of the time even more than QuickDraw did, and wasn’t stabilized in 1984)
I also don’t see that they should have done things different after they made the choices they made for the 1984 Mac. They could have opened the hardware somewhere in the 1980s, but it isn’t certain that that would have been beneficial for them.
I think the only real difference is the openness of the PC hardware architecture, and that was accidental/unintended.
Not saying it’s a bad thing, mind.