Difficult realm for me to investigate myself. I have found that better than any book is to pull old VMS images out and get them up on a SIMH/Vax emulator and poke around! It's a quite laborious process and documentation of the process is sparse, last time I ended up using the Computer History Museum remote access program to interact with ancient VMS.
To actually learn how to use the thing I ended up diving deep into textfiles.com archives for references to VMS and got the hang of the basics enough to start learning via the online (built in) documentation.
I would also point you towards usenet archives if you want to learn more about the development history of VMS, what what i have been able to piece together it was really rock solid and loved and quite a shame that "open"VMS is mostly just a legacy compatibility licensing agreement and the technology is in a bit of a glacial deep freeze.
The fact that every file on the entire OS had version control - by default - before 1990... that blows me away and I feel like there's so many beautiful secrets to learn from this codebase. I always imagined it would be written in some abstract arcane language, when I learned it was in C i got a bit scared knowing what microsoft calls C has very little resemblance to how C is typically used on unix likes, and with the knowledge that many VAX admins dumped VMS for BSD I was quite afraid I may be out of my element.
I was surprised to find the C library is essentially just posix, I felt pretty much completely at home on VMS compared to any version of windows I have had to write C for. Pleasant experience, shame what happened to it, wish there was more about its story told in an authoritative context like a book.
If you stumble across anything nifty please feel free to share I may come back and read this read again someday.
Cheers
