Advice for the author: if you use a netlist-based abstraction instead of hardcoding the circuit in code, you will be able to simplify the code, and make it much easier to modify and inspect.
Having no experience with netlists other than briefly reading through the Wikipedia page, it seems they are basically just a struct of subcomponents which is what I'm using. How would I go about abstracting that?
For what it's worth, there are many existing C-code emulations of actual 1970s 8-bit CPUs on the Net. (6800, 8080, Z80, 6502, etc, etc, etc.) Studying those may assist you in managing to get past the tricky bits.
In my own Z80 emulation work, I have leaned heavily on 'Yaze' and 'Dasm', with a few extra features and debugging of my own.
For a simple but still practical instruction set, you can probably sanely get away with about two dozen or so different instructions, but at the expense of having larger programs (by number of instructions) to accomplish the same tasks compared to a more featureful instruction set.
Here's a fairly simple set that might be a reasonable place to start...
ALU: and, or, xor, complement, shift left/right, rotate left/right, add, subtract, multiply, divide, compare (or just use subtract and set flags for this one)
Memory: move (copy between registers), load and store (between memory and a register), maybe also push and pop for sanity if you want to provide explicit stack instructions.
Control flow: jump, branch (maybe depending on flags from ALU), call/return
It might also be helpful to look at the 8-bit AVR instruction set (quite popular for small microcontrollers, including what you might find on many Arduino boards). It contains about a hundred or so instructions, with all of the above list present in one form or another (although many are just aliases for some other instruction with the same opcode but otherwise fixed parameters). In the case of AVR, where code size is often the limiting constraint, having a larger vocabulary of instructions is useful for expressing a program in the smallest amount of space possible, but it certainly isn't necessary to have all of them just for completeness.
Typically whenever there's a reference to an N-bit instruction set, it's referring to the widths of the registers and data paths. Very often this is not the same as the instruction size, and in many cases instructions may even be of variable length.
So while it's much more restricting to commit yourself to only an 8-bit instruction size, it should still be possible to build something with it.
I didn't go as far as writing my own OS, but I did develop some simple functions to render characters on a screen and accept keyboard input.
