You can write mostly C code. Each processor will have different quirks.
https://en.wikipedia.org/wiki/CPU_cache
https://www.akkadia.org/drepper/cpumemory.pdf
These are great places to start.
Especially Ulrich Drepper's PDF on memory.
edit: even simple things like looping can require knowledge of "bits" and "memory layout" and "CPU cache" to write the fastest code. Most people don't need to write the fastest code though.