Historically, it was as likely to be part of the math department as it was to be part of electrical engineering. In part, this was because at the time electrical engineering was much more about analog circuits, power systems, etc.
in germany, you can still see the impact on whether CS emerged out of math or EE in the curriculums. When you have a lot of low-level programming, operating-systems, designing hardware etc. the departement was born in EE. If the emphasis is more on math, they teach you an abstract view on "computers" (the turing machine/lambda calculus as the foundation of computing) and view processors as just an application, it came from the math-departement. It's probably the same in the rest of the world.
