To get you with Spectre, the attacker must be able to run code on your CPU.
This affects browsers with Javascript enabled because your Javascript engine runs foreign code on the CPU. The bad guy puts nasty code in a page, you visit the page, the code executes on your machine--boom.
And it affects public cloud web servers because multiple cloud servers (virtual machines) run on one CPU. So some attacker might be able to jump out of their VM and read your VM's memory.
BUT, on a dedicated hardware web server, there shouldn't be any foreign code running--no foreign VMs, and no browser.