Yes, you can definitely get very severe attacks from backdoored hardware. Some of them appear almost impossible to defend against with software alone.

On the bright side, it's hard to imagine that many of these attacks will be self-propagating, which is the particularly insidious thing about the Trusting Trust attack. Yes, hardware is used to design hardware, but typically in a more indirect and heterogeneous way than the "compiler compiling itself" scenario. To be concrete, I'd say Microsoft or Canonical has much more to fear from a Trusting Trust sort of attack than Intel does, but the software developers also have better options to contain or detect such an attack.

There's an idea for hard sci-fi. Silicon backdoored with nanobots in sand.