I went to a talk on "general purpose GPU programming" at the Colorado School of Mines around 2001 that covered exactly that topic. It was very disappointing to have my interest in FPGAs for this purpose be so entirely destroyed by a quirk of graphics card design.
Hinton also addressed the contribution of hardware performance advances to practical deep neural net applications in his talks in the mid-2000s.
