Writing GPU drivers is not hard per se. It is impossible if you don't work at the vendor and have access to internal documentation.

Knowing how to write a kernel (which incidentally I am doing for the second time) doesn't mean you have years to dedicate to reverse engineer something as complicated as a GPU