At a very basic level it can be summed up with: the JVM approach to GLSL was to just throw it at the GPU, whereas the browsers worked on restricting WebGL to a super constrained subset (this was my fault, but is the correct thing to do :D )
The applets also made it hard to do a bunch of the simple games that were super popular - flash included a large amount of multimedia functions built in, and an editing environment that was geared towards interactive design.
The reason java took so long to move out of process was because the java<->browser bindings were unique to java, it did not use any of the normal plugin apis, and expected a lot of direct linkage to the host system.
