Blobs of glass are actually overwhelmingly linear :) Non-linear effects in optics are generally negligible unless you go to very high energies (scattering, like raman scattering) or special materials (like fluorescent ones). You need a high powered laser or certain materials for those regimes.
That said, the geometric functions (i.e. the positions of rays w.r.t. other objects) are probably non-linear in the object position parameters indeed, and intensities certainly are (example: moving an object in front of a screen by X amount changes the illumination in front of the screen non-linearly): it's important to keep in mind what we mean by linearity (it's in terms of light field intensities for a fixed geometry scene) -- the scene are the transfer function coefficients.
What really makes things complicated in real life is (1) the presence of noise; (2) imperfections (and unknowns) in your physical/geometric apparatus. Even if you know the system perfectly, noise generally disallows reverting (or easily reverting) transfer functions, i.e. undoing blurs and arbitrarily refocusing images with simple detectors. The imperfections and even lack of rigidity of lenses and your system add even more difficulty. That's why making a simple computational lens is not so easy.