Generating transparent images using Stable Diffusion XL (opens in new tab)

The specific issue that bothers me is the incorrect imitation bokeh. The hair farther from the head is too blurry despite being well within what should be the focal plane.

This is inherent to what I think is Apple’s model of depth of field: Apple takes a picture that is fairly sharp everywhere and generates an ordinary RGB image plus a depth map (an estimated distance for each pixel). Then it applies some sort of blur that depends on depth.

This is a decent approximation if the scene contains opaque, pixel-sized or larger objects, so that each pixel’s content actually has a well defined depth. But hair tends to be much smaller than a pixel, and a pixel containing both hair and background can’t be correctly represented.

This was an issue in older (circa 2000?) Z-buffered rendering — if you naively render hair and then render an object behind the person based on the Z data from the hair rendering, you get very wrong-looking hair. It turns out that just having a GPU that can handle a zillion vertices doesn’t mean that rendering each hair independently gives good results!

I think it is how it fills in "waves" in the hair, notice how each time a bunch of hair intersects with another, at least one set of hair curves unnaturally? Valleys and peaks where the hair bends should be similar in natural hair, here they are not.

Working link: https://github.com/layerdiffusion/sd-forge-layerdiffusion/as....

It's not too far off. Vid2vid is already decent at keeping character consistency when configured correctly, background/environment flickering is hard to control but since the process is currently done using img2img on successive frames that makes sense. I think we'll see new models that do temporal convolution soon that will make video -> video transformations absolutely stunning.