Stable Diffusion with Core ML on Apple Silicon (opens in new tab)

It would increase my interest in experimenting with these models 1000% at the least. I really can't be bothered to spend hours fucking around with pip/pipenv/poetry/virtualenv/anaconda/god knows what other flavour of the month package manager is in use. I just want to clone it and run it, like a Go project. I don't want to download some files from a random website and move them into a special directory in the repo only created after running a script with special flags or some bullshit. I want to clone and run.

It's one of the reasons I recently ported the Whisper model to plain C/C++. You just clone the repo, run `make [model]` and you are ready to go. No Python, no frameworks, no packages - plain and simple.

https://github.com/ggerganov/whisper.cpp

PyTorch has libtorch as its purely native library. There are also Rust bindings for libtorch:

https://github.com/LaurentMazare/tch-rs

I used this in the past to make a transformer-based syntax annotator. Fully in Rust, no Python required:

https://github.com/tensordot/syntaxdot

If you are okay with using nvidia-ecosystem, check out tensor rt.

What part of the SD license prohibits that?

But it's not optimised to work with Apple's CoreML (yet), isn't it?

Yeah, it is just PyTorch MPS backend is not fully baked and have some slowness. You should be able to get close to that number with maple-diffusion (probably 10% slower) or my app: https://drawthings.ai/ (probably around 20% slower, but it supports samplers that takes less steps (50 -> 30)).

For comparison, it's also taking ~3min @ 50 iterations on my 12c Threadripper using OpenVino. It sounds like the improvements bring the M1 performance roughly in line with a GTX 1080.

How do dreamstudio/craiyon/hugging face manage to do seemingly quicker on their interfaces? Are they hosting these models on super beefy and costly GPUs for free?

Haven't tried this yet, but sounds slower than SD itself if you use one of the alt builds that supports mps where it had been cuda.

Mac Studio with M1 Ultra gets 3.3 iters/sec for me.

MacBook Pro M1 Max gets 2.8 iters/sec for me.

Agreed.

And the posted benchmarks for the M2 Macbook Air make me consider 'upgrading' to an Air.

Not really, everyone will have their own flavor on how to rapidly train the model.

Dall-e et. al will still be able to bandwagon off of all the free ecosystem being built around the $10M SD1.4 model that is showing what is possible.

E.g. Dall-e could go straight to Hollywood if their model training works better than SD’s. The toolsets will work

yeah, finally we see the real openAI

The true metric contains the output quality of the image, not just the speed. DALL-E output is, generally, much better for things that aren't standard looking.

I think they can move upmarket just as well as anyone else.

It's less versatile out of the box. Give it a couple months for the community to catch up. Everyone is still figuring out what goes where, and SD 1.x was "everything goes in one spot." It was cool and powerful, but limited.

You can still do nice things with SD2, it just requires a different approach. https://news.ycombinator.com/item?id=33780543

Also, can you not "upgrade" but still run new models?

There are different requirements for generating video -- at a minimum, continuity is tough. There are models for producing video, but (as far as I've seen) they're still a bit wobbly.

Yeah, sure. The issue is with temporal consistency. Meta and Google have some successes in that area.

https://mezha.media/en/2022/10/06/google-is-working-on-image...

Give it some time and SD will be able to do the same.

They already do, with varying levels of performance and success.

See deforum[1] and andreasjansson‘s stable-diffusion-animation[2]

[1]: https://deforum.github.io/

[2]: https://replicate.com/andreasjansson/stable-diffusion-animat...

Video is really a series of frames, the framerate for film/human can get away with 24 frames/second-- so maybe ~40ms/image for real-time at least?

What's cool about the era in which we live is if you look at high-performance graphics for games or simulations, for instance, it may in fact be faster to a the model to "enhance" a low-resolution frame rather than trying to render it fully on the machine.

ex. AMD's FSR vs NVIDIA DLSS

- AMD FSR (Fidelity FX Super Resolution): https://www.amd.com/en/technologies/fidelityfx-super-resolut...

- NVIDIA DLSS (Deep Learning Super Sampling): jhttps://www.nvidia.com/en-us/geforce/technologies/dlss/

AMD's approach renders the game at a crummy, low-detail resolution then each frame uses "upscales"

Both FSR and DLSS aim to improve frames-per-second in games by rendering them below your monitor’s native resolution, then upscaling them to make up the difference in sharpness. Currently, FSR uses spatial upscaling, meaning it only applies its upscaling algorithm to one frame at a time. Temporal upscalers, like DLSS, can compare multiple frames at once, to reconstruct a more finely-detailed image that both more closely resembles native res and can better handle motion. DLSS specifically uses the machine learning capabilities of GeForce RTX graphics cards to process all that data in (more or less) real time.

Video is really a series of frames, the framerate for film/human could get away with 24 frames/second-- ~40ms/image for real-time.

What's cool about the era in which we live is if you look at high-performance graphics for games or simulations, it may in fact be faster to run the model on each frame to "enhance" a low-resolution frame rather than trying to render it fully on the machine.

ex. AMD's FSR vs NVIDIA DLSS

- AMD FSR (Fidelity FX Super Resolution): https://www.amd.com/en/technologies/fidelityfx-super-resolut...

- NVIDIA DLSS (Deep Learning Super Sampling): https://www.nvidia.com/en-us/geforce/technologies/dlss/

AMD's approach renders the game at a crummy, low-detail resolution then use "spatial upscaling" to enhance the images one frame at a time.

NVIDIA DLSS uses "temporal upscaling" to pass over multiple frames and uses other capabilities exclusive to Nvidia's cards to stitch together the frames.

This is a different challenge than generating the content from scratch

I don't think this is possible in real-time yet, but someone put a filter trained on the German country side to produce photorealistic Grand Theft Auto driving gameplay:

https://www.youtube.com/watch?v=P1IcaBn3ej0

Notice the mountains in the background go from Southern California brown to lush green

https://www.rockpapershotgun.com/amd-fsr-20-is-a-more-demand....

This is sort of like if you learned English from reading a book and the author said they owned all your English sentences after that.

Of course you can see the original images (https://rom1504.github.io/clip-retrieval/), it was legal to collect them (they used robots.txt for consent just like Google Image Search) and it was legal to do this with them (but not using US legal principles since it's made in Germany).

"Crediting the artist" isn't a legal principle - it's more like some kind of social media standard which is enforced by random amateur artists yelling at you if you don't do it. It's both impossible (there are no original artists for a given output) and wouldn't do anything to help the main social issue (future artists having their jobs taken by AIs).

Do your point is that Apple and those grifters are equally reputable?

two wrongs don't make a right.

A bit different take is DiffusionBee, if you're curious to try it out in a GUI form.

https://diffusionbee.com

Let's give it a few days and someone will have something semi-automatic ready

> Oh gosh that's an intimidating installation process

I'm not seeing any installation instructions on either link - what am I missing?

Where are you seeing the installation process?

I could be wrong but I think part of the issue is this needs some large files for the trained dataset?

Thanks!

I suspect it was virtual memory - the CoreML conversion progress was at 32Gi at one point and there's only 16GB in this laptop. That would explain why it was consuming 30Gi+ of disk space when the output CoreML models only totalled 2.5Gi.

Just used this again on 3 different computers, including mine. Works fantastically still.

Found a >100GB accidental “livestream” recording on one computer. Would have taken forever to find what was taking up all the room otherwise.

ncdu is the best in my book. TUI, supports deletion of files and folders, and very simple to understand.

GUI apps for this task like GP and the like are more visually complex than they need to be.

    /dev/disk3s5  926Gi  857Gi   52Gi    95% 8067489 540828800    1%   /System/Volumes/Data

It normally hovers around 30-35Gi free.

Great, thank you. Look like there’s already a GH issue: https://github.com/AUTOMATIC1111/stable-diffusion-webui/issu...

Is it? I just ran it on my M1 MacBook Air and am getting 3 it/sec, same as I was using Stable Diffusion for M1. Maybe I'm doing something wrong?

Stability! The main reason why I use it locally is because I don't want some random dev unilaterally deciding to change or "sunsetting" features I rely on.

Centralized services small and large are guilty of this and I'm sick of it.

"A long time ago in a galaxy far, far away"

but seriously, I wonder when you'll be able to paste in a script, and get out a storyboard or a movie

The laptop body still heats up and over long periods of time this can degrade the battery, I’ve measured a sharp drop in capacity from the device itself.

ah 3.56 minutes, my mistake