Show HN: Self-host Whisper As a Service with GUI and queueing (opens in new tab)

Can you not separate into two phases? Speech separation to get source per speaker, and then whisper on each in isolation (maybe interlacing prompts)?

https://github.com/ggerganov/whisper.cpp has diarization

I'm badly looking for that! Is there a repo I can follow?

Ok. Our service is pretty fast. Also the M-Macs is really fast imo

whats your training rig like?

And there's a fork of that that uses DirectCompute to run it on GPUs without Cuda on Windows:

https://github.com/Const-me/Whisper

I used whisper.cpp to build a tool to transcribe audio files, either as a one off or as a folder watcher:

https://github.com/kardianos/audioclerk

Built in Go/cgo.

Is there a fork running it on Apple's Neural Engine?

Yes. This runs the OpenAI Whisper model locally.

This have the same output formats plus it's own Jojo-format to open in an atm internal Mac-app and in the editor-feature-branch.

I was working on this yesterday. It seems that the most common approach with Whisper is simply to break the audio into chunks and transcribe each one separately. This works but as you'd expect sometimes has trouble at the edges. The segments also have to be sufficiently long (like 10s) or the accuracy suffers, meaning it's not truly real-time.

You could do better by overlapping the segments, except then stitching the transcriptions together becomes an issue since whisper doesn't provide reliable per-token timestamps [0], and the output of the common part of overlapping segments isn't necessarily the same. I can imagine a cool approach where you transcribe long, overlapping chunks in real-time and intelligently merge the stream of words somehow though.

Some more useful discussion here (whisper.cpp project, but still relevant) [1].

0. https://github.com/openai/whisper/discussions/332

1. https://github.com/ggerganov/whisper.cpp/issues/10

You can prefix a prompt for Whisper with a small text section containing desired vocab, and it will likely improve accuracy for that specific domain.

Whisper source is very readable, check out https://github.com/openai/whisper/blob/main/whisper/decoding...

Vosk

https://alphacephei.com/vosk/lm

You can restrict the vocabulary the way you like, for example, here is the chess app built with Vosk

https://www.chessvis.com/

Awesome!

We use whisper at $dayjob and we found that it is far better than Azures transcripts and we found that Azure and GCP had about the same correctness.

I'd assume whisper will be better than YT auto ones for sure, especially if you choose the right model

Generally yes when it produces sane output at all, but while YT can get stuff comically wrong I've never seen it just go off the rails and start hallucinating and mindlessly repeating itself, which Whisper sometimes does especially if you're also trying to get it to translate something. Like Whisper will sometimes output a stream of things like "Please subscribe to my channel and follow me on Twitter!" or "Thank you for watching.".

On one source I tried the other day, the first 90 seconds or so is just generic opening music, no speech, but it "transcribes" it as "This is the end of the video. Thank you for watching. Please subscribe to the channel if you like. See you in the next video. Thank you for watching. Please subscribe to the channel if you like. Thank you for watching. ..." If you help it along by cutting up the source into only spoken segments you can get it to do better but just throwing it at a directory of material is probably going to leave you with some disappointment.

Then sometimes it does something surprising, on a j-pop song after hallucinating a bit during the intro it spit out a translation in the form you might find on a lyrics site, that is each line was "japanese-characters romaji-version english-translation". I haven't been able to get it to do it again (even for the same source).

But more impressive is that it understand a lot of other languages then just English. Really impressed with hard-to-understand Norwegian dialects.

It is pretty good. I tried it a few times and there were minor mistakes at best. The Whisper readme itself shows a live transcription use case.

That's just the Swedish part of Schibsted. We're vg.no and Norwegian.

Yes, you need to fine-tune the model with your data. This might be easy or hard, depending on your experience level and complexity of the model and available tooling.

For this model specifically (https://github.com/openai/whisper) it would be a significant challenge for a newcomer. Luckily Huggingface has a blog post that will get you started: https://huggingface.co/blog/fine-tune-whisper

Is it running on a max or are you accessing iCloud from a non-Mac?

Thanks for feedback. Looks like the Readme need more work

Working on it. https://github.com/schibsted/WAAS/pull/101