POC works using 'npx efficient-recorder'.
Is this useful to anyone?
The thing that enabled rewind.ai and MS Recall is storing the series of screenshots more like a HEIF, allowing for massive compression ratio and on-device storage + OCR provided by the OS (Live Text since Monterey 2021 [0], Microsoft introduced it last year for Snapdragon based AI-PCs [1])
I guess this is a good starting point if the goal is to fill S3 buckets with screencaps of multiple users, but then we're just back to corporate spyware, not tools for helping individuals use their machine more effectively.
That said, if I was using my own minio backend, it would be neat to archive my screen captures but I would change it so it captures after, say, every keystroke, and every time my mouse stops moving, and after every click. That way I have high density capture of taking actions, and low density otherwise. In any case collecting the data is not the issue, making an interface where that data becomes useful to help me remember something is.
[0] https://support.apple.com/guide/preview/interact-with-text-i...
[1] https://learn.microsoft.com/en-us/windows/ai/apis/text-recog...
If I was to build one of these, which I'm not, I would try for a RTSP to bucket uploader. That way you could do the actual capture and compression with OBS like any other streamer - or use it with IP security cameras etc. You'd probably end up with a pile of video-ts pieces which could be replayed later using HLS.
I've also explored Swift AVFoundation to drop frames and colours at the moment of recording, but won't be implementing it at this time.
This was just a POC, and couldn't hack it in a day.
Also, automatically doesn't cost storage for identical screenshots (no activity) and very cheap for just moving your mouse around or typing a few characters.
I wish the LLMs were tuned into the open source landscape more, so when someone has an idea for a POC and asks the bot to write it for them, it would go clippy mode and say "it looks like you're trying to build an open source rewrite of Rewind, would you like to clone the rem repo and contribute to that instead?" lol
Most feel that Recall is also this.
It's not quite screen captures, but the way in which any given email is responded to by competent users in your own organization is highly relevant in this context, especially if you place original+reply email pairs into a RAG framework and add function calls for structured domain knowledge.
Unified APIs like https://www.nylas.com/ which an admin can unilaterally connect across an entire org can make this quite viable - assuming you've done the work to build a culture where radical transparency is seen as an opportunity rather than a threat.
There's a lot of nuance required to avoid hallucinations, but organizations that are merely training chatbots on explicit Q&A documents are just scratching the surface of the depth of their semi-structured data.
> Attempt to create an Open Source Privacy Focused Rewind.ai Alternative for data capture
I'd assume this was something local or at least for your local network. But this exclusively sends the data over to S3. And based on the lack of encryption keys or even passwords, I'm assuming this is even unecrypted?
I'm happy to think about e2e encryption.
Also I'd buy a pendant if I can send it to my own S3!!!!
Native MacOS in swift (the popular one with OCR / text selection from history), and cross platform (rust) without text selection from history and very much POC.
The repo looks like it has quite a few stars and a smattering of issues which I thought meant real usage.
I think it should be done in Swift tbh to get battery impact under 10%
I really like the rewind.ai retrieval mechanism. I believe their recording mechanism is highly broken. It often fails to sync to the os calendar and will ask you to record meetings you deleted months ago.
I don’t understand the webcam recording need. I’m not sure what signal you get from that since if you are in a web meeting you already have that on screen. Or if you are coding you might get a few WTF frown faces if working on a hard bug. But you made it optional, so that’s good.
Must one set up a S3 compatible stack on a home server somewhere?
While we’re here, has anyone been able to export audio from Rewind.ai’s local storage?
It's also useful to me in that it's a solid example of what can be done with LLMs these days, wow!
Also, tangentially, a long long time ago I had a similar system set up, except for packets, not screencaps or audio. A 24h ringbuffer on my router to log _everything_ was a cool-to-have that made debugging network issues easier.
The issue with this one is that it misses the most important feature, the searchability. But you could probably focus on the low overhead aspect of your version.
[1] * Screenpipe https://github.com/mediar-ai/screenpipe * Memento https://github.com/apirrone/Memento * Rem https://github.com/jasonjmcghee/rem
