Kudos to Wyze for doing the things noted in the thread like being honest and prompt with notification etc, but "thumbnails from your Events were visible... and that thumbnail was tapped" is a pretty mealy-mouthed way to say "another person saw your private pictures and videos taken with your Wyze cameras"
I actually appreciate the more specific details on how the private pictures and videos were actually viewed using terminology from the Wyze app.
On the Wyze subreddit, people have been griping about the fact that it took them days to even acknowledge something happened. When I read the Wyze email about it, there was no new info in it, it had pretty much all already been discussed online.
https://support.wyze.com/hc/en-us/articles/360026245231-Wyze...
A good response to this might be to put it back, and to extend other devices to be dual-use (Wyze Cloud or HA).
I've use the official RTSP firmware in the past on some v2 cams, but I remember it having some problems and not being as good as this solution.
I continue to be amazed that there is not a reasonably priced, open source, audited, local-first solution, which doesn’t require a significant personal investment of time to install and maintain.
Local-first cameras are super cheap and easy to find. Most just run their own local RTSP server, which you can connect to live with VLC, homeassistant, whatever. OpenIPC is like ddwrt for ip cameras, here is the supported devices page: https://github.com/OpenIPC/wiki/blob/master/en/guide-support...
But unless you're Richard Stallman, go with the closed source Reolink RTSP camera, which is about $100 and used by big corporate installs. It can integrate with the cloud, but you can set it up to just have each camera run a RTSP server with user/pass auth. You have to secure your own network.
But there isn't a really great open source platform for the kind of multi cam security that businesses might need. You have to do your own storage. But grab four reolinks, send the feeds to homeassistant, and most homeowners will be fine.
I have spent days searching for an affordable solution that meets my requirements. I know there are a lot of people that want the same thing, have done their own research, and have reached a similar conclusion. What we want simply does not exist.
I will stop posting this kind of comment when someone finally gives me a straight answer, but no one has done that yet.
Edit: Another comment mentioned https://gitlab.com/Shinobi-Systems/Shinobi which I haven’t explored but am definitely going to also try out now.
I don't have time to mess around with hacking a camera to do RTSP and then figure out how to set up and use something unfriendly like ZoneMinder.
The margins on these devices are already pretty high because they're enterprise and business focused - I "buy once cry once"'d when I bought my <vendor>* drive for the home. I would have bought a different vendor if I was forced to pay an additional premium for a render capable CPU.
(* - I'm probably tin foil hating here, but realized from a security posture perspective I don't want to publicly state the vendors I use in my network.)
The V3 models need to be downgraded to a specific firmware first and patching it exposes RSTP streams using https://github.com/AlexxIT/go2rtc. Everything doable without ever installing Wyze app on an environment air gapped environment with no internet.
1. all cameras (firmware v4.36.9.139) have 64gb+ micro SD cards and record to local storage -- many people seem to have issues with anything greater than 32gb in v3's but I've found that this Verbatim tool [0] formats FAT32 at high capacity with no problems
2. all cameras have wz_mini_hacks [1] on the SD card with RTSP enabled
3. all cameras are connected via ethernet instead of wifi using this adapter [2] and wz_mini_hacks config
4. network blocks all outgoing internet connections for all cameras to keep them LAN-only -- this means I have to connect to VPN to review video when outside the house, but I'm cool with that
5. all RTSP streams are also recorded over the network via Agent DVR [3] to a NAS
6. the Wyze app (free tier, not paid) works normally with all of the above in place -- I find it much more intuitive to review recent videos in-app (streamed off the SD card), and then review the very occasional older video from a computer off the NAS (scrubbing through in VLC on a computer)
For what it's worth, I don't use them like a Ring camera where you're responding to realtime video events / talking through the camera to a delivery person -- this is mostly just for 24/7 recording. I have all object/motion detection events turned off, just a straight uninterrupted feed recording local and on the network.
Links:
0. https://www.verbatim.com/index/search.php?words=fat32+tool
1. https://github.com/gtxaspec/wz_mini_hacks
If you don't and IP addresses cycle, your RTSP stream URLs will change and your recording software will just consider the camera disconnected. The Wyze app sometimes will get confused too (with ethernet in particular, if IP addresses change), and rebooting the cameras sometimes means physically cycling power if you don't know where it's at in your network. Better to just reserve a specific IP up front, write it down, and never think about it again.
I also use a lot of these types of crappy smart plugs [0] throughout the house (as well as starting to replace some with actual wifi outlets [1] for aesthetics) and it's actually really useful to be able to force a power cycle on certain things from my phone.
Links:
0. https://www.amazon.com/gp/product/B09LXGHR5X
1. https://www.amazon.com/KP200-Outlet-Required-Control-Certifi...
Do you have any Wyze cams outdoors, but under cover of a porch? Or are they indoor only?
As time and motivation permit, I've been converting the cameras I care about over to POE. But having to run cable across the house for each one means I haven't done them all.
Even a bad implementation with cloud-synced encryption keys (which defeats most of the benefits of e2e) would have stopped this.
The response in this case (notifying customers and specifically stating whether they were affected or not) is excellent, but this seems to be a repeat of a previous incident from September 2023: https://www.theverge.com/2023/9/8/23865255/wyze-security-cam...
https://www.consumerreports.org/home-garden/home-security-ca...
Responses typically range from "I'm not that interesting" to "I really don't care". I think it's too abstract of a threat for most people to take seriously before it happens to them.
I’d go with “don’t put internet-connected cameras in your house if you don’t want those images on the internet”. I’ve got a Wyze in my garage looking over my mountain bikes, and for $35 I don’t really care if somebody else sees that image. But I’d never put one in my living space, regardless of their security track record.
Unless the camera has lasers, it isn't very good at stopping said theft. ;)
We quickly realized that cache is fast but not infallible. Use proper security on all your resources. Don’t rely on UUIDs to obfuscate your data as security.
That seems like enough of a line of bullshit to steer me away from ever using wyze.
Given their openness in the rest of the communications, I don't see why they would make this part up.
Edit: Of course, I'm also curious what the actual bug was. A discussion below is suggesting several plausible ways (e.g. concurrency issues, insufficient entropy in some key) how a problem could happen under load (although many of these would also lead to the problem happening with less load, just much less often).
No, I'm not questioning whether or not it was a caching issue. I'm taking exception to the lack of accountability. They chose the library. They (probably) chose to ignore a documented or common failure mode of caching systems through either poor choice of key or lack of synchronization. They've obviously designed their infrastructure in a way that isn't resilient to its current level of usage (cold start is a normal part of software's lifecycle).
They could have chosen to own that, but instead they blamed everyone else. That's not a sign of a trustworthy service provider.
Hardware. Rowhammer-type effects occurring accidentally under sudden load spikes. The hardware has just got too dense.
(I should clarify this is speculation, but reading the recent article included here on sudo using special maximum-distance bitfields to hold state internally (https://news.ycombinator.com/item?id=39165342)... it must be a problem that's being observed in the wild)
Caching is normally read heavy, not write heavy, so it's plausible it wouldn't be something you'd see much under typical operation. After an outage, they'd be dealing with a thundering herd level of traffic as everything tries to reconnect, that'd be very different from normal write loads, even different than the write load they'd have seen when they first enabled caching.
Very little ownership on Wyze's side.
-- Phil Karlton
> The incident was caused by a third-party caching client library that was recently integrated into our system. This client library received unprecedented load conditions caused by devices coming back online all at once. As a result of increased demand, it mixed up device ID and user ID mapping and connected some data to incorrect accounts.
What? How does load on the system affect correctness?
> The outage originated from our partner AWS
What does this mean? Was there an AWS outage for a service they use, or was this just a normal loss of an instance?
It's interesting that they blame external entities for the root causes of the incident and don't take responsibility for what is ultimately on them.
Seen this happen quite often with code that is not multi-thread safe, especially in languages like c# and java, such as using a static class property for data that should be request-scoped, or not using the appropriate concurrent collection classes etc.
Instead of blaming the users, we must hold the companies responsible. Data privacy laws must be stricter and these incidents must be taken more seriously.
You might be able to cause consequences after the fact, but your data will still get leaked first. You can't undo a privacy violation with tort law, and there won't ever be criminal penalties.
If you want your information to stay private, don't store it on other people's computers. IDGAF what "the default today" is. (Also, that's wrong - everyone that is serious and actually wants their data to remain private doesn't store it in the cloud. This is why the CIA got Amazon to build a custom airgapped on-prem AWS region at Langley, for instance.)
I had a heck of a time finding a proper POE recording DVR camera system for my mom's house without online or cloud bullshit, but still I isolated it on the network to not take any chances of UPnP port opening or dial-home crap.
The only system I would trust would be one that laid out their security model, source to their apps, and had a self-hosted server DVR option. The captological signals of 99.9% of security system websites do not instill confidence in my mind.
First of all, google "Wyze RTSP firmware". It's the official firmware from the vendor that enables the RTSP protocol. Now you can enable RTSP via the app and give the camera a fixed IP address in your DHCP server.
RTSP is a pretty standard protocol, so you can now view the feed via VNC player, record it 24/7 via ffmpeg, use tools like motion, etc.
The camera will still try to connect to cloud, but you can move it to a local-only Wi-Fi network, or outright block it from reaching the outer world on the router side.
And if you want advanced stuff (multiple streams, organized recording, etc), there is a plethora of free/open-source security camera tools (iSpy for instance). It all takes time to learn and configure, but you can have your own fully closed-circuit surveillance network, while still using the Wyze's rather cheap hardware.
and don't overlook that user's other repos, as seems like there are quite a few fun things in there: https://github.com/koush?tab=repositories
As an software engineer who's dealt with caches for large high throughput services, this does not make sense to me why they are blaming a caching client. It's your own code that will decide what is the cache key, and what value to pass as the cache key. Did the caching library have a bug where when you ask for a given key, it returned results for a different key? Or more likely did your own code have a bug where you mixed up the keys? I think we need more details on what went wrong in here.
Yes, of course. Blame a third party library which was probably created by an open source maintainer instead of testing your own systems.
but even with this, it's still an security camera app that can't send push notification without cloud access.
Waze could come out with a line of dash cams, and then you'd have Waze cams and Wyze cams.
In the era of cloud and microservices why each user does not have their own dedicated resources?
Because race to the bottom.
People want incredibly cheap products that are internet connected. Your average home user should not have to worry (and won't) about cybersecurity concerns, so this will continue to happen. The only out I can foresee is government regulation stepping in to make these incidents actually hurt for the companies, but America has basically no appetite for that.
