It's hard to imagine, but I wish they would have taken advantage of him walking in with the compromised device in the first place.
I once stumbled upon a really bad vulnerability in a traditional telco provider, and the amount of work it took to get them to pay attention when only having the front door available was staggering. Took dedicated attempts over about a week to get in touch with the right people - their support org was completely ineffective at escalating the issue.
Cox's support organization was presented with a compromised device being handed to them by an infosec professional, and they couldn't handle it effectively at all.
I can't really blame them. The number of customers able to qualify that a device has actually been hacked is nearly zero. But do you know how many naive users out there that will call/visit because they think they've been hacked? It's unfortunately larger than the former. And that'll cost the business money. When 99.9% of those cases, the user is wrong. They have not been hacked. I say this as someone who supported home users in the 2000s. Home users that often think they'd been "hacked".
He probably should have gone the responsible disclosure route with the modem too. Do you really expect a minimum wage front desk worker to be able to determine what’s a potential major security flaw, and what’s a random idiot who thinks his modem is broken because “modern warfare is slow”?
I've seen this across most companies I've tried reporting stuff to, two examples.
Sniffies (NSFW - gay hookup site) was at one point blasting their internal models out over a websocket, this included IP, private photos, salt + password [not plaintext], reports (who reported you, their message, etc), internal data such as your ISP and push notification certs for sending browser notifications. First line support dismissed it. Emails to higher ups got it taken care of in < 24 hours.
Funimation back in ~2019(?) was using Demandware for their shop, and left the API for it basically wide open, allowing you to query orders (with no info required) getting Last 4 of CC, address, email, etc for every order. Again frontline support dismissed it. This one took messaging the CTO over linkedin to get it resolved < a week (thanksgiving week at that).
Sounds to me like their support org was reasonably effective at their real job, which is keeping the crazies away from the engineers.
It's even harder for me to imagine them saying "Oh, gee, thanks for discovering that! Please walk right into the office, our firmware developer Greg is hard at work on the next-gen router but you can interrupt him."
“I’m a three star infosec General, if I’m contacting you it’s not to waste your time.”
They were presented with some random person who wanted to get a new modern on their rental but also keep the old one, for free. They had no way of knowing if they were an actual security professional.
Its stuff like this that company's should REWARD people for finding.
https://www.cox.com/aboutus/policies/cox-security-responsibl...
Frankly he could have just sold the vulnerability to the highest bidder
I would, too. Not sure we will ever learn. Maybe a load balancer config that inadvertently included "test" backends which didn't check authorization?
Thank god most things use HTTPS these days.
Personally, I built up a rube goldberg of software and hardware with bypass nics so if my firewall was off (or rebooting), traffic would flow through their modem, and when it was on, my firewall would take the traffic and selectively forward traffic through from the modem, but there's really no need for that when you just use an unmanaged switch. I can find the code if you're interested (requires FreeBSD), but you sound more sensible than that ;)
The XGS-PON workaround that DannyBee looks promising though:
https://pon.wiki/guides/masquerade-as-the-att-inc-bgw320-500...
I probably could pay to upgrade my speed to 2Gbps and then downgrade it back to 1Gbps and keep the XGS-PON.
> https://docs.netgate.com/pfsense/en/latest/recipes/authbridg...
There's another method that doesn't require Plus called pfatt, but I'm not sure what the state of it is.
* Plus is the paid version, yeah I know I agree I don't like what they did with the licensing changes but that's a different story
Unfortunately, my praise of Cox ends there. I've been having intermittent packet loss issues for 2 years, and there doesn't appear to be a support escalation path available to me, so I can't reach anyone that will understand the data I've captured indicating certain nodes are (probably) oversubscribed.
It's "plug in sfp+, upload firmware using web interface, enter equipment serial number"
You can even skip step 2 depending on the sfp stick you use.
The 802.1x state is not actually verified server side. The standard says modems should not pass traffic when 802.1x is required but not done. Most do anyway or can be changed to do so. AT&T side does not verify, and always passes traffic. That is what is happening under the covers.
I would like to bypass the BGW320 because not only it is a large, power hungry box, but it also requires me to jump through hoops to get IPV6 working with VLANs. I need to either use multiple physical links (simulating multiple devices) or simulate that using a VRRP hack, otherwise AT&T will not give out multiple ranges at all (and will not care about what I request). Under Comcast I didn't have to do any of that, I'd just carve out smaller IPV6 ranges, as many as needed.
If you really care you can configure a VPN directly on the router, so nothing leaves the network unencrypted.
I can't say for certain, and the OP if they're here I'd love for you to validate this - but I'm not convinced requests to the local admin interface on these Nokia routers is properly authenticated. I know this because I recently was provisioned with one and found there were certain settings I could not change as a regular admin, and I was refused the super admin account by the ISP. turns out you could just inspector hack the page to undisable the fields and change the fields yourself, and the API would happily accept them.
if this is the case, and an application can be running inside your network, it wouldn't be hard to compromise the router that way, but seems awfully specific!
That suggested to me that we shouldn't have ISPs that are this big. Cox is clearly a juicy target and a single vulnerability compromises, as an example from the article, even FBI field offices.
> After reporting the vulnerability to Cox, they investigated if the specific vector had ever been maliciously exploited in the past and found no history of abuse
Feel like author should have written "...they claimed to have investigated...".
I'm sure* they don't keep raw request logs around for 3+ years. I know what next steps I'd recommend, but even if they undertook those, they're not sharing that in the ticket.
(just based on industry experience; no insider knowledge.)
Would you trust a thing they say? It seems their whole network is swiss cheese.
As a typical user the noticeable symptoms for me were: - internet speed noticeably slows down - WiFi signal drops and personal devices either don't see it, or struggle to connect. At the same time the router is still connected to the internet - router's internal admin page (192.168.8.1) stopped responding
I imagine many users haven't updated their routers and thus may be hacked. In my case the hacker installed Pawns app from IPRoyal, which makes the router a proxy server and lets hacker and IPRoyal make money. The hacker also stole system logs containing information about who and when they use the device, whether any NAS is attached. They also had a reverse shell.
Solution: 1. Upgrade firmware to ensure these vulnerabilities are patched. 2. Then wipe the router to remove the actual malware. 3. Then disable SSH access, e.g. for GL.iNet routers that's possible within the Luci dashboard. 4. Afterwards disable remote access to the router, e.g. by turning Dynamic DNS off in GL.iNet. If remote access is needed, consider Cloudflare Tunnel or Zero Trust or similar. There is also GoodCloud, ZeroTier, Tailscale, etc. I am not too sure what they all do and which one would be suitable for protected access remotely. If anyone has advice, I would appreciate a comment.
Consider avoiding GL.iNet routers. They do not follow principle of least privilege (PoLP) - router runs processes using root user by default. SSH is also enabled by default (with root access), anyone can try to bruteforce in (10 symbol password consisting of [0-9A-Z] and possibly might be more predictable). I set mine to only allow ssh keys rather than a password to prevent that. Despite running OpenWrt they are actually running their own flavor of OpenWrt. So upgrading from OpenWrt 21.02 to 23.05 is not possible at the moment.
Could also be the neighbours and their big microwave oven :)
Because they didn't have enough logging or auditing to start with, or no logs or audit data left since the hack.
I mean, if you think about it from Cox's point of view — why would you disclose to someone outside the company if there had been history of abuse? Why would you disclose anything at all in fact?
function foo(token: string) {}
function bar(token: string) {}
function baz(token: string) {}
// hmm, this is annoying
let token;
.get((req) => { token = req.data.headers.token }
function foo() {}
It is even possible to do it by "accident" with only subtly more complicated code! I constantly see secrets leak to the frontend because a company is bundling their backend and frontend together and using their frontend as a proxy. This lack of separation of concerns leads to a very easy exploit:
If I'm using, say, Next.js, and I want access to the request throughout the frontend, I should use context. Next even provides this context for you (though honestly even this is really really scary), but before my code was isomorphic I could just assign it to a variable and access that.
At least in regards to the scaryness of the next provided global context, at least now node has AsyncLocalStorage which properly manages scoping, but plenty of legacy...
The entire ecosystem is awful.
From my distrust in bundlers, I'm now fuzzing within CI for auth issues. Hitting the server 10k times as fast as possible from two different users and ensuring that there is no mixup. Also, scanning the client bundle for secrets. I haven't had an issue yet, but I've watched these things happen regularly and I know that this sort of test is not common
Some CPEs have a cloud Wireshark-like capability for debugging. I'm not sure if those are even on the Cox production firmware images. Usually there's a set of firmware for production and a set for test (which obviously makes it hard to test for problems in production).
I suppose Cox could do a check to see what firmware versions are out there. ISPs can auto-upgrade firmware that doesn't match a specific firmware revision, and this was a Cox modem so they probably have firmware for it. So if it was a debug firmware how did it get there and survive?
Intercept all data on port 80, parse the http headers, do whatever you need with them, easy.
Not sure why anybody would replay the requests though.
You, I and most of the HN crowd may be well capable of maintaining a reasonably secure state of our own hardware and troubleshoot our way through common errors. However, the average internet user isn’t that experienced nor are most people interested in learning those skills.
I just put it in bridge mode, disable wifi, and all network functionality is served by my own devices.
The last modem I rented from ISP, the ISP didn't bother with any firmware updates for ~10 years. It was rock stable because of that, though. :)
Otherwise I would not be managing my own high quality one, based on the latest Linux kernel, and a standard, well supported and maintained software and carefully selected wifi HW with active manufacturer provided support.
I also would not be trying to isolate and disable most of the ISP provided HW/FW mess, if I believed otherwise. I don't trust ISP that did not upgrade their modem in 10 years, one bit with security of key entry point to my home network.
Same. Somehow I got them to install a simple modem, one without all of the router and access point features. I thought those single purpose devices didn't exist anymore.
Bought a relatively good router, installed OpenWRT on it then bridged it to the ISP's network via their equipment. It's working well. I even have HTTPS in my LAN now.
It's not a great idea to host services (especially if they can be used to identify you) on a home IP address you browse the internet with, and this is one way to get 1 IP address for browsing the net, and different 1 for serving services from home, pretty much for free.
We also initially thought we were the subject of a breach, but after the investigation we determined that the network's IDS was monitoring all traffic, and upon certain triggers, would make identical requests from external networks.
We found a way to identify all other similar IDSs across the internet and even "weaponize" this behavior. We ended up writing a paper on it: https://ian.ucsd.edu/papers/cset2023_fireye.pdf
Although, in fairness to Cox, this Information Asymmetry -- also exists between most companies that produce tech consumer goods and most tech consumers (i.e., is it really a big deal if most other big tech companies engage in the same practices?), with the occasional exception of the truly rare, completely transparent, 100% Open Source Hardware, 100% Open Source Software company...
https://en.wikipedia.org/wiki/Information_asymmetry
Anyway, a very interesting article!
At what point did you inform Cox about your findings? It doesn't sound like you were ever given the green light to poke at their management platform. Isn't work like this legally dubious, even if it is done purely in white-hat fashion?
If the request does use TLS, then even a compromised router should be unable to decrypt it. TLS is end-to-end encryption.
If the request doesn't use TLS, then the compromised router can already see the request and response that it is relaying. So why does it have to replay the request from somewhere else? It can just exfiltrate the session back to the attacker silently, without replaying it first.
==
If I had to guess, the attacker isn't sure what they're looking for in the HTTP sessions, so they can't push a detection for interesting sessions down to the compromised routers, and they also don't have the bandwidth to simply receive all unencrypted traffic from their router botnet, so instead they're collecting the URLs and building up a list of detection patterns over time through scanning and using heuristics for which requests are worth investigating, something like that?
In Germany you would get minimum 3 years in jail for this, people got in front of court for way way way way less.
In my opinion (as a security engineer) the biggest benefit of such programs is not amoral "hackers will always sell exploits to the highest bidder so companies must provide a high bounty for bugs in their software"[1] but "having a responsible disclosure process makes it totally clear that it's ok to report vulnerabilities without being sued".
Looking at the timeline below the post I can't see anything problematic. The author even waited the usual[2] 90 days before disclosure, even though the vulnerability was hotpatched a day after report (congrats to Cox btw). They also shared a draft blog post with them a month ago.
[1]They certainly should, in the ideal world.
[2]A deadline popularized (or even invented) by Google's project zero.
Great way to make sure researchers don't notify the victim of vulnerabilities, but rather stay quiet or sell it.
You'll note they never tried to change anything but their own equipment; doing otherwise would have been immoral and, yes, likely illegal. Without testing you have no idea whether or not you're actually looking at something that needs to be reported.
(As for the vendor, I'm sympathetic to the argument that there should be vendor liability under some circumstances.)
Sometimes they even try you in court if you don't publish it (yet)
I struggle to think of anything, short of auth handling being a separate service injected between a load balancer and the API servers, and someone somehow forgot to include that in autoscaling config; but surely this is not how you do things, is it?
Global singleton shared across requests, instead of request scoped.
1. [Client 1/You] Auth/write to variable (failed).
2. [Client 2/ISP] Auth/write to variable (success).
3. Verify what the result was (success)
A race condition combined with a global singleton can easily explain such behavior.
Are you describing some kind of server-side global object that statefully says a session/api key is "authenticated" and will then allow the request during that time frame? That seems like a bug you could drive container ships through. Yes I know saas s/w sucks out there but this would seem to at least be something an audit could easily flag.
Also, it looks like he hit a front-end API that drives the TR-069 backend. Changing the WiFi SSID is a long way from being able to "...execute commands on the device"
But the key point here is device independence - by law, providers need to give you all information required to establish a connection to them. This allows you to run a Linux or BSD box as a router should you wish to. It somehow makes up for the slow broadband speeds you can get.
*Edit: complaints about slow broadband speeds
I recently moved ISP, partly because of cost, but also because they offered a great home router as part of their bundle. The installer could not utilise any of the existing wiring in my house, had to be all drilled a second time...
Conversely, my last ISP used some awful Nokia modem that barely supported any kind of routing or customisation and I picked them specifically because it was a rental and the fibre wiring had already been done.
It's fairly common for ISP's in Australia to also give you a choice of BYOD or buying one of theirs. Usually you pay outright for the modem, however, so its yours to keep. That said, this is changing with the national fibre roll-out. But with ADSL being the de-facto choice, BYOD makes sense.
In France, I've noticed that some ISPs (Free for FTTH and SFR for FTTC + cable attached to the router) they'll offer the possibility of configuring the provided router in "bridge" mode, where you basically get the external IP to whichever equipment is hooked up to their router.
I've also had FTTH with SFR, which have a separate device which terminates the optical connection (ONT) and speaks ethernet with the main router. I don't remember if the main router was able to work in bridge mode. It was possible to connect your own router to the ONT but you had to jump through hoops [0] to actually receive a working DHCP response.
Bouygues also had the separate device for terminating the optical connection, connected via ethernet to the main router. The only catch was that it talked over vlan 100 for some reason, but other than that it was smooth sailing.
I've never had Orange, but I hear it's a pain to replace the actual router with them.
---
[0] IIRC you had to send some custom DHCP options pretending more or less to be an actual SFR router.
This aligns incentives nicely:
* Company creates a responsible disclosure program, users/researchers report problems for money/blog post fame, users are secure. Also security team becomes more important, because vulnerabilities cost (more) actual money. * Or company doesn't create a responsible disclosure program, someone exploits the bug in the wild, users are angry and the company is fined.
One of the things I'll never understand was why the attacker was replaying my traffic? They were clearly in my network and could access everything without being detected, why replay all the HTTP requests? So odd.
EDIT: I'd be hard-pressed to know how one could exploit this, given TLS would encrypt the requests. Maybe they're counting on using badly encrypted requests, encrypted with e.g. TLSv1.0?
That being said, we could all do with a bit more input sanitization, and I hope Cox learned their lesson here.
Did you determine if POSTs were replayed? As in, logging into accounts and sending payment info and account info?
Normally I'd laugh and assume device compromise but...
The largest ISP in Australia (Telstra) got caught doing exactly this over a decade ago. People got extra paranoid when they noticed the originating IP was from Rackspace as opposed to within Telstra. Turned out to be a filter vendor scraping with dubious acceptance from customers. The ToS was quietly and promptly updated.
Also, my wifi firmware occasionally crashes and needs to be restarted.
I don't work in cyber security or on anything sensitive, but if I was told I'm under surveillance by some government or some criminal, I would not be surprised.
That's more API endpoints than some first tier public clouds, wow. For a modem.
Somebody wanted (and sorta deserves) promo..
But also not, because the whole platform turned out to be incredibly insecure! Egregious!!!
I wonder if it’s a mix of exhilaration and being terrified!
What does this mean?
They either were paid and think it's nobody's business or weren't and have no ideological reason for making a stink. For what it's worth, I sympathize with people who feel shafted for their work by large companies, but think it is a little silly to go looking for it.