ABC: Can you tell us why the system's gone down?
Optus CEO: Unfortunately, I don't have more information to give at this stage. We have had issues since 4am. The team has tried a number of parts of restoration and so far we have not had the results that we have hoped for. And we're pursuing every avenue to get everybody back online as soon as possible.
ABC: Do they know what happened though? Who's there the team that you've got working on it?
Optus CEO: Our team is still pursuing every possible avenue. We had a number of hypotheses and each one so far that we've tested and put in place new actions for has not resolved the fundamental issue. So we're still working on it. And when we have a identified root cause and a time for restoration, we'll be updating everybody as soon as we can.
---
Pretty wild to be down for over 7 hours no with no sign of a fix yet. Note they have also said they don't believe it's a 'cyber attack' (Optus had an enormous data breach last year with 9.7 million customers data leaked: https://en.wikipedia.org/wiki/2022_Optus_data_breach )
They hired a new Director of "Core Network", and in her first all-hands meeting she loudly asked what IPV6 was. (serious, she had never heard of it).
The Director of IT had me setup her home wifi because she had no idea what DHCP was.
The VP of "IT/Internet" literally had other people make her Powerpoints because she couldn't. She had never done anything more on a computer than send emails.
It was shocking, but also really eye-opening to realize how important business acumen and playing the political game were.
In response to hours of silence from Optus executives (in that they weren't able to string two words together about the technical issues or the source of the fault) one caller to ABC Local Radio in Sydney asked whether the fault was in Australia or in Singapore which was an excellent question as Optus is owned by Singtel (Singapore Telecommunications Limited).
The logic behind the call was to ask about how much of the Optus infrastructure actually exists in Singapore. If, say, much of the technical control is done from there then it may have been hard to quickly diagnose the problem with the system infrastructure split across two countries.
This is important because when Optus was sold to Singtel there was much concern about the security of its telecommunications if another country had access to user data, etc. At the time we were told the Australian Government was happy with the security arrangements that were put in place but many were not completely satisfied with the assurance.
Given the huge data breach that Optus experienced some months ago and that that matter has not been resolved by any stretch there will have to be a fully-fledged inquiry into the failure and hopefully it will get to the bottom of how Singtel actually runs Optus.
That said, some background is needed for overseas readers.
Australia has a long, long history of screwing up telecommunications, radio and TV broadcasting not to mention spectrum management. Too involed to describe here in detail but briefly the stuff-ups and political shenanigans would fill volumes and they go right back to the Wireless Telegraphy Act of 1905 and the Postmaster General's Act. The Govt. PMG's Dept. had exclusive monopoly over the phone network for nearly a century (like Ma Bell but govt. owned). It later became Telecom Australia but still govt. owned. Still later the then Government sold it off and it's now a public corp under the name Telstra.
To maximize income from the Telstra sell-off the Government sold it off in two stages, and to make the sale attractive to investors it sold the telco complete—lock, stock and barrel without due consideration of the public interest. That is, instead of just selling off the exchanges and phone network infrastructure the Government also sold off all cableways and rights-of-way across the nation as a bundled deal. Also, it never put sufficiently tight regulations or conditions in place to protect customers and they've since been screwed rotten by all telcos—horrible deals, little or no service—for instance, witness today's Optus outage!
The sale of the cableways came back to bite the Government later when it needed to establish the NBN—National Broadband Network—as it had to buy back access to Telstra cableways which cost many billions—it's a huge scandal the Government somehow swept under the carpet. Also, the competition couldn't use the Telstra cableways, Optus had to completely install its new network onto power poles.
This first-class fuckup has now been 'amortized' into a surcharge that everyone across Australia has to pay for internet and phone. The extra billions have had to come from somewhere and it's the poor hapless consumer that has been paying the bill.
(In the past I've suggested some enterprising person do a PhD on the subject and in the process determine the extra per capita costs that every citizen is now paying over and above what he/she ought to have been paying had there been prudent management and less destructive politics.)
Incidentally, the NBN itself was further fucked up by politics, half-measures and technical compromises, it's story in itself. Just this week there's been an announcement that NBN network prices are to be increased.
Into this almighty shemozzle comes Telstra's competition, Vodafone and Optus. These telcos have also benefited from slack regulations, very poor consumer law and seemingly nonexistent technical oversight by Government.
Right, when it comes to telecommutations, Australia is the laughing stock of the world and its citizens have been taken for a very expensive ride.
Many are saying Govt. is largely at fault and asleep at the wheel for not keeping stricter tabs on these telcos.
Given there are three major networks in Australia and a few minor ones and that only Optus went down the level of disruption was alarming. Essentially, much of the country was in turmoil for better part of a day.
If a comparatively small outage can cause such disruption to a country then just imagine the chaos that'll follow when the next Carrington Event hits us.
“Our on-site technician is actively prioritising establishing a console connection [a physical cable connection],” the message to Optus’ partners early on Wednesday said. “Rest assured that said technician is also being provided additional technical support remotely.”
[1] https://www.smh.com.au/technology/what-caused-the-optus-outa...
[1] https://www.optus.com.au/dafiles/OCA/Wholesale/ProductAndSer...
[2] https://maprad.io/us/search/text/optus?filters=WyJkdl9saWNlb...
#SysAdminLife
Source: https://www.stuff.co.nz/world/australia/301004965/major-optu...
I’m sure some useless agile product owner scrum type person came up with the brilliant idea of needing an internet connection just to turn the motor on and off.
If you are a software developer it's best to take a day off or read a book (good for those who have bought a physical one). No reason to stress, isn't your project delayed by more than a day anyway?
In theory, you could have line 1 forward to line 2 when 'unreached'. Of course you'd want the second line to be different network.
Their primary and their backup were both Optus. Not much of a backup.
* https://www.productreview.com.au/listings/launtel
Note - I don't work for them or anything, am just a happy customer. :)