tl;dr - the fire destroyed over 1,000 homes, two deaths. The local electrical utility, Xcel, was found as a contributing cause from sparking power lines during a strong wind storm. As a result, electrical utilities now cut power to affected areas during strong winds.
The failure of a single such server is far from a disaster.
Time services are available from other locations. That's the disaster plan. I'm sure there will be some negative consequences from this downtime, especially if all the Boulder reference time sources lose power, but disaster plans mitigate negative consequences, they can't eliminate them.
Utility power fails, automatic transfer switches fail, backup generators fail, building fires happen, etc. Sometimes the system has to be shut down.
There's no present need for important hard-to-replace sciencey-dudes to go into the shop (which is probably both cold, and dark, and may have other problems that make it unsafe: it's deliberately closed) to futz around with the the time machines.
We still have other NTP clocks. Spooky-accurate clocks that the public can get to, even, like just up the road at NIST in Fort Collins (where WWVB lives, and which is currently up), and in Maryland.
This is just one set.
And beyond that, we've also got clocks in GPS satellites orbiting, and a whole world of low-stratum NTP servers that distribute that time on the network. (I have one such GPS-backed NTP server on the shelf behind me; there's not much to it.)
And the orbital GPS clocks are controlled by the US Navy, not NIST.
So there's redundancy in distribution, and also control, and some of the clocks aren't even on the Earth.
Some people may be bit by this if their systems rely on only one NTP server, or only on the subset of them that are down.
And if we're following section 3.2 of RFC 8633 and using multiple diverse NTP sources for our important stuff, then this event (while certainly interesting!) is not presently an issue at all.
That's some strong winds! What's causing such strong sustained/gusty winds that long? I'm hearing about this weather phenomenon for the first time.
[1] https://www.eecis.udel.edu/~mills/ntp.html
That's why the most stable ones are insulated and ovenized[1].
So an AC failure which would lead to higher room temperatures would lead to stronger or more frequent correction by the NTP client, as the local oscillator would drift more.
Not sure about the fire case though. I mean the same applies there but I'm not imaginative enough to think of a realistic scenario where NTP would be useful for averting a fire.
[1]: https://blog.bliley.com/anatomy-of-an-ocxo-oven-controlled-c...
Having a parallel low bandwidth, low power, low waste heat network infrastructure for this suddenly seems useful.
WWV still seems to be up, including voice phone access.
NIST Boulder has a recorded phone number for site status, and it says that as of December 20, the site is closed with no access.
NIST's main web site says they put status info on various social media accounts, but there's no announcement about this.
Being unfamiliar with it, it's hard to tell if this is a minor blip that happens all the time, or if it's potentially a major issue that could cause cascading errors equal to the hype of Y2K.
And most enterprises, including banks, use databases.
So by bad luck, you may get a couple of transactions reversed in order of time, such as a $20 debit incorrectly happening before a $10 credit, when your bank balance was only $10 prior to both those transactions. So your balance temporarily goes negative.
Now imagine if all those amounts were ten thousand times higher ...
Asking for a friend.
The main problem will be services that assume at least one of the NIST time servers is up. Somewhere, there's going to be something that won't work right when all the NIST NTP servers are down. But what?
If you have information on what they actually are using internally, please share.
The network will route around the damage with no real effects. Maybe a few microseconds of jitter as you have to ask a more distant server for the time.
The answer is no. Anyone claiming this will have an impact on infrastructure has no evidence backing it up. Table top exercises at best.
Perhaps, "We don't know." will become popular?
https://jila.colorado.edu/news-events/articles/spare-time
Discussed here: https://news.ycombinator.com/item?id=46042946
And how many randomly chosen wristwatches would you need to get anything reasonable?
But yes, good point.
Given two time changes per year I guess something like 1 min per year is acceptable
It's open to the public for visits. They have a small science museum, offices, a library, etc. I highly recommend anyone with interest and opportunity to visit the mesa lab soon. It may not be open much longer. The view alone is worth the trip, and the building is cool too.
THE TIME RIFT OF 2100: How We lost the Future --- and Gained the Past.
https://tech.slashdot.org/comments.pl?sid=7132077&cid=493082...
This is some level of eldritch magic that I am aware of, but not familiar with but am interested in learning.
Probably more interesting is how you get a tier 0 site back in sync - NIST rents out these cyberpunk looking units you can use to get your local frequency standards up to scratch for ~$700/month https://www.nist.gov/programs-projects/frequency-measurement...
Also thank you for that link, this is exactly the kind of esoteric knowledge that I enjoy learning about
To quote the ITU: "UTC is based on about 450 atomic clocks, which are maintained in 85 national time laboratories around the world." https://www.itu.int/hub/2023/07/coordinated-universal-time-a...
Beyond this, as other commenters have said, anyone who is really dependent on having exact time (such as telcos, broadcasters, and those running global synchronized databases) should have their own atomic clock fleets. There are thousands and thousands of atomic clocks in these fleets worldwide. Moreover, GPS time, used by many to act as their time reference, is distributed by yet other means.
Nothing bad will happen, except to those who have deliberately made these specific Stratum 0 clocks their only reference time. Anyone who has either left their computer at its factory settings or has set up their NTP configuration in accordance to recommended settings will be unaffected by this.
NIST currently offers this service free of charge. We require written requests to arrive by U.S. mail or fax containing:
Your organization’s name, physical address, fax number (if desired as a reply method).
One or more point-of-contact personnel or system operators authorized to receive key data and other correspondence: names, phone numbers, email addresses. Up to four static IPv4 network addresses under the user’s control which will be allowed to use the unique key. By special arrangement, additional addresses or address ranges may be requested.
Desired hash function (“key type”). NIST currently supports MD5, SHA1, SHA256, and HMAC-SHA256. Please list any limitations your client software places on key values, if known: maximum length, characters used, or whether hexadecimal key representations are required. If you prefer, please share details about your client software or NTP appliance so we can anticipate key format issues. Desired method for NIST’s reply: U.S. mail, fax, or a secure download service operated by Department of Commerce.
NIST will not use email for sending key data.
ps. there actually seems to be improvement over what they had year ago. they added "secure download service". and previously they had message that nobody assigned to actively monitor mailbox so if you didn't get key, please email us so we will check it
While underground distribution systems are less prone to interruption from bad weather it depends on the circuit design. If the underground portion of the circuit is fed from overhead power lines coming from the distribution substation you will still experience interruptions from faults on the overhead. These faults can also occur on overhead transmission circuits (the lines feeding the distribution substations and/or very large industrial customers).
Underground distribution comes at a cost premium compared to overhead distribution. It’s akin to the cost of building a picket fence vs installing a geothermal heating system for your home. This is why new sub divisions will commonly have underground cable installed as the entire neighborhood is being constructed - there’s no need to retrofit underground cable into an existing area and so the costs are lower and borne upfront.
It’s more cost effective for them to turn the power off as a storm rolls through, patrol, make repairs and reenergize then to move everything underground. Lost revenue during that period is a small fraction of the cost of taking an existing grid and rebuilding it underground. This is especially true for transmission circuits that are strung between steel towers over enormous distances.
Some of it is physical infrastructure (transformers, wire, poles), but a lot of it is labor.
Labor is expensive in US. It’s a lot of labor to do, plus they’ll likely need regulatory approval, buying out land, working through easements.
At the same time you have people screaming about how expensive energy is.
Furthermore they have higher priorities, replacing ancient aging infrastructure that’s crumbling and being put on higher load every day.
https://practical.engineering/blog/2021/9/16/repairing-under...
remote refid st t when poll reach delay offset jitter
==============================================================================
+time-e-b.nist.g .NIST. 1 u 372 1024 377 125.260 1.314 0.280 #time-e-b.nist.g .NIST. 1 u 1071 1024 377 125.260 1.314 0.280Maybe their generator failing was DOGE related, but wouldn’t have happened if state level shenanigans were better handled
Don't forget Solar Roof.
Some relevant DOGE’s effects:
-time and frequency division director quit
-NIST emergency management staff at least 50% vacant
-NIST director of safety retired, and NIST safety was already understaffed when compared to DOE labs
-NOAA emergency manager on the same Boulder campus laid off
etc
It’s just a good idea, though, not a greedy one… so it won’t happen.
And far from a disgrace, NTP was a brilliant design for its time, and has proven to be far more future-proof than should have been expected.
Future proof? Are you living in the same future I am where our trades are front run? I notice you didn’t engage with the substance of my attack at all.