https://dev.arie.bovenberg.net/blog/python-datetime-pitfalls...
https://news.ycombinator.com/item?id=39417231 (147 comments)
Now at least there’s an LLM that might spot a bug every now and then so that’s nice.
I wonder what benefits this choice has that outweigh the risks of this behavior.
https://en.wikipedia.org/wiki/Acid3
I like this new lib (Thank You) but the name unfortunately implies the opposite of what it is. "Whenever" sounds like you don't care, but you'd only be using this if you did care! Also Shakira, haha. Hmm, pedantic is taken. Timely, precise, punctual, meticulous, ahorita, pronto, etc. I like that temporal name.
Finally, none of these links mention immutability, but it should be mentioned at the top.
I learned the hard way, that dependencies kill projects.
Not saying this isn't great, thanks for creating it! It does have its use cases, of course.
I work in healthcare. If I have a choice between "reading docs/changelogs carefully, implementing functions", and "adding an extra dependency", I'm taking the dependency every single time.
I don't want footguns in my code, I don't want code I have to write and test myself, and I don't want to have to become an expert in a domain before I can write something that serves my purpose.
For the datetime library, specifically, I'm switching to whenever for everything, because I've been bitten by conversions and naive/aware datetime confusion too many times.
Link to Tom Scott & Computerphile from 10y ago on tz madness. https://www.youtube.com/watch?v=-5wpm-gesOY
This kinda sums up the sorry state of software engineering. People can't even be bothered to read docs but will just blindly install a package just because someone was able to package it and upload it to PyPI.
Taking on a dependency does not relieve you of reading docs, but it also adds a further burden as you now need to trust the code. The stdlib is much more heavily tested and documented than any 3rd party library will be.
Sure, but the opposite applies as well. Sticking with the flawed stdlib means you are trusting that every single future developer is as careful in reading all the docs as you are - even when it's someone reviewing that random trivial-looking patch which is secretly hiding a massive footgun. A junior developer submitted a five-line patch touching datetime? Better schedule several hours for a proper analysis!
Or you can write your own wrapper code, of course. Which will almost certainly have worse documentation and testing than a popular third-party library.
External libraries aren't evil. When chosen properly, they relieve you of burdens. You shouldn't grab any random "leftpad"-like dependency like they are going out of fashion, but something as complicated as timezone handling is best left to code written by domain experts - which means using a library.
You initially said you write your own code instead of using libraries, I replied to that, and now it's that you use the stdlib instead of libraries. I won't argue against shifting goalposts.
Or do it with sight?
That's a straw man argument. No one said "blindly". You can very well carefully consider the pros and cons of adding a dependency and arrive at the conclusion that it makes sense. Many PyPI packages are in the Debian stable repositories, you could use that as an additional barrier as well.
That's why I use a Flake8 plugin to prohibit especially egregious footguns.
These things are really frustrating
As long as there is a conscious decision to build or ‘buy’, it’s fine. I think some people can be a little too careless with adding dependencies though, not realising they can have an equal if not greater maintenance burden.
The only crazier idea I can think of is implementing character encoding conversions myself.
Nobody needs a package for "left-pad", which is the most infamous example.
But there are a lot of areas where it's not a good use of your time to reinvent the wheel. There's no moral virtue in writing everything yourself. And popular packages have gone through a more bug-finding and bug-fixing than your personal code probably ever will.
Timedates are hard, and units may require even harder historical/present "political" tracking as how they are defined, and I would never want to maintain this kind of dependency: https://github.com/ryantenney/gnu-units/blob/master/units.da...
And what comes to timedate problems, I try to keep it simple if the project allows: store and operate with UTC timestamps everywhere and only temporarily convert to to local time (DSTs applied, if such) when displaying it in user-facing UI. This functionality/understanding can be locked into own 20-line microlibrary-dependency, which forces its responsible person to understand country's timezone and when e.g. DST changes and where/how/who decides DST changes and what APIs is used to get UTC time (and of course, its dependencies, e.g. NTP, and its dependencies. e.g. unperturbed ground-state hyperfine transition frequency of the caesium-133 atom, which result is then combined with the Pope Gregory XIII's Gregorian calendar, which is a type of solar calendar mixed with religious event for fixing the time, which which is then finally corrected by rewinding/forwarding "the clock" because its too bright or dark for the politicians).
I would still rather using a library for dates, a million times so.
And this for every single problem: time, text, maths, network, parsing, formatting, validating, authenticating...
You know, they import 5 libraries, each of which imports 5 more libraries, each of which imports 5 more libraries, and suddenly they're buried in 'critical' updates because there's a denial-of-service bug in the date parser used by the yaml parser used by the configuration library used by the logging library used by the application.
E.g the JS project that uses the stdlib Date API, and pulls in moment.js, and also uses date-fns.
Or the one that pulls in bits and pieces of lodash, ramda, and other functional libraries.
And maybe it uses native fetch and axios depending on the current phase of the moon.
They don’t die but time is wasted in code review trying to understand if there is any kind of deliberate approach behind the scattershot application of packages with duplicated purposes.
(picking on JS is perhaps unfair but it’s probably the most egregious example of dependency hell)
Some problems simply require using the right tools. They aren't necessarily hard, but they will be if you try to hammer a nail in with a screwdriver. The Date API, and to a certain extent Python's datetime module, are screwdrivers for a nail-shaped problem.
The rest of your example seem to have more to do with bad dependency practices than using dependencies in the first place. If you are going to include a dependency, think about it, consider whether it's worth it, document that decision, and then consistently use that dependency. Just because you've seen projects use dependencies poorly doesn't mean dependents are bad by themselves.
But yeah, you will have to create tests for the codepath, instead of relying on the tests the library maintainers create.
Hard pass. The complexity of having to use binary packages or build things is not worth the performance benefit. The pure-Python version requires building from source and passing special flags, so it is not possible to specify it in requirements.txt.
An issue was closed as not planned: https://github.com/ariebovenberg/whenever/issues/158
1. I _love_ pure Python packages. Not everybody should be forced to use Rust. I want installing pure-Python to be as easy as possible
2. Having separate names on PyPi (with or without extras) creates confusion for libraries depending on whenever: should they depend on whenever-py or whenever-rust? If one “overwrites” the other, this adds confusion.
3. Most users expect to “pip install whenever” and start using the “fast” version
For me, points (3) and (2) weigh heavy enough to make (1) slightly more cumbersome.
But: Maybe I’ve missed something. Have a read in the above issue or add your 2 cents.
edit: formatting
In particular, default implementation of datetime in cpython is a C module (with a fallback to pure python one) https://github.com/python/cpython/blob/main/Modules/_datetim...
Not saying it's necessarily justified in case of this library, but if they want to compete with stdlib datetime in terms of performance, some parts will need to be compiled.
You can put any flags in requirements.txt, including -r[equiring] another txt etc.
Your point may apply to modern pyproject.toml tooling though, or at least that it wouldn't be simply another entry in the dependencies array.
Avoid general terms like "Pacific Standard Time" and stick to location-specific ones like: "Vancouver/Canada". The latter is how people expect their time to work, and correctly handles whatever quirky choices jurisdictions choose to do with their time.
Searching the list here: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
I cannot find an entry for "Pacific Standard Time" nor "Vancouver/Canada", but I can see: "America/Vancouver".
Then it would have been nice to see the benchmarks of the pure Python implementation as well. What if it's worse than arrow?
> In casual benchmarks, the pure-Python version is about 10x slower than the Rust version, making it 5x slower than the standard library but still (in general) faster than Pendulum and Arrow.
"(in general)" here since the speed compares differently per operation, while the Rust version is faster across the board. That said, there's no operation that is _significantly_ (or unnecessarily) slower than Arrow or Pendulum.
edit: I'm considering adding comparison to the pure Python version once I get the time for a more expanded "benchmarks" page in the docs
Almost all of the time UTC is enough, if I need to filter/bucket/aggregate by some range, I can reach for datetime with tz for these filter/bucket/aggregate criteria, convert them to UTC and on continues `int` comparison.
I'd imagine all of the cases handled by Whenever are mostly when datetime is a long lived object, which I don't see a need for at all.
I use it purely for allowing tz input from client, convert to UTC immediately when it arrives, or, if I really need the tz, then save it separately, which is rare (one example is calendar, where tz should be stored, although probably not even next to every UTC but at the user level, another is workforce scheduling, where 8am-4pm or 8pm-4am can mean different things for different locations -- but this is no longer datetime, it's purely time in a timezone).
And so anything server-related with calendars will be making tons of these conversions constantly. And you can't cache things long-term in UTC because the conversions of future events can change, when countries change DST etc.
So you would not store that in UTC but just in time.
But yes, I’m ignoring the standard of calendar formats , maybe they are simpler .
I read through the article listing all the weirdness of other datetime libraries and I’d say many were covering cases where you behave that timezoned datetime is long lived .
One case even pointed out datetime construction with an impossible hour.
I am currently enjoying DST-free life in Japan, and feel that people around the world deserve to get this much respect from their own official clocks.
(Schools tend to have earlier times. It's not so unusual for a school's workday to have its midpoint at about noon, I think.)
I've always been in favour of keeping the clocks at non-DST all year, but now I have a new proposal: keep them at DST and just hibernate in the winter. Work an hour or two less in the winter when it's miserable.
Maybe adjust the work schedule to e.g. start at 8 instead of 9?
Rather than mess with the actual clock.
And for program code, it wouldn’t really help as long as it’s still expected to be able to correctly handle dates in the past.
I’m not sure how real those costs would be beyond a transitional period, but that’s the discussions that have been going on. It’s a political risk, and nobody wants to be on the losing side of such a change.
I am an amateur dev, though, so maybe someone who masters the language will be better off using the raw standard libraries.
I'm sure I'm in the top 1% of software devs for the most number of timestamps parsed. [1]
DST is not a problem in Python. It's parsing string timestamps. All libraries are bad, including this one, except Pandas. Pandas does great at DST too btw.
And I'm not shilling for Pandas either. I'm a Polars user who helicopters Pandas in whenever there's a timestamp that needs to be parsed.
Pandas has great defaults. Here's string timestamps I expect to be paesed by default. I'm willing to pass timezone in case of naive timestamps:
* All ISO 8601 formats and all its weird mutant children that differ by a tiny bit.
* 2025-05-01 (parsed not as date, but as timestamp)
* 2025-05-01 00:00:00 (or 00.0 or 00.000 or 0.000000 etc)
* 2025-05-01 00:00:00z (or uppercase Z or 00.0z or 00.000z or 0.000000z)
* 2025-05-01 00:00:00+02:00 (I don't need this converted to some time zone. Store offset if you must or convert to UTC. It should be comparable to other non naive timestamps).
* 2025-03-30 02:30:00+02:00 (This is a non existent timestamp wrt European DST but a legitimate timestamp in timestamp representation, therefore it should be allowed unless I specify CET or Europe/Berlin whatever)
* There's other timestamps formats that are non standard but are obvious. Allow for a Boolean parameter called accept_sensible_string_parsing and then parse the following:
\* 2025-05-01 00:00 (HH:mm format)
\* 2025-05-01 00:00+01:00 (HH:mm format)
Disclaimer: I'm on the phone and on the couch so I wasn't able to test the lib for its string parsing before posting this comment.
Javascript's big datetime redesign (Temporal) has an interesting overview of the decisions they made [1]. Whenever is currently undergoing an expansion of ISO support as well, if you'd like to chime in [2].
[1] https://tc39.es/proposal-temporal/#sec-temporal-iso8601gramm... [2] https://github.com/ariebovenberg/whenever/issues/204#issueco...
Your customers are software devs like me. When we're in control of generating timestamps, we know we must use standard ISO formatting.
However, what do I do when my customers give me access to an S3 bucket with 1 billion timestamps in an arbitrary (yet decipherable) format?
In the GitHub issue you seem to have undergone an evolution from purity to pragmatism. I support this 100%.
What I've also noticed is that you seem to try to find grounding or motivation for "where to draw the line" from what's already been done in Temporal or Python stdlib etc. This is where I'd like to challenge your intuitions and ask you instead to open the flood gates and accept any format that is theoretically sensible under ISO format.
Why? The damage has already been done. Any format you can think of, already exists out there. You just haven't realized it yet.
You know who has accepted this? Pandas devs (I assume, I don't them). The following are legitimate timestamps under Pandas (22.2.x):
* 2025-03-30T (nope, not a typo)
* 2025-03-30T01 (HH)
* 2025-03-30 01 (same as above)
* 2025-03-30 01 (two or more spaces is also acceptable)
In my opinion Pandas doesn't go far enough. Here's an example from real customer data I've seen in the past that Pandas doesn't parse.
* 2025-03-30+00:00 (this is very sensible in my opinion. Unless there's a deeper theoretical regex pattern conflicts with other parts of the ISO format)
Here's an example that isn't decipherable under a flexible ISO interpretation and shouldn't be supported.
* 2025-30-03 (theoretically you can infer that 30 is a day, and 03 is month. BUT you shouldn't accept this. Pandas used to allow such things. I believe they no longer do)
I understand writing these flexible regexes or if-else statements will hurt your benchmarks and will be painful to maintain. Maybe release them under an new call like `parse_best_effort` (or even `youre_welcome`) and document pitfalls and performance degradation. Trust me, I'd rather use a reliable generic but slow parser than spend hours writing a write a god awful regex that I will only use once (I've spent literal weeks writing regexes and fixes in the last decade).
Pandas has been around since 2012 dealing with customer data. They have seen it all and you can learn a lot from them. ISOs and RFCs when it comes to timestamps don't mean squat. If possible try to make Whenever useful rather than fast or pure. I'd rather use a slimmer faster alternative to pandas for parsing Timestamps if one is available but there aren't any at the moment.
If time permits I'll try to compile a non exhaustive list of real world timestamp formats and post in the issue.
Thank you for your work!
P.S. seeing BurntSushi in the GitHub issue gives me imposter syndrome :)
I honestly do not know the answer to that question myself. But I wouldn't necessarily look to Pandas as the shining beacon on a hill here. Not because Pandas is doing anything wrong per se, but because it's a totally different domain and use case. On the one hand, you have a general purpose library that needs to consider all of its users for all general purpose datetime use cases. On the other hand, you have a data scienc-y library designed for trying to slurp up and make sense of messy data at scale. There may be things that make sense in the latter that don't in the former.
In particular, a major gap in your reasoning, from what I can see, is that constraints beget better error reporting. I don't know how to precisely weigh error reporting versus flexible parsing, but there ought to be some deliberation there. The more flexible your format, the harder it is to give good error messages when you get invalid data.
Moreover, "flexible parsing" doesn't actually have to be in the datetime library. The task of flexible parsing is not, in and of itself, overtly challenging. It's a tedious task that can be build on top of the foundation of a good datetime library. I grant that this is a bit of a cop-out, but it's part of the calculus when designing ecosystem libraries like this.
Speaking for me personally (in the context of Jiff), something I wouldn't mind so much is adding a dedicated "flexible" parsing mode that one can opt into. But I don't think I'd want to make it the default.