Parse, Don't Validate (2019) (opens in new tab)

I've found this to be simply a matter of experience, not tooling. As the years go by I find the majority of my code just working right - never touched anything like pydantic or validation boilerplate for my own code, besides having to write unit tests as an afterthought at work to keep the coverage metric up.

Sounded familiar. Worked correctly the first time it was run, sans types.

https://www.python.org/about/success/esr/

Curious, but how does pydantic compare to marshmallow?

I'm currently using marshmallow in a project, specifically using the functionality that builds parsers from dataclasses.

I was curious what the differences were.

I've also found Pydantic to be a valuable library to use.

However, it does have a strongly opinionated approach to casting that can sometimes yield non-obvious results. This behavior is documented and I would suggest new potential adopters of the library to explore this casting/coerce feature in the context of your product/app requirements.

For the most part, it's not an huge issue, but I've run into a few surprising cases. For example, sys.maxint, 0, '-7', 'inf', float('-inf') are all valid datetime formats.

- https://pydantic-docs.helpmanual.io/usage/models/#data-conve... - https://gist.github.com/mpkocher/30569c53dc3552bc5ad73e09b48...

Just started working on a new SaaS startup and using FastAPI & Pydantic. The development experience has been great.

Can't you do this already with Python type annotations? I am a fan of typing in general (not just for data model as this seems to be) and using types everywhere saves a lot of debugging hassle and even allows for catching some bugs with static analysis.

There is one thing concerns me about type annotation. With annotation Python code looks like Java code, but with it I don't have advantages of Java (like speed). Besides I see things like `x: typing.Union[str, typing.Any]` - maybe better no annotation.

We've had a similar experience using pydantic. We integrated it quite tightly with a django project and it's been awesome.

I agree. It's a very well-written post. I am not a Haskell person, but it was quite clear to me.

I feel that we don't put enough value, these days, on the ability to write clear, articulate exposition. Also, I believe that many people are not willing to read articles, books, or papers, of any meaningful length.

Everything needs to be boiled down to <10 min. read time, or <18 min. TED talks.

Those are the extras. This is the post:

https://fsharpforfunandprofit.com/posts/recipe-part2/

I think the article goes a little further than what you describe -- it would have you use a strong type that cannot represent illegal values.

There's a follow-up article by the same author (that I unfortunately can't find), in which she explains this point.

As an example, returning a NonZero newtype over Int is not as type safe as using an ADT that lacks a zero value altogether. Using a NonEmpty newtype over List is not as type safe as using the NonEmpty ADT that has an element as part of its structure.

Basically newtype still has use, but it is not as airtight as a well-designed ADT.

> This principle can be applied to dynamic languages as well if you have some mechanism such as type hinting, pre-conditions etc. that are checked by a linter during development, even if it isn't, you can still use it at runtime with sufficient error handling.

Every time I read something like this my mind translates it to "after building an ad-hoc compiler you can do all the things a compiler can do. Just not as well, but you can do it." -- Same with "I don't need a compiler, my tests stop all this kind of bugs"

I think the original formulation is "make illegal states unrepresentable" from Yaron Minsky: https://blog.janestreet.com/effective-ml/ and https://blog.janestreet.com/effective-ml-revisited/ (or maybe there are older sources than 2010?)

I agree but I also think it’s on the right path. This seems partially like a “why Haskell” in disguise to me.

I’ve run across DSLs that have three or more layers of parsing and validation. Embedding different languages within each other (eg: JSON snippets within your own DSL) definitely leads to the issues the article talks about.

Also, growing your own parser without understanding standard lexer/parser basics seems far more common than it ought to be. I’m not talking brilliant design, rather the extremely naive one-character-at-a-time-in-a-really-complex-loop variety of design.

The better level of bad is, “I know what lexers/parsers are, now I’ll write something to basically implement a type-checking parser with the lexed+parsed tree as input.”

This article is basically stating, “Why not just get your parser to do it all for you in one swell foop?” When I have refactored code to follow this kind of design, I have never regretted the outcome.

I don't think types are a red herring here. Because if you follow this advice, then just using logic and your source code, you can prove what data is valid. And, since types and (constructive) logic are so strongly related, then the types are, in some sense "there" even if you don't see them. To put it another way, it's nice if your computer can make the proofs but if it can't, does that make the theorems any less true?

The advantage of the type checker is that it automatically check types for you. If you have only one function that produces a parsedArray and multiple functions that accept a parsedArray, you can be sure where they come from.

I wish it had actual proper examples. I've no idea how to use that.

To everyone in this subthread: sigbovik is a conference published every 1st of April.

This paper is an April's fool joke. I didn't think people could take that one seriously. I guess it's a good April's fool then. :)

Software engineers like efficiently running software. Mathematicians like beautiful definitions. Scientists like non-trivial discoveries.

This paper.... uh.... what exactly is it good for?

I suppose it could be kind of nice as some kind of undergraduate paper writing project kind of thing but it looks too professional for that.... I am kind of at a loss why this was written. Maybe it is some strange kind of satire....

The point being, the converse of ‘parsing is validation’ is not true.

For js / typescript I like: https://github.com/paperhive/fefe

basically ist’s just functions that take a value of one type and return a other one

How does Maat compare with pydantic? https://github.com/samuelcolvin/pydantic

I have the same feeling after spending a few years with Haskell, StandardML, Agda, Idris, Coq, etc.

One trick I've found very useful is to realise that Maybe (AKA Option) can be though of as "a list with at most one element". Dynamic languages usually have some notion of list/array, which we can use as if it were a Maybe/Option type; e.g. we can follow a 'parse, don't validate' approach by wrapping a "parsed" result in a list, and returning an empty list otherwise. This allows us to use their existing 'map', 'filter', etc. too ;)

(This is explored in more detail, including links to logic programming, in https://link.springer.com/chapter/10.1007%2F3-540-15975-4_33 )

If we want to keep track of useful error messages, I've found Scala's "Try" type to be useful ('Try[T]' is isomorphic to 'Either Throwable T'). Annoyingly, built-in sum type; the closest thing is usually a tagged pair like '[true, myFoo]'/'[false, myException]', which is pretty naff.

For what its worth: People don't use dynamic language because they don't know better or never used a static language. To better understand what dynamic languages bring to the table, here are some disadvantages of static types to consider:

Static types are awesome for local reasoning, but they are not that helpful in the context of the larger system (this already starts at the database, see idempotency mismatch).

Code with static types is sometimes larger and more complex than the problem its trying to solve

They tightly couple data to a type system, which (can) introduce incidental complexity >(I'm still waiting for pattern matching + algebraic data types) This is a good example, if you pattern match to a specific structure (e.g. position of fields in your algebraic data type), you tightly coupled your program to this particular structure. If the structure change, you may have to change all the code which pattern matches this structure.

Plenty of people know multiple statically typed and dynamic languages, and multiple functional, imperative, and other languages; and use dynamic languages for some things but not other things. The set of people using dynamic languages isn't just "those that haven't had their eyes opened yet to what static languages can do". Different languages and paradigms make different things easier.

I do believe that, for long lasting, larger projects, static typing tends to make the code easier to maintain as time goes on. But not every project is like that. In fact, not every project uses a single language. Some use statically typed languages for some parts, and dynamically typed for others (this is common in web dev).

> It's weird how long it's taking for people to rediscover why strong static types were a good idea.

I don't think it's weird. Most of those languages were not popular in industry for various reasons, and the ones that were (especially in say, the 90s) did not have particularly capable static type systems. The boilerplate/benefit ratio was all off.

The way I describe this dichotomy personally is, I would rather use Ruby than Java 1.5. I would rather use Rust than Ruby. (Java 1.5 is the last version of Java I have significant development experience in, and they've made the type system much more capable since those days.)

Yeah, I remember I used to get frustrated when I had to read code that used map() or even .forEach() extensively, thinking a simple, imperative for loop would suffice. I slowly came to realize that a for loop gives you too much power. It's a hammer. It holds the place of a bug you just haven't written yet. Now I'm the one writing JavaScript like it's Haskell. Although Haskell could learn a thing or two from TypeScript about row polymorphism.

This matches my personal experience, for sure. I started out writing Python (pre-type-annotations) and JS, with lots of “raw” dicts/objects. For the past ~7 years though, I’ve written mostly Scala, but also a decent amount of TypeScript, Go and Java, and it’s completely transformed how I code, dramatically for the better.

Now, even in the rare case where I write some Python, JS or PHP, I write it in a very statically typed style, immediately parsing input into well-thought-out domain classes. And for backend services, I almost always go with 3 layers of models:

1) Data Transfer Objects. Map directly to the wire format, e.g. JSON or Protobuf. Generally auto-generated from API specs, e.g. using Open API Generator or protoc. A good API spec + code gen handles most input validation well

2) Domain Objects. Hand written, purely internal to the backend service, faithfully represent the domain. The domain layer of my code works exclusively with these. Sometimes there’s a little more validation when transforming a DTO into a domain model

3) Data Access Objects. Basically a representation of DB tables. Generally auto-generated from DB schemas, e.g. using libs like Prisma for TS or SQLBoiler for Go

Can’t imagine going back to the “everything is a dictionary” style for any decent sized project, it becomes such a mess so quickly. This style is a little more work up front, when you first write the code, but WAYYYYYY easier to maintain over time, fewer bugs and easier to modify quickly and confidently, with no nasty coupling of your domain models to either DB or wire format concerns. And code gen for the DTO and DAO layers makes it barely more up-front work.

> It's weird how long it's taking for people to rediscover why strong static types were a good idea.

Its weird how long its taken for languages with static typing and type systems designed for correctness (and designed well for that end) rather than princupally for convenience of compilation to be available that are generally usable (considering licensing model, features, ecosystem, etc.)

"It's weird how long it's taking for people to rediscover why strong static types were a good idea." It sounds like you're projecting your limited experience eg you've dismissed all Ms dev. Many people have been using strong static for decades. The benefits have never been out of sight. Many use dynamic, but many have always used strong static.

> > IME, people in dynamic languages almost never program this way, though—they prefer to use validation

I wonder how many people the author met.

Every programming paradigm is a good idea if the respective trade-offs are acceptable to you.

For example, one good reason why strong static types are a bad idea... they prevent you from implementing dynamic dispatch.

Routers. You can't have routers.

You can use the now widely adopted Great Expections[0] library, which fits exactly this use-case for data validation!

[0] https://greatexpectations.io

I think that is in fact validating in the sense that the article means it.

Here's validating a CSV in Python (which I'm using because it's a language that's, well, less excited about types than the author's choice of Haskell, to show that the principle still applies):

    def validate_data(filename):
        reader = csv.DictReader(open(filename))
        for row in reader:
            try:
                date = datetime.datetime.fromisoformat(row["date"])
            except ValueError:
                print("ERROR: Invalid date", row)
            if date < datetime.datetime(2021, 1, 1):
                print("ERROR: Last year's data", row))
            # etc.
        return errors

    def actually_work_with_data(filename):
        reader = csv.DictReader(open(filename))
        for row in reader:
            try:
                date = datetime.datetime.fromisoformat(row["date"])
            except ValueError:
                raise Exception("Wait, didn't you validate this already???")
            # etc.

Yes, it's a kind of silly example, but - the validation routine is already doing the work of getting the data into the form you want, and now you have some DRY problems. What happens if you start accepting additional time formats in validate_data but you forget to teach actually_work_with_data to do the same thing?

The insight is that the work of reporting errors in the data is exactly the same as the work of getting non-erroneous data into a usable form. If a row of data doesn't have an error, that means it's usable; if you can't turn it into a directly usable format, that necessarily means it has some sort of error.

So what you want is a function that takes the data and does both of these at the same time, because it's actually just a single task.

In a language like Haskell or Rust, there's a built-in type for "either a result or an error", and the convention is to pass errors back as data. In a language like Python, there isn't a similar concept and the convention is to pass errors as exceptions. Since you want to accumulate all the errors, I'd probably just put them into a separate list:

    @attr.s # or @dataclasses.dataclass, whichever
    class Order:
        name: str
        date: datetime.datetime
        ...

    def parse(filename):
        data = []
        errors = []
        reader = csv.DictReader(open(filename))
        for row in reader:
            try:
                date = datetime.datetime.fromisoformat(row["date"])
            except ValueError:
                errors.append(("Invalid date", row))
                continue
            if date < datetime.datetime(2021, 1, 1):
                errors.append(("Last year's data", row))
                continue
            # etc.
            data.append(Order(name=row["name"], date=date, ...))
        return data, errors

And then all the logic of working with the data, whether to actually use it or to report errors, is in one place. Both your report of bad data and your actually_work_with_data function call the same routine. Your actual code doesn't have to parse fields in the CSV itself; that's already been done by what used to be the validation code. It gets a list of Order objects, and unlike a dictionary from DictReader, you know that an Order object is usable without further checks. (The author talks about "Use a data structure that makes illegal states unrepresentable" - this isn't quite doable in Python where you can generally put whatever you want in an object, but if you follow the discipline that only the parse() function generates new Order objects, then it's effectively true in practice.)

And if your file format changes, you make the change in one spot; you've kept the code DRY.

I thought of input validation for web forms. Similar thing I guess. In Haskell you can create a type that you know is a validated email address but you still need a validation function from String -> Maybe Email to actually validate it at runtime

Data validation?

You seem to suggest that it's possible to parse without validating, which I'm not sure I follow. Surely validation is just one of the phases or steps of parsing?

The point is that validation is (or should/can be) a byproduct of parsing. I.e., you shouldn't "do both", rather the validation should be encompassed by the parsing, as much as it makes sense.

The post is saying:

- don't drop the info gathered from checks while validating, but keep track of it

- if you do this, you'll effectively be parsing

- parsing is more powerful that validating

"Extra steps" would be keeping track of info gathered from checks.

Validation is checking if something looks like a data structure. Parsing is smashing data into a data structure, and failing out if you can't do it.

At the end of parsing, you have a structure with a type. After validation, you may or may not have a structure with that type, depending on how you chose to validate.

But I think the big win is, parsers are usually much easier to compose (since they themselves are structured functions) and so if you start with the type first, you often get the "validation" behavior aspect of parsing for "free" (usually from a library). Maybe title should have been "Parse, don't manually validate."

But if your type doesn't catch all your invariants, yeah it does feel kinda just like validation.

It's more like putting the "extra steps" at the right place in the code.