I prefer a slightly more general rule: Make Illegal States Unrepresentable. The "Parse, don't validate" rule is a special case of MISU.
What's the difference? MISU applies even when there's no parsing-like transformation happening. For example, if you have a variable that represents the current state of a network connection, and let's say it can be Disconnected, or Connected to some IP address (this is an oversimplification).
Then one way to do it would be
struct {
connected: bool,
peer_ip: int32
}
The trouble is that this allows us to represent an illegal/meaningless state: we're disconnected but there's still some junk old peer_ip hanging in there. Even worse, we might have written
Now we could have connected = true but peer_ip = None.
The solution is to use a sum type:
type connection =
Disconnected
| Connected of int32
(sorry for using made-up syntax; I hope it's clear to anyone familiar with Rust.)
"Make Illegal States Unrepresentable" applies throughout your program, at every interface between modules or functions in the program, including but not limited to parsing input.
You'd have to manually implement the traits to support the ergonomics of slices and iteration and costly reallocation if you need to pass ownership as a Vec:
I'd expect:
pub struct NonEmpty<T> {
v: Vec<T>,
}
The constructor would enforce the invariant and then you'd impl Deref and DerefMut for [T] to gain normal len/is_empty/indexing/iteration, passing as &[T] to other funcs and mutating values (which can't break the invariant).
To mutate length while preserving the invariant it's dealers choice e.g.
- add .into_vec() for unwrap/mutate/rewrap
- add invariant preserving mutators of your choice
Not sure switching between languages and proofs makes for a compelling argument:
"Look how easy it is to accidentally bypass the invariant of a rust newtype by transliterating the data shape into Haskell and deriving a new type". Uh, ok.
If comparing the "risk of accident" between a newtype wrapper whose only role is enforcing the the invariant versus manually reimplementing vector and iterator semantics to use a different layout... I'd say the newtype wins that.
It would be good advice to keep a newtype that enforces invariant as a single purpose primitive type. A building block and not a place to add other features.
There might be times I'd prefer structural enforcement e.g. something serialisation related. Converting into a non-rust format is what they are doing in their "accident"!
Deref/DerefMut enables implicit type coercion rather than exposing an interface. You can choose the target type and immutable/mutable but not parts of the target type.
You can use all the slice reference methods (that do not require ownership) with:
If you DerefMut to a Vec then you won't be able to preserve the invariant.
If you want control over methods to expose then you need wrapper methods for those you want. If you want to expose some of the traits the inner type implements then there are likely derive macros available e.g. with derive_more you could expose just indexing as:
This is great. Alexis King actually stated that, had she known how popular “Parse, Don’t Validate” had been, she would have written it in a language more widely used than Haskell.
With its crap type system Python might be even better, in a strange way.
Haskell's strong, static, non-reflective type system tends to make "parse, don't validate" produce code that also looks nicer. Which is great. So great that it steals a bit of the main message's valor.
In Python, though, it's really easy to just let your data be a dynamically typed list of dicts forever. So easy that parsing into something more strongly typed looks like a whole lot of extra effort. Upon looking at that sort of thing many a working Python programmer, myself included, hears the voice of GvR murmuring disparaging things about "academic" programmers down in the pit of their brain.
Which creates an opportunity to demonstrate all the ways the (arguably) more Pythonic way is actually a royal PITA when you try to make your code robust. Handling and reporting data validity errors gets scattered all over the code, which makes it annoying to maintain. Unit test suites get bloated because it's not obvious what inputs a function should be able to handle. Comments and docstrings to help keep track of this stuff begin to proliferate.
I’m a big fan of “Parse dont validate” and I feel like injecting that approach into python has so many benefits.
there’s all the code correctness stuff that we all love. But the biggest benefit is making it so much easier to maintain other parts of the stack.
Following a stack trace into somebody else’s functions and you see that the args are completely untyped or just a bunch of ‘dict[str, Any]’ is the worst. But if you see those inputs as more narrowly typed data classes it makes it so much easier to grok what the function is supposed to do
I write Python. While I read the original post and nodded along, nothing changed about how I worked until I started playing with Rust. Seeing a strongly typed language force the constraints and stop stupid errors was when the rationale finally clicked for me. Knowing that this function will never blow-up in an unexpected way changes how you think about problems.
You otherwise get accustomed to some wishy-washy blobs of data that get passed around and find it normal. Maybe you include some ad hoc guard rails here and there, which catches the egregious errors, but there is always some lingering uncertainty. Some string that should have been an int, missing key here, the object which never had the validation check, etc
It is like unit-testing - more-up front work, but I could never go back to a world where I did not get these automated assurances. Sadly, I am a grug-brain which could only feel the lesson from personal experience.
Good article on Rust's strengths with types. If you like this, you may be curious how you might build your own parse capabilities. I like the Rust crates Winnow and Nom, and also the Rust traits From and Into.
Another take, from the primary example: This is what `unwrap()` is for. I understand that the author is looking at this from a correctness and safety(?) perspective. For practical purposes, I would unwrap here. If it's less trivial than the example, unwrap with a comment explaining why it's fine.
Another angle: Unfortunately, the `first()` method being fallible here is just an issue of using an imperfect method/datatype here. This is where the author gets in to a non-empty-vec custom type. Then you are balancing using a more correct type that takes custom wiring vs a std lib thing everyone understands and takes no setup. I would lean towards this setup if I were using this non_empty_vec.first() unwrap pattern a number of times in the code base; then the setup would be worth it, at least for my own code bases. If I were exposing this in a lib others would use, I would keep the standard Vec so as to be more transparent for others.
In both views: "This is what unwrap is for" does it for me in all cases I've encountered to date. Maybe for aerospace or safety critical systems, I would have a different take.
A third take: I notice this trend in the rust community. It's not my cup of tea. Keep things simple, easy to maintain, and don't let "correctness" get in the way. In this example, I don't think it gets in the way, but I have seen this mindset lead to it getting in the way, especially in embedded, where mapping the Owernership model to hardware ends up in messy patterns and surprising assertions about embedded-101 concepts like DMA being "unsolved", "no good way", "difficult" etc.
Rust provides tools to make sure specific logic is correct if it passes the compiler. People sometimes go overboard and assume you have to type-maxx your code, regardless of complexity added by doing so.
I was thinking about its implementation, too, and ended up deciding that it was probably a performance optimization.
`Vec<T>` stores all data on the heap, so getting anything out of it involves a pointer deref and possibly also an array bounds check. This `NonEmpty<T>` type keeps the first element of the list in a location that supports some low-level optimization that might make a significant difference in situations where accessing the first element is much more common than accessing subsequent elements.
I lamented patterns of what I consider over-use of the type system in Rust (This applies to other languages as well). There is a balance: I prefer this over-use/strict use (e.g. implementing a custom non-empty Vec type) over not using typing at all, but a big margin!
For a untype language like js array, since it can be empty, you have to either always check the length, the item returned, or have a precondition to know the array is not empty.
All three of those cases is either code or context you’re holding in your head.
I prefer a slightly more general rule: Make Illegal States Unrepresentable. The "Parse, don't validate" rule is a special case of MISU.
What's the difference? MISU applies even when there's no parsing-like transformation happening. For example, if you have a variable that represents the current state of a network connection, and let's say it can be Disconnected, or Connected to some IP address (this is an oversimplification).
Then one way to do it would be
The trouble is that this allows us to represent an illegal/meaningless state: we're disconnected but there's still some junk old peer_ip hanging in there. Even worse, we might have written Now we could have connected = true but peer_ip = None.The solution is to use a sum type:
(sorry for using made-up syntax; I hope it's clear to anyone familiar with Rust.)"Make Illegal States Unrepresentable" applies throughout your program, at every interface between modules or functions in the program, including but not limited to parsing input.
In rust this would simply be
Not sure that type is good advice:
You'd have to manually implement the traits to support the ergonomics of slices and iteration and costly reallocation if you need to pass ownership as a Vec:I'd expect:
The constructor would enforce the invariant and then you'd impl Deref and DerefMut for [T] to gain normal len/is_empty/indexing/iteration, passing as &[T] to other funcs and mutating values (which can't break the invariant).To mutate length while preserving the invariant it's dealers choice e.g.
- add .into_vec() for unwrap/mutate/rewrap
- add invariant preserving mutators of your choice
There's a whole follow up post about this: https://lexi-lambda.github.io/blog/2020/11/01/names-are-not-...
Not sure switching between languages and proofs makes for a compelling argument:
"Look how easy it is to accidentally bypass the invariant of a rust newtype by transliterating the data shape into Haskell and deriving a new type". Uh, ok.
If comparing the "risk of accident" between a newtype wrapper whose only role is enforcing the the invariant versus manually reimplementing vector and iterator semantics to use a different layout... I'd say the newtype wins that.
It would be good advice to keep a newtype that enforces invariant as a single purpose primitive type. A building block and not a place to add other features.
There might be times I'd prefer structural enforcement e.g. something serialisation related. Converting into a non-rust format is what they are doing in their "accident"!
Which deref must I use to get most of the existing interface sans `retain()`?
Deref/DerefMut enables implicit type coercion rather than exposing an interface. You can choose the target type and immutable/mutable but not parts of the target type.
You can use all the slice reference methods (that do not require ownership) with:
https://doc.rust-lang.org/std/primitive.slice.htmlIf you DerefMut to a Vec then you won't be able to preserve the invariant.
If you want control over methods to expose then you need wrapper methods for those you want. If you want to expose some of the traits the inner type implements then there are likely derive macros available e.g. with derive_more you could expose just indexing as:
This is great. Alexis King actually stated that, had she known how popular “Parse, Don’t Validate” had been, she would have written it in a language more widely used than Haskell.
With its rich type system, Haskell is the perfect language to demonstrate the dictum.
With its crap type system Python might be even better, in a strange way.
Haskell's strong, static, non-reflective type system tends to make "parse, don't validate" produce code that also looks nicer. Which is great. So great that it steals a bit of the main message's valor.
In Python, though, it's really easy to just let your data be a dynamically typed list of dicts forever. So easy that parsing into something more strongly typed looks like a whole lot of extra effort. Upon looking at that sort of thing many a working Python programmer, myself included, hears the voice of GvR murmuring disparaging things about "academic" programmers down in the pit of their brain.
Which creates an opportunity to demonstrate all the ways the (arguably) more Pythonic way is actually a royal PITA when you try to make your code robust. Handling and reporting data validity errors gets scattered all over the code, which makes it annoying to maintain. Unit test suites get bloated because it's not obvious what inputs a function should be able to handle. Comments and docstrings to help keep track of this stuff begin to proliferate.
I’m a big fan of “Parse dont validate” and I feel like injecting that approach into python has so many benefits.
there’s all the code correctness stuff that we all love. But the biggest benefit is making it so much easier to maintain other parts of the stack.
Following a stack trace into somebody else’s functions and you see that the args are completely untyped or just a bunch of ‘dict[str, Any]’ is the worst. But if you see those inputs as more narrowly typed data classes it makes it so much easier to grok what the function is supposed to do
I write Python. While I read the original post and nodded along, nothing changed about how I worked until I started playing with Rust. Seeing a strongly typed language force the constraints and stop stupid errors was when the rationale finally clicked for me. Knowing that this function will never blow-up in an unexpected way changes how you think about problems.
You otherwise get accustomed to some wishy-washy blobs of data that get passed around and find it normal. Maybe you include some ad hoc guard rails here and there, which catches the egregious errors, but there is always some lingering uncertainty. Some string that should have been an int, missing key here, the object which never had the validation check, etc
It is like unit-testing - more-up front work, but I could never go back to a world where I did not get these automated assurances. Sadly, I am a grug-brain which could only feel the lesson from personal experience.
Good article on Rust's strengths with types. If you like this, you may be curious how you might build your own parse capabilities. I like the Rust crates Winnow and Nom, and also the Rust traits From and Into.
Another take, from the primary example: This is what `unwrap()` is for. I understand that the author is looking at this from a correctness and safety(?) perspective. For practical purposes, I would unwrap here. If it's less trivial than the example, unwrap with a comment explaining why it's fine.
Another angle: Unfortunately, the `first()` method being fallible here is just an issue of using an imperfect method/datatype here. This is where the author gets in to a non-empty-vec custom type. Then you are balancing using a more correct type that takes custom wiring vs a std lib thing everyone understands and takes no setup. I would lean towards this setup if I were using this non_empty_vec.first() unwrap pattern a number of times in the code base; then the setup would be worth it, at least for my own code bases. If I were exposing this in a lib others would use, I would keep the standard Vec so as to be more transparent for others.
In both views: "This is what unwrap is for" does it for me in all cases I've encountered to date. Maybe for aerospace or safety critical systems, I would have a different take.
A third take: I notice this trend in the rust community. It's not my cup of tea. Keep things simple, easy to maintain, and don't let "correctness" get in the way. In this example, I don't think it gets in the way, but I have seen this mindset lead to it getting in the way, especially in embedded, where mapping the Owernership model to hardware ends up in messy patterns and surprising assertions about embedded-101 concepts like DMA being "unsolved", "no good way", "difficult" etc.
Rust provides tools to make sure specific logic is correct if it passes the compiler. People sometimes go overboard and assume you have to type-maxx your code, regardless of complexity added by doing so.
Nonempty type wrappers are a stark reminder that we are missing out on refinement types.
I was thinking about its implementation, too, and ended up deciding that it was probably a performance optimization.
`Vec<T>` stores all data on the heap, so getting anything out of it involves a pointer deref and possibly also an array bounds check. This `NonEmpty<T>` type keeps the first element of the list in a location that supports some low-level optimization that might make a significant difference in situations where accessing the first element is much more common than accessing subsequent elements.
Rust has a proposal for this, in the form of pattern types. It's on nightly, but quite far from being ready for prime time.
...I'm so happy that I write in untyped languages so I don't have to deal with any of this.
I lamented patterns of what I consider over-use of the type system in Rust (This applies to other languages as well). There is a balance: I prefer this over-use/strict use (e.g. implementing a custom non-empty Vec type) over not using typing at all, but a big margin!
Balance in all things?
But you kind of do.
For a untype language like js array, since it can be empty, you have to either always check the length, the item returned, or have a precondition to know the array is not empty.
All three of those cases is either code or context you’re holding in your head.
All that stuff is equivalent to a type system
Congratulations on having to implement your checks at runtime, but stupid, repetitive, boring work is kind of why we invented compilers.