Pipelining might be my favorite programming language feature

(herecomesthemoon.net)

186 points | by Mond_ 8 hours ago ago

163 comments

  • Straw 3 hours ago ago

    Lisp macros allow a general solution to this that doesn't just handle chained collection operators but allows you to decide the order in which you write any chain of calls.

    For example, we can write: (foo (bar (baz x))) as (-> x baz bar foo)

    If there are additional arguments, we can accommodate those too: (sin (* x pi) as (-> x (* pi) sin)

    Where expression so far gets inserted as the first argument to any form. If you want it inserted as the last argument, you can use ->> instead:

    (filter positive? (map sin x)) as (->> x (map sin) (filter positive?))

    You can also get full control of where to place the previous expression using as->.

    Full details at https://clojure.org/guides/threading_macros

  • bnchrch 7 hours ago ago

    I'm personally someone who advocates for languages to keep their feature set small and shoot to achieve a finished feature set quickly.

    However.

    I would be lying if I didn't secretly wish that all languages adopted the `|>` syntax from Elixir.

    ```

    params

    |> Map.get("user")

    |> create_user()

    |> notify_admin()

    ```

    • Cyykratahk 7 hours ago ago

      We might be able to cross one more language off your wishlist soon, Javascript is on the way to getting a pipeline operator, the proposal is currently at Stage 2

      https://github.com/tc39/proposal-pipeline-operator

      I'm very excited for it.

      • TehShrike 6 hours ago ago

        I was excited for that proposal, but it veered off course some years ago – some TC39 members have stuck to the position that without member property support or async/await support, they will not let the feature move forward.

        It seems like most people are just asking for the simple function piping everyone expects from the |> syntax, but that doesn't look likely to happen.

        • packetlost 6 hours ago ago

          I don't actually see why `|> await foo(bar)` wouldn't be acceptable if you must support futures.

          I'm not a JS dev so idk what member property support is.

          • cogman10 an hour ago ago

            Seems like it'd force the rest of the pipeline to be peppered with `await` which might not be desirable

                "bar"
                |> await getFuture(%);
                |> baz(await %);
                |> bat(await %);
            
            My guess is the TC committee would want this to be more seamless.

            This also gets weird because if the `|>` is a special function that sends in a magic `%` parameter, it'd have to be context sensitive to whether or not an `async` thing happens within the bounds. Whether or not it does will determine if the subsequent pipes are dealing with a future of % or just % directly.

            • packetlost 40 minutes ago ago

              It wouldn't though? The first await would... await the value out of the future. You still do the syntactic transformation with the magic parameter. In your example you're awaiting the future returned by getFuture twice and improperly awaiting the output of baz (which isn't async in the example).

              In reality it would look like:

                  "bar"
                  |> await getFuture()
                  |> baz()
                  |> await bat()
              
              (assuming getFuture and bat are both async). You do need |> to be aware of the case where the await keyword is present, but that's about it. The above would effectively transform to:

                  await bat(baz(await getFuture("bar")));
              
              I don't see the problem with this.
              • porridgeraisin 11 minutes ago ago

                Correct me if I'm wrong, but if you use the below syntax

                  "bar"
                    |> await getFuture()
                
                How would you disambiguate it from your intended meaning and the below:

                  "bar"
                    |> await getFutureAsyncFactory()
                
                Basically, an async function that returns a function which is intended to be the pipeline processor.

                Typically in JS you do this with parens like so:

                (await getFutureAsyncFactory())("input")

                But the use of parens doesn't transpose to the pipeline setting well IMO

      • chilmers 4 hours ago ago

        It also has barely seen any activity in years. It is going nowhere. The TC39 committee is utterly dysfunctional and anti-progress, and will not let any this or any other new syntax into JavaScript. Records and tuples has just been killed, despite being cited in surveys as a major missing feature[1]. Pattern matching is stuck in stage 1 and hasn't been presented since 2022. Ditto for type annotations and a million other things.

        Our only hope is if TypeScript finally gives up on the broken TC39 process and starts to implement its own syntax enhancements again.

        [1] https://2024.stateofjs.com/en-US/usage/#top_currently_missin...

      • zdragnar 7 hours ago ago

        I worry about "soon" here. I've been excited for this proposal for years now (8 maybe? I forget), and I'm not sure it'll ever actually get traction at this point.

      • hinkley 3 hours ago ago

        All of their examples are wordier than just function chaining and I worry they’ve lost the plot somewhere.

        They list this as a con of F# (also Elixir) pipes:

            value |> x=> x.foo()
        
        The insistence on an arrow function is pure hallucination

            value |> x.foo()
        
        Should be perfectly achievable as it is in these other languages. What’s more, doing so removes all of the handwringing about await. And I’m frankly at a loss why you would want to put yield in the middle of one of these chains instead of after.
      • hoppp 7 hours ago ago

        Cool I love it, but another thing we will need polyfills for...

        • hathawsh an hour ago ago

          I believe you meant to say we will need a transpiler, not polyfill. Of course, a lot of us are already using transpilers, so that's nothing new.

        • bobbylarrybobby 7 hours ago ago

          How do you polyfill syntax?

          • jononor 7 hours ago ago

            Letting your JS/TS compiler convert it into supported form. Not really a polyfill, but it allows to use new features in the source and still support older targets. This was done a lot when ES6 was new, I remember.

            • zdragnar 6 hours ago ago

              Polyfills are for runtime behavior that can't be replicated with a simple syntax transformation, such as adding new functions to built-in objects like string.prototype contains or the Symbol constructor and prototype or custom elements.

              I haven't looked at the member properties bits but I suspect the pipeline syntax just needs the transform to be supported in build tools, rather than adding yet another polyfill.

    • valenterry 7 hours ago ago

      I prefer Scala. You can write

      ``` params.get("user") |> create_user |> notify_admin ```

      Even more concise and it doesn't even require a special language feature, it's just regular syntax of the language ( |> is a method like .get(...) so you could even write `params.get("user").|>(create_user) if you wanted to)

      • elbasti 7 hours ago ago

        In elixir, ```Map.get("user") |> create_user |> notify_admin ``` would aso be valid, standard elixir, just not idiomatic (parens are optional, but preferred in most cases, and one-line pipes are also frowned upon except for scripting).

        • MaxBarraclough 3 hours ago ago

          With the disclaimer that I don't know Elixir and haven't programmed with the pipeline operator before: I don't like that special () syntax. That syntax denotes application of the function without passing any arguments, but the whole point here is that an argument is being passed. It seems clearer to me to just put the pipeline operator and the name of the function that it's being used with. I don't see how it's unclear that application is being handled by the pipeline operator.

          Also, what if the function you want to use is returned by some nullary function? You couldn't just do |> getfunc(), as presumably the pipeline operator will interfere with the usual meaning of the parentheses and will try to pass something to getfunc. Would |> ( getfunc() ) work? This is the kind of problem that can arise when one language feature is permitted to change the ordinary behaviour of an existing feature in the name of convenience. (Unless of course I'm just missing something.)

    • hinkley 3 hours ago ago

      The pipe operator relies on the first argument being the subject of the operation. A lot of languages have the arguments in a different order, and OO languages sometimes use function chaining to get a similar result.

      • Terr_ 2 hours ago ago

        IIRC the usual workaround in Elixir involves be small lambda that rearranges things:

            "World"
            |> then(&concat("Hello ", &1))
        
        
        I imagine a shorter syntax could someday be possible, where some special placeholder expression could be used, ex:

            "World"
            |> concat("Hello ", &1)
        
        However that creates a new problem: If the implicit-first-argument form is still permitted (foo() instead of foo(&1)) then it becomes confusing which function-arity is being called. A human could easily fail to notice the absence or presence of the special placeholder on some lines, and invoke the wrong thing.
        • hinkley an hour ago ago

          Yeah I really hate that syntax and I can’t even explain why so I kind of blot it out, but you’re right.

          My dislike does improve my test coverage though, since I tend to pop out a real method instead.

      • BoingBoomTschak 18 minutes ago ago
    • jasperry 5 hours ago ago

      Yes, a small feature set is important, and adding the functional-style pipe to languages that already have chaining with the dot seems to clutter up the design space. However, dot-chaining has the severe limitation that you can only pass to the first or "this" argument.

      Is there any language with a single feature that gives the best of both worlds?

      • bnchrch 5 hours ago ago

        FWIW you can pass to other arguments than first in this syntax

        ```

        params

        |> Map.get("user")

        |> create_user()

        |> (&notify_admin("signup", &1)).() ```

        or

        ```

        params

        |> Map.get("user")

        |> create_user()

        |> (fn user -> notify_admin("signup", user) end).() ```

        • Terr_ 2 hours ago ago

          BTW, there's a convenience macro of Kernel.then/2 [0] which IMO looks a little cleaner:

              params
              |> Map.get("user")
              |> create_user()
              |> then(&notify_admin("signup", &1))
          
              params
              |> Map.get("user")
              |> create_user()
              |> then(fn user -> notify_admin("signup", user) end)
          
          
          [0] https://hexdocs.pm/elixir/1.18.3/Kernel.html#then/2
    • layer8 5 hours ago ago

      It would be even better without the `>`, though. The `|>` is a bit awkward to type, and more noisy visually.

      • MyOutfitIsVague 5 hours ago ago

        I disagree, because then it can be very ambiguous with an existing `|` operator. The language has to be able to tell that this is a pipeline and not doing a bitwise or operation on the output of multiple functions.

        • layer8 4 hours ago ago

          Yes, I’m talking about a language where `|` would be the pipe operator and nothing else, like in a shell. Retrofitting a new operator into an existing language tends to be suboptimal.

    • Symmetry 6 hours ago ago

      I feel like Haskell really missed a trick by having $ not go the other way, though it's trivial to make your own symbol that goes the other way.

      • jose_zap 6 hours ago ago

        Haskell has & which goes the other way:

            users
              & map validate
              & catMaybes
              & mapM persist
        • taolson 5 hours ago ago

          Yes, `&` (reverse apply) is equivalent to `|>`, but it is interesting that there is no common operator for reversed compose `.`, so function compositions are still read right-to-left.

          In my programming language, I added `.>` as a reverse-compose operator, so pipelines of function compositions can also be read uniformly left-to-right, e.g.

              process = map validate .> catMaybes .> mapM persist
          • 1-more 4 hours ago ago

            Elm (written in Haskell) uses |> and <| for pipelining forwards and backwards, and function composition is >> and <<. These have made it into Haskell via nri-prelude https://hackage.haskell.org/package/nri-prelude (written by a company that uses a lot of Elm in order to make writing Haskell look more like writing Elm).

            There is also https://hackage.haskell.org/package/flow which uses .> and <. for function composition.

            EDIT: in no way do I want to claim the originality of these things in Elm or the Haskell package inspired by it. AFAIK |> came from F# but it could be miles earlier.

        • Symmetry 6 hours ago ago

          I guess I'm showing how long it's been since I was a student of Haskell then. Glad to see the addition!

  • amai 3 hours ago ago

    Pipelining looks nice until you have to debug it. And exception handling is also very difficult, because that means to add forks into your pipelines. Pipelines are only good for programming the happy path.

    • mpalmer 3 hours ago ago

      At the risk of over generalized pronouncements, ease of debugging is usually down to how well-designed your tooling happens to be. Most of the time the framework/language does that for you, but it's not the only option.

      And for exceptions, why not solve it in the data model, and reify failures? Push it further downstream, let your pipeline's nodes handle "monadic" result values.

      Point being, it's always a tradeoff, but you can usually lessen the pain more than you think.

      And that's without mentioning that a lot of "pipelining" is pure sugar over the same code we're already writing.

    • bergen an hour ago ago

      Depends on the context - in a scripting language where you have some kind of console you just don't copy all lines, and see what each pipe does one after another. This is pretty straight forward. (Not talking about compiled code though)

  • duped 5 hours ago ago

    A pipeline operator is just partial application with less power. You should be able to bind any number of arguments to any places in order to create a new function and "pipe" its output(s) to any other number of functions.

    One day, we'll (re)discover that partial application is actually incredibly useful for writing programs and (non-Haskell) languages will start with it as the primitive for composing programs instead of finding out that it would be nice later, and bolting on a restricted subset of the feature.

    • choult 4 hours ago ago

      ... and then recreate the scripting language...

      • stogot 4 hours ago ago

        I was just thinking does this not sound like a shell language? Using | instead of .function()

  • SimonDorfman 8 hours ago ago

    The tidyverse folks in R have been using that for a while: https://magrittr.tidyverse.org/reference/pipe.html

    • thom 7 hours ago ago

      I've always found magrittr mildly hilarious. R has vestigial Lisp DNA, but somehow the R implementation of pipes was incredibly long, complex and produced stack traces, so it moved to a native C implementation, which nevertheless has to manipulate the SEXPs that secretly underlie the language. Compared to something like Clojure's threading macros it's wild how much work is needed.

    • madcaptenor 7 hours ago ago

      And base R has had a pipe for a couple years now, although there are some differences between base R's |> and tidyverse's %>%: https://www.tidyverse.org/blog/2023/04/base-vs-magrittr-pipe...

    • flobosg 8 hours ago ago

      Base R as well: |> was implemented as a pipe operator in 4.1.0.

      • tylermw 2 hours ago ago

        Importantly, the base R pipe implements the operation at the language parsing level, so it has basically zero overhead.

  • dpc_01234 42 minutes ago ago

    I think there's a language syntax to be invented that would make everything suffix/pipeline-based. Stack based languages are kind of there, but I don't think exactly the same thing.

    BTW. For people complaining about debug-ability of it: https://doc.rust-lang.org/std/iter/trait.Iterator.html#metho... etc.

  • epolanski 7 hours ago ago

    I personally like how effect-ts allows you to write both pipelines or imperative code to express the very same things.

    Building pipelines:

    https://effect.website/docs/getting-started/building-pipelin...

    Using generators:

    https://effect.website/docs/getting-started/using-generators...

    Having both options is great (at the beginning effect had only pipe-based pipelines), after years of writing effect I'm convinced that most of the time you'd rather write and read imperative code than pipelines which definitely have their place in code bases.

    In fact most of the community, at large, converged at using imperative-style generators over pipelines and having onboarded many devs and having seen many long-time pipeliners converging to classical imperative control flow seems to confirm both debugging and maintenance seem easier.

  • kordlessagain 8 hours ago ago

    While the author claims "semantics beat syntax every day of the week," the entire article focuses on syntax preferences rather than semantic differences.

    Pipelining can become hard to debug when chains get very long. The author doesn't address how hard it can be to identify which step in a long chain caused an error.

    They do make fun of Python, however. But don't say much about why they don't like it other than showing a low-res photo of a rock with a pipe routed around it.

    Ambiguity about what constitutes "pipelining" is the real issue here. The definition keeps shifting throughout the article. Is it method chaining? Operator overloading? First-class functions? The author uses examples that function very differently.

    • Mond_ 8 hours ago ago

      > Pipelining can become hard to debug when chains get very long. The author doesn't address how hard it can be to identify which step in a long chain caused an error.

      Yeah, I agree that this can be problem when you lean heavily into monadic handling (i.e. you have fallible operations and then pipe the error or null all the way through, losing the information of where it came from).

      But that doesn't have much to do with the article: You have the same problem with non-pipelined functional code. (And in either case, I think that it's not that big of a problem in practice.)

      > The author uses examples that function very differently.

      Yeah, this is addressed in one of the later sections. Imo, having a unified word for such a convenience feature (no matter how it's implemented) is better than thinking of these features as completely separate.

    • zelphirkalt 8 hours ago ago

      You can add peek steps in pipelines and inspect the in between results. Not really any different from normal function call debugging imo.

      • krapht 7 hours ago ago

        Yes, but here's my hot take - what if you didn't have to edit the source code to debug it? Instead of chaining method calls you just assign to a temporary variable. Then you can set breakpoints and inspect variable values like you do normally without editing source.

        It's not like you lose that much readability from

          foo(bar(baz(c)))
        
          c |> baz |> bar |> foo
        
          c.baz().bar().foo()
        
          t = c.baz()
          t = t.bar()
          t = t.foo()
        • Mond_ 7 hours ago ago

          I feel like a sufficiently good debugger should allow you to place a breakpoint at any of the lines here, and it should break exactly at that specific line.

            fn get_ids(data: Vec<Widget>) -> Vec<Id> {
                data.iter()
                    .filter(|w| w.alive)
                    .map(|w| w.id)
                    .collect()
            }
          
          It sounds to me like you're asking for linebreaks. Chaining doesn't seem to be the issue here.
          • Merad 3 hours ago ago

            Jetbrains Rider does this does for C# code (I think Visual Studio does as well). Its inlay hints feature will also show you hints with the result type of each line as the data is transformed. I haven't explicitly tested but I would imagine their IDEs for other languages behave the same.

          • krapht 6 hours ago ago

            I'm only familiar with C++, Python, and SQL. Neither GDB nor PDB helps here, and I've never heard of a SQL debugger that will break apart expressions and let you view intermediate query results.

            • Mond_ 6 hours ago ago

              That'd be problematic, but also sounds like a (solvable) tooling problem to me.

            • jen20 6 hours ago ago

              It’s been a while since I’ve used one, but I’m fairly sure the common debuggers for C#, F#, Rust and Java would all behave correctly when breakpointed like this.

        • erichocean 5 hours ago ago

          The Clojure equivalent of `c |> baz |> bar |> foo` are the threading macros:

              (-> c baz bar foo)
          
          But people usually put it on separate lines:

              (-> c
                  baz
                  bar
                  foo)
          • joeevans1000 4 hours ago ago

            And with the Emacs Enlighten feature the second version enables seeing the results of each step right in the editor, to the right of the step.

    • AYBABTME 7 hours ago ago

      It's just as difficult to debug when function calls are nested inline instead of assigning to variables and passing the variables around.

    • fsckboy 8 hours ago ago

      the paragraph you quoted (atm, 7 mins ago, did it change?) says:

      >Let me make it very clear: This is [not an] article it's a hot take about syntax. In practice, semantics beat syntax every day of the week. In other words, don’t take it too seriously.

    • bena 7 hours ago ago

      I think you may have misinterpreted his motive here.

      Just before that statement, he says that it is an article/hot take about syntax. He acknowledges your point.

      So I think when he says "semantics beat syntax every day of the week", that's him acknowledging that while he prefers certain syntax, it may not be the best for a given situation.

    • pavel_lishin 8 hours ago ago

      The article also clearly points that that it's just a hot-take, and to not take it too seriously.

  • 0xf00ff00f 6 hours ago ago

    First example doesn't look bad in C++23:

        auto get_ids(std::span<const Widget> data)
        {
            return data
                | filter(&Widget::alive)
                | transform(&Widget::id)
                | to<std::vector>();
        }
    • Shorel an hour ago ago

      This looks awesome!

      I'm really want to start playing with some C++23 in the future.

    • inetknght 6 hours ago ago

      This is not functionally different from operator<< which std::cout has taught us is a neat trick but generally a bad idea.

      • senderista 6 hours ago ago

        Unlike the iostreams shift operators, the ranges pipe operator isn't stateful.

  • osigurdson 8 hours ago ago

    C# has had "Pipelining" (aka Linq) for 17 years. I do miss this kind of stuff in Go a little.

    • bob1029 6 hours ago ago

      I don't see how LINQ provides an especially illuminating example of what is effectively method chaining.

      It is an exemplar of expressions [0] more than anything else, which have little to do with the idea of passing results from one method to another.

      [0]: https://learn.microsoft.com/en-us/dotnet/csharp/language-ref...

      • osigurdson 2 hours ago ago

        Example from article:

        fn get_ids(data: Vec<Widget>) -> Vec<Id> { data.iter() // get iterator over elements of the list .filter(|w| w.alive) // use lambda to ignore tombstoned widgets .map(|w| w.id) // extract ids from widgets .collect() // assemble iterator into data structure (Vec) }

        Same thing in 15 year old C# code.

        List<Guid> GetIds(List<Widget> data)

        {

            return data
        
                   .Where(w => w.IsAlive())
        
                   .Select(w => w.Id)
        
                   .ToList();
        
        }
      • hahn-kev 5 hours ago ago

        So many things have been called Linq over the years it's hard to talk about at this point. I've written C# for many years now and I'm not even sure what I would say it's referring to, so I avoid the term.

        In this case I would say extension methods are what he's really referring to, of which Linq to objects is built on top of.

        • osigurdson 2 hours ago ago

          I'd say there are just two things:

          1) The method chaining extension methods on IEnumerable<T> like Select, Where, GroupBy, etc. This is identical to the rust example in the article.

          2) The weird / bad (in my opinion) language keywords analogous to the above such as "from", "where", "select" etc.

      • delusional 6 hours ago ago

        You might be talking about LINQ queries, while the person you are responding to is probably talking about LINQ in Method Syntax[1]

        [1]: https://learn.microsoft.com/en-us/dotnet/csharp/linq/get-sta...

    • vjvjvjvjghv 7 hours ago ago

      Agreed. It would be nice if SQL databases supported something similar.

      • sidpatil 7 hours ago ago

        PRQL [1] is a pipeline-based query language that compiles to SQL.

        [1] https://prql-lang.org/

      • NortySpock 7 hours ago ago

        I've used "a series of CTEs" to apply a series of transformations and filters, but it's not nearly as elegant as the pipe syntax.

  • singularity2001 7 hours ago ago

    I tried to convince the julia authors to make a.b(c) synonymous to b(a,c) like in nim (for similar reasons as in the article). They didn't like it.

    • queuebert 6 hours ago ago

      What were their reasons?

      • pansa2 5 hours ago ago

        I suspect:

        Julia's multiple dispatch means that all arguments to a function are treated equally. The syntax `b(a, c)` makes this clear, whereas `a.b(c)` makes it look like `a` is in some way special.

  • cutler 7 hours ago ago

    Clojure has pipeline functions -> and ->> without resorting to OO dot syntax.

    • jolt42 6 hours ago ago

      As well as some-> (exit on null) and cond-> (with predicates) that are often handy.

      • 3036e4 3 hours ago ago

        Both Fennel and Janet has the Clojure threading macros, with -?> and -?>> for false/null checks, but not any other variants as far as I know.

      • joeevans1000 4 hours ago ago

        As well as a lot of flexibility on where the result of the previous step feeds into the current one.

  • mrkeen 7 hours ago ago

      data.iter()
          .filter(|w| w.alive)
          .map(|w| w.id)
          .collect()
    
      collect(map(filter(iter(data), |w| w.alive), |w| w.id))
    
    The second approach is open for extension - it allows you to write new functions on old datatypes.

    > Quick challenge for the curious Rustacean, can you explain why we cannot rewrite the above code like this, even if we import all of the symbols?

    Probably for lack of

    > weird operators like <$>, <*>, $, or >>=

    • esafak 7 hours ago ago
      • rikthevik 3 hours ago ago

        Came here for the Uniform function call syntax link. This is one of the little choices that has a big impact on a language! I love it!

        I wrote a little pipeline macro in https://nim-lang.org/ for Advent of Code years ago and as far as I know it worked okay.

        ``` import macros

          macro `|>`\* (left, right : expr): expr =
            result = newNimNode(nnkCall)
        
            case right.kind
            of nnkCall:
              result.add(right[0])
              result.add(left)
              for i in 1..<right.len:
                result.add(right[i])
            else:
              error("Unsupported node type")
        ```

        Makes me want to go write more nim.

      • vips7L 4 hours ago ago

        I really wish you couldn't write extensions on nullable types. It's confusing to be able to call what look like instance functions on something clearly nullable without checking.

            fun main() {
                val s: String? = null
                println(s.isS()) // false
            }
        
            fun String?.isS() = "s" == this
    • Mond_ 7 hours ago ago

      > The second approach is open for extension - it allows you to write new functions on old datatypes.

      I prefer to just generalize the function (make it generic, leverage traits/typeclasses) tbh.

      > Probably for lack of > weird operators like <$>, <*>, $, or >>=

      Nope btw. I mean, maybe? I don't know Haskell well enough to say. The answer that I was looking for here is a specific Rust idiosyncrasy. It doesn't allow you to import `std::iter::Iterator::collect` on its own. It's an associated function, and needs to be qualified. (So you need to write `Iterator::collect` at the very least.)

      • higherhalf 6 hours ago ago

        > It doesn't allow you to import `std::iter::Iterator::collect` on its own. It's an associated function, and needs to be qualified.

        You probably noticed, but it should become a thing in RFC 3591: https://github.com/rust-lang/rust/issues/134691

        So it does kind of work on current nightly:

          #![feature(import_trait_associated_functions)]
          
          use std::iter::Iterator::{filter, map, collect};
        
          fn get_ids2(data: Vec<Widget>) -> Vec<Id> {
              collect(map(filter(Vec::into_iter(data), |w| w.alive), |w| w.id))
          }  
        
          fn get_ids3(data: impl Iterator<Item = Widget>) -> Vec<Id> {
              collect(map(filter(data, |w| w.alive), |w| w.id))
          }
        • Mond_ 6 hours ago ago

          Oh, interesting! Thank you, I did not know about that, actually.

  • RHSeeger 6 hours ago ago

    I feel like, at least in some cases, the article is going out of its way to make the "undesired" look worse than it needs to be. Compairing

        fn get_ids(data: Vec<Widget>) -> Vec<Id> {
            collect(map(filter(map(iter(data), |w| w.toWingding()), |w| w.alive), |w| w.id))
        }
    
    to

        fn get_ids(data: Vec<Widget>) -> Vec<Id> {
            data.iter()
                .map(|w| w.toWingding())
                .filter(|w| w.alive)
                .map(|w| w.id)
                .collect()
        }
    
    The first one would read more easily (and, since it called out, diff better)

        fn get_ids(data: Vec<Widget>) -> Vec<Id> {
            collect(
                map(
                    filter(
                        map(iter(data), |w| w.toWingding()), |w| w.alive), |w| w.id))
        }
    
    Admittedly, the chaining is still better. But a fair number of the article's complaints are about the lack of newlines being used; not about chaining itself.
    • the_sleaze_ 5 hours ago ago

      In my eyes newlines don't solve what I feel to be the issue. Reader needs to recognize reading from left->right to right->left.

      Of course this really only matters when you're 25 minutes into critical downtime and a bug is hiding somewhere in these method chains. Anything that is surprising needs to go.

      IMHO it would be better to set intermediate variables with dead simple names instead of newlines.

      fn get_ids(data: Vec<Widget>) -> Vec<Id> {

          let iter = iter(data);
      
          let wingdings = map(iter, |w| w.toWingding());
      
          let alive_wingdings = filter(wingdings, |w| w.alive);
      
          let ids = map(alive_wingdings, |w| w.id);
      
          let collected = collect(ids);
      
          collected
      
      }
    • tasuki 5 hours ago ago

      Oh wow, are we living in the same universe? To me the one-line example and your example with line breaks... they just... look about the same?

      See how adding line breaks still keeps the `|w| w.alive` very far from the `filter` call? And the `|w| w.id` very far from the `map` call?

      If you don't have the pipeline operator, please at least format it something like this:

          fn get_ids(data: Vec<Widget>) -> Vec<Id> {
              collect(
                  map(
                      filter(
                          map(
                              iter(data),
                              |w| w.toWingding()
                          ),
                          |w| w.alive
                      ),
                      |w| w.id
                  )
              )
          }
      
      ...which is still absolutely atrocious both to write and to read!

      Also see how this still reads fine despite being one line:

          fn get_ids(data: Vec<Widget>) -> Vec<Id> {
              data.iter().map(|w| w.toWingding()).filter(|w| w.alive).map(|w| w.id).collect()
          }
      
      It's not about line breaks, it's about the order of applying the operations, and about the parameters to the operations you're performing.
      • RHSeeger 4 hours ago ago

        > It's not about line breaks, it's about the order of applying the operations

        For me, it's both. Honestly, I find it much less readable the way you're split it up. The way I had it makes it very easy for me to read it in reverse; map, filter, map, collect

        > Also see how this still reads fine despite being one line

        It doesn't read fine, to me. I have to spend mental effort figuring out what the various "steps" are. Effort that I don't need to spend when they're split across lines.

        For me, it's a "forest for the trees" kind of thing. I like being able to look at the code casually and see what it's doing at a high level. Then, if I want to see the details, I can look more closely at the code.

    • TOGoS 5 hours ago ago

      They did touch on that.

      > You might think that this issue is just about trying to cram everything onto a single line, but frankly, trying to move away from that doesn’t help much. It will still mess up your git diffs and the blame layer.

      Diff will still be terrible because adding a step will change the indentation of everything 'before it' (which, somewhat confusingly, are below it syntactically) in the chain.

      • RHSeeger 4 hours ago ago

        Diff can ignore whitespace, so not really an issue. Not _as_ nice, but not really a problem.

  • layer8 4 hours ago ago

    The one thing that I don’t like about pipelining (whether using a pipe operator or method chaining), is that assigning the result to a variable goes in the wrong direction, so to speak. There should be an equivalent of the shell’s `>` for piping into a variable as the final step. Of course, if the variable is being declared at the same time, whatever the concrete syntax is would still require some getting used to, being “backwards” compared to regular assignment/initialization.

    • jiggunjer 2 hours ago ago

      Exists in R: Mydata %>% myfun -> myresult

  • hliyan 7 hours ago ago

    I always wondered how programming would be if we hadn't designed the assignment operator to be consistent with mathematics, and instead had it go LHS -> RHS, i.e. you perform the operation and then decide its destination, much like Unix pipes.

    • RodgerTheGreat 7 hours ago ago

      Plenty of LTR languages to choose from, especially concatenative languages like Forth, Joy, or Factor.

      The APL family is similarly consistent, except RTL.

    • remram 2 hours ago ago

      For function calls too? List the arguments then the function's name?

    • donatj 6 hours ago ago

      TI-BASIC is like this with its store operator →. I always liked it.

          10→A
          A+10→C
  • dapperdrake 7 hours ago ago

    Pipelining in software is covered by Richard C. Waters (1989a, 1989b). Wrangles this library to work with JavaScript. Incredibly effective. Much faster at writing and composing code. And this code executes much faster.

    https://dspace.mit.edu/handle/1721.1/6035

    https://dspace.mit.edu/handle/1721.1/6031

    https://dapperdrake.neocities.org/faster-loops-javascript.ht...

  • true_blue 3 hours ago ago

    That new Rhombus language that was featured here recently has an interesting feature where you can use `_` in a function call to act as a "placeholder" for an argument. Essentially it's an easy way to partially apply a function. This works very well with piping because it allows you to pipe into any argument of a function (including optional arguments iirc) rather than just the first like many pipe implementations have. It seems really cool!

  • weinzierl 5 hours ago ago

    I suffer from (what I call) bracket claustrophobia. Whenever brackets get nested too deep I makes me uncomfortable. But I fully realize that there are people who are the complete opposite. Lisp programmers are apparently as claustrophil as cats and spelunkers.

    • monsieurbanana 4 hours ago ago

      Forget the parenthesis, embrace the automatic indentation and code source manipulations that only perfectly balanced homoiconic expressions can give you.

  • taeric 4 hours ago ago

    A thing I really like about pipelines in shell scripts, is all of the buffering and threading implied by them. Semantically, you can see what command is producing output, and what command is consuming it. With some idea of how the CPU will be split by them.

    This is far different than the pattern described in the article, though. Small shame they have come to have the same name. I can see how both work with the metaphor; such that I can't really complain. The "pass a single parameter" along is far less attractive to me, though.

  • amelius 7 hours ago ago

    Am I the only one who thinks yuck?

    Instead of writing: a().b().c().d(), it's much nicer to write: d(c(b(a()))), or perhaps (d ∘ c ∘ b ∘ a)().

    • vinceguidry 7 hours ago ago

      Why, if you don't have to, would you write the functions in reverse order of when they're applied?

      • gloxkiqcza 7 hours ago ago

        Presumably because they’ve been doing so for decades so it seems logical and natural in their head while the new thing is new and thus unintuitive.

      • tgv 5 hours ago ago

        It's a bit snarky, but would you rather write FORTH then? So instead of

             draw(line.start, line.end);
             print(i, round(a / b));
        
        you'd write

            line.start, line.end |> draw;
            i, a, b |> div |> round |> print;
      • nh23423fefe 7 hours ago ago

        function application is right associative?

      • dboreham 7 hours ago ago

        Because it makes more sense?

        • kortex 5 hours ago ago

          I would wager within a rounding error, all humans have a lifetime of experience in following directions of the form:

          1. do the first step in the process

          2. then do the next thing

          3. followed by a third action

          I struggle to think of any context outside of programming, retrosynthesis in chemistry, and some aspects of reverse-Polish notation calculators, where you conceive of the operations/arguments last-to-first. All of which are things typically encountered pretty late in one's educational journey.

          • amelius 5 hours ago ago

            Consistency is more important. If you ever wrote:

            a(b())

            then you're already breaking your left-to-right/first-to-last rule.

            • regular_trash 9 minutes ago ago

              This is a foolish consistency, and a contrived counterexample. Consistency is not an ideal unto itself.

            • ndriscoll 5 hours ago ago

              There are some math books out there that use (x)f. My understanding is (some) algebraists tried to make it a thing ~60 years ago but it never really caught on.

              • hollerith 5 hours ago ago

                There are, but they leave out the parens and use context to distinguish function application from multiplication.

        • Mond_ 7 hours ago ago

          Does it actually make more sense, or is it just more familiar?

    • duped 5 hours ago ago

      A subtlety that I think many people overlook is that putting function application in lexicographical order means that tools can provide significantly better autocomplete results without needing to add a magic keybinding.

    • adzm 2 hours ago ago

      When a b c d are longer expressions, the pipeline version looks more readable especially when split on multiple lines since it only has one level of indentation and you don't have to think about the number of parentheses at the end.

    • queuebert 6 hours ago ago

      It probably wouldn't hurt for languages to steal more ideas from APL.

  • shae 8 hours ago ago

    If Python object methods returned `self` by default instead of `None` you could do this in Python too!

    This is my biggest complaint about Python.

  • wavemode 3 hours ago ago

    > At this point you might wonder if Haskell has some sort of pipelining operator, and yes, it turns out that one was added in 2014! That’s pretty late considering that Haskell exists since 1990.

    The tone of this (and the entire Haskell section of the article, tbh) is rather strange. Operators aren't special syntax and they aren't "added" to the language. Operators are just functions that by default use infix position. (In fact, any function can be called in infix position. And operators can be called in prefix position.)

    The commit in question added & to the prelude. But if you wanted & (or any other character) to represent pipelining you have always been able to define that yourself.

    Some people find this horrifying, which is a perfectly valid opinion (though in practice, when working in Haskell it isn't much of a big deal if you aren't foolish with it). But at least get the facts correct.

  • flakiness 4 hours ago ago

    After seeing LangChain abusing the "|" operator overload for pipeline-like DSL, I followed the suite at work and I loved it. It's especially good when you use it in a notebook environment where you literally build the pipeline incrementally through repl.

  • TrianguloY 7 hours ago ago

    Kotlin sort of have it with let (and run)

        a().let{ b(it) }.let{ c(it) }
    • hombre_fatal 4 hours ago ago

      Yeah, Kotlin's solution is nice because it's so general: you can chain on to anything instead of needing everyone to implement a builder pattern.

      And it's already idiomatic unlike bolting a pipeline operator onto a language that didn't start with it.

      • jillesvangurp an hour ago ago

        If you see somebody using a builder in Kotlin, they're basically doing it wrong. You can usually get rid of that stuff with a 1 line extension function (for example if it's some Java API that's being called).

          // extension function on Foo.Companion (similar to static class function in Java)
          fun Foo.Companion.create(block: FooBuilder.() -> Unit): Foo =
            FooBuilder().apply(block).build()
        
          // example usage
          val myFoo = Foo.create {
            setSomeproperty("foo")
            setAnotherProperty("bar")
          }
        
        Works for any Java/Kotlin API that forces you into method chaining and calling build() manually. Also works without extension functions. You can just call it fun createAFoo(..) or whatever. Looking around in the Kotlin stdlib code base is instructive. Lots of little 1/2 liners like this.
  • pxc 4 hours ago ago

    Maybe it's because I love the Unix shell environment so much, but I also really love this style. I try to make good use of it in every language I write code in, and I think it helps make my control flow very simple. With lots of pipelines, and few conditionals or loops, everything becomes very easy to follow.

  • chewbacha 7 hours ago ago

    Is this pipelining or the builder pattern?

    • meltyness 7 hours ago ago

      Pipes and filters are considered an architectural pattern, whereas Builder is a GoF OOP pattern, so yes.

    • ivanjermakov 4 hours ago ago

      I usually call it method chaining. Where the builder pattern use it.

    • Mond_ 7 hours ago ago

      "These are the same picture." (Sort of.)

  • mexicocitinluez 8 hours ago ago

    LINQ is easily one of C#'s best features.

    • adzm 2 hours ago ago

      Interestingly though the actual integrated query part is much less useful or widely used as the methods on IEnumerable etc.

  • zelphirkalt 8 hours ago ago

    To one up this: Of course it is even better, if your language allows you to implement proper pipelining with implicit argument passing by yourself. Then the standard language does not need to provide it and assign meaning to some symbols for pipelining. You can decide for yourself what symbols are used and what you find intuitive.

    Pipelining can guide one to write a bit cleaner code, viewing steps of computation as such, and not as modifications of global state. It forces one to make each step return a result, write proper functions. I like proper pipelining a lot.

    • Mond_ 7 hours ago ago

      > if your language allows you to implement proper pipelining with implicit argument passing by yourself > You can decide for yourself what symbols are used and what you find intuitive

      i mean this sounds fun

      but tbh it also sounds like it'd result in my colleague Carl defining an utterly bespoke DSL in the language, and using it to write the worst spaghetti code the world has ever seen, leaving the code base an unreadable mess full of sharp edges and implicit behavior

      • 0x1ceb00da 33 minutes ago ago

        Even veterans mess things up if you use too much of these exotic syntaxes. For loops and if statements rock, but they aren't cool and functional so they aren't discussed much.

  • 1899-12-30 7 hours ago ago

    You can somewhat achieve a pipelined like system in sql by breaking down your steps into multiple CTEs. YMMV on the performance though.

    • infogulch 7 hours ago ago

      Yeah, the way to get logical pipelining in SQL without CTEs is nested subqueries in the FROM clause. Unfortunately, the nesting is syntactically ugly and confusing to read which is basically the whole idea behind pipeline syntax.

  • otsukare 3 hours ago ago

    I wish more languages would aim for infix functions (like Haskell and Kotlin), rather than specifically the pipe operator.

  • bluSCALE4 4 hours ago ago

    Same. The sad part is that pipelining seems to be something AI is really good at so I'm finding myself writing less of it.

  • relaxing 3 hours ago ago

    These articles never explain what’s wrong with calling each function separately and storing each return value in an intermediate variable.

    Being able to inspect the results of each step right at the point you’ve written it is pretty convenient. It’s readable. And the compiler will optimize it out.

  • joeevans1000 4 hours ago ago

    Clojure threading, of course.

  • kuon 8 hours ago ago

    That's also why I enjoy elixir a lot.

    The |> operator is really cool.

  • drchickensalad 8 hours ago ago

    I miss F#

    • aloisdg 3 hours ago ago

      So do I sibling. so do I

  • tantalor 6 hours ago ago

    > allows you to omit a single argument from your parameter list, by instead passing the previous value

    I have no idea what this is trying to say, or what it has to do with the rest of the article.

    • delusional 6 hours ago ago

      It's getting at the essential truth that for all(?) mainstream languages since object orientation and the dot syntax became a thing `a.b()` implicitly includes `a` as the first argument to the actual method `b(a self)`. Different languages have different constructs on top of that, C++ for example includes a virtual dispatch mechanism, but the one common idea of the _method call_ is that the `self` pointer is passed as the first argument.

  • guerrilla 5 hours ago ago

    This is just super basic functional programming. Seems like we're taking the long way around...

    • Mond_ 4 hours ago ago

      Have you read the article? This isn't about functional vs. imperative programming, it's (if anything) about two different ways to write functional code.

      • guerrilla 2 hours ago ago

        Keywords "super basic". You learn this in a "my first Haskell" tutorials. Seems tortured in whatever language that is though.

  • wslh 6 hours ago ago

    I also like a syntax that includes pipelining parallelization, for example:

    A

    .B

    .C

      || D
    
      || E
    • regular_trash 4 minutes ago ago

      Wouldn't this complicate variable binding? I'm unsure how to think about this kinda of syntax if either D or E are expected to return some kind of data instead of "fire and forget" processes.

  • jaymbo 8 hours ago ago

    This is why I love Scala so much

    • rad_gruchalski 8 hours ago ago

      Scala is by far one of the nicest programming languages I have ever worked with. Scala with no JVM dependency would a killer programming language BUT only when all async features work out of the box like they do JVM. It’s been attempted a couple of times and it never succeeded.

    • minebreaker 8 hours ago ago

      `tap` is cool.

  • tpoacher 4 hours ago ago

    pipelines are great IF you can easily debug them as easily as temp variable assignments

    ... looking at you R and tidyverse hell.

  • blindseer 7 hours ago ago

    This article is great, and really distills why the ergonomics of Rust is so great and why languages like Julia are so awful in practice.

    • jakobnissen 3 hours ago ago

      You mean tab completion in Rust? Otherwise, let me introduce you to:

          imap(f) = x -> Iterators.map(f, x)
          ifilter(f) = x -> Iterators.filter(f, x)
          v = things |>
              ifilter(isodd) |>
              imap(do_process) |>
              collect