A lot of this university research on AI feels dated. This is something we all faced months ago. Many times they're also reaching conclusions using obsolete models, etc. Feels like things move too fast for traditional research cycles to catch up to AI dev
Its not resolved though, by any measure, except by removing human code review and coherence at the code level. Which is a pretty big tradeoff that has some at this point well-known downsides.
Not that I think most academic research is worth the time consumed producing it, but it feels dated because it actually takes time to do research instead of just rapidly changing direction based on feels, which is what developers are currently doing. It's the intellectual rigor of javascript frameworks, but with slop cannons.
In particular, code review is not solved. Pointing another error-prone bot at the output of your first error-prone bot, and then pressing "Enter" if the test suite passes, is not a solution. But you have to do it, because the next guy will beat you to prod if you don’t. And before too long, I'm sure that pressing Enter will be the bottleneck...
Right now what we have is a prisoner’s dilemma, where you can’t be slower than the guy next to you. So we rationalize slop and call it innovation.
Exactly. What a weird clickbait title to come out of a Tier 1 institution. You have Harvard PhD's concluding this!? This is what a PhD at that insitution is putting out? That is really sad. FTA:
The paper’s interviews show what that looks like on a team: a junior software engineer described a project where only one senior engineer could approve changes and could review the interviewee’s code only once a week.
So the issue is not AI. The issue is organizations and certain personalities who think that a human is going to be a better coder than an engineer. At this point rolling and writing your own code is akin to making up your own encryption scheme.
The problem is that when nobody writes code or feels ownership over it anymore, nobody will be qualified to or really interested in doing the code reviews. The way forward could involve dropping code reviews altogether. You could argue that the useful pingpong between coder and reviewers has been replaced by the similar pingpong between the prompter and the agent.
In my company, people found out that if they use an agent that commits and do the pull/merge request, they can approve it themselves and bypass a second set of eyes.
Since they are the one doing the prompting, they quickly got lazy of actually reviewing the output from the LLM which means that nobody is reviewing anything.
It hasn't happened yet but I guess that is how you end up with major outages.
This is everywhere to my knowledge, and engineers are burning out left and right with stress because of how little anyone understands the code, and how much of a mountain of nobody knowing anything they have to handle. We are turning our critical infrastructure mechanisms into stochastic grey goo.
I find the engineers currently thriving are the ones who never gave a fuck about anything but their own advancement and are happy to drop weeks of work on others and take the credit.
Understanding is overrated, application code (user-space) is anyway just a way to make money. If you're making more money over time, you're doing it right. We're not writing kernel code or safety critical code, we can let AI do the heavy lifting of putting characters into a file.
There are plenty of apps that while not safety critical, are financially critical to people's lives. When you're working on an app that processes billions of dollars in payments per month you can't just yolo it.
> The way forward could involve dropping code reviews altogether.
Yes, if we stop doing quality controls, we don't have the problem of being hung up on quality controls. You do of course have SOC-2 compliance which is a legally auditable attestation, so if you claim you do code review but don't you're committing fraud unless you drop that.
That would be great, IF there were valuable ping-pong between the prompter and the agent. For skilled engineers the game between themselves and the agent is enough. But most people are submitting senseless slop grenades for review.
Claude always comes up with more things to add to the code that are pure bloat and become inbred bombs over several iterations, and some people can't distinguish what suggestions to reject and go along with everything Claude suggests to the detriment of the program #garrytangate
I'm having good luck with the approach of building a thorough requirements specs that are a few pages long, are quite reviewable, and are human written/AI reviewed to prevent it from becoming slop. I call it layered spec driven development, where the one layer is AI read only, and the other layer is the slop layer that AI can play in. The problem is that it requires slowing down just a smidge, which to management these days is more unacceptable than no one having any clue how anything works.
From real-world experience, I have to disagree. It always has been and continues to be a faulty, up-front specification problem. The tail don't wag the dog.
If I increased the output of my pump jack and identified the refinery as the new bottleneck would you say the same pithy thing? NB a single pump jack can pump different grades of crude oil at the same time.
I’m not avoiding. But you have to state your assumption that everything before the pipeline is working well and it’s just the refinery that is struggling.
Then your analogy would break down as we all know that the PR machine is a slop producer.
With all respect to your personal experience, I could not possibly disagree more strongly with this.
In my own real world experience, the bottleneck has never been crafting the perfect specification up front. In fact, I have never, not even once, seen the initial specification for any real world software project of any mentionable size or scale survive unscathed to delivery. In reality, it is much like the old wartime maxim. "No Plan Survives First Contact With the Enemy".
The real world is messy and unpredictable. Every single software project of significance that I have been a part of has always involved testing, validating, and reworking the assumptions in the initial design and spec, refining them to arrive at the final product.
Furthermore, this iterative process (pre-AI) used to involve one or more humans following the development story and understanding what design changes and corresponding trade-offs were happening along the way (and why they were being made). By the time there was a finished deliverable ready for review, the person involved was privy to the story of why the development shaped up the way it did. They were able to account for the big picture, adapt accordingly along the way, and be able to explain why things ended up the way they did.
IME, with fully-agentic PRs, AI tends to view the specification as gospel and to view every point of friction between the specification and the real world as something to be worked around or minimized. Where a real human would pause, zoom out, and consider the system and bigger picture, considering what design changes might be needed, the AI will assume that its job is to make the specification a reality at all costs.
You can of course mitigate this by staying more closely involved with the work as it's being done, but this is of course slower than just throwing a spec over the wall and letting an agent run with it. It seems that people are far too eager to optimize for raw speed, and not nearly concerned enough with correctness or human understanding.
A sister comment mentioned a plethora of PRs that don't solve the right problem. If you are swamped with PRs solving the wrong problem, the root cause is not that you didn't foresee every future possibility or undiscovered devil in the details in order to create the right spec, the problem is that you thought you could come up with the right prompt at the outset and that an agent living in a clean digital world could make the journey all the way to a product fit for the messy real world on its own.
I don't think they're talking about "the perfect specification" they're talking about correctly describing what the program should do at all, formally or informally.
Here’s the thing: a human engineer is embedded in their company. They sit in meetings, they know whose opinions are more important and whose they can disregard. They know the history of how people at the company like to do things. They use all of this context to feed more precise specifications into a coding agent, and as a result, they get an output that better suits the environment of their company. Ergo, they spend less time reviewing agent output which was built to solve the wrong problems in the wrong way, and which ultimately must be discarded or significantly rewritten.
This again increases the demand for good programmers.
Write only programmers are dime on dozen. Those who can read code written by others (and themselves) and analyze it are harder to find.
You would hope, though we'll have to see how management actually sifts the chaff from the wheat in practice.
Code reviews have been notoriously neglected. In the most charitable cases, this was the result of being too bogged down with work or some kind of implementation minutia or time-consuming plumbing to have enough time to actually review code with enough care. That and you aren't necessarily rewarded for code reviews by mediocre companies, so there are perverse economic incentives. Which is disconcerting, because it creates a compulsion to produce without understanding, and understanding is more valuable than artifact.
Reading code is indeed more difficult, because the code's intent, composition, and architecture must be pieced together and inferred with the help of context (formalized languages are infamously bad at communicating intent and most code in the world is full of mixed and half-baked abstractions that further serve to obfuscate that intent, and intent is the measure). So a good code review of anything substantial involves thoughtful effort.
Now that LLMs can catch all sorts of trivial anti-patterns, the need for good judgement and reasoning is hopefully made more conspicuous, because as we know, vibe coders will continue to brush off code reviews, and mediocre programmers will continue to perform them poorly.
No, it is quite the opposite. In the scale that AI produces code, it isn't feasible to have humans reviewing it. And, frankly, reviewing code sucks big time.
The solution is to find ways to automate the process of code approval: more thorough tests and reviews, both made by AI.
If reviewing code "sucks big time", then I don't know what to tell you. Reading code is more important than writing it. LLMs can help out with reviews, but it cannot replace human judgement.
Maybe 6-12 months ago when everyone was reviewing in the same way we did pre AI. Nowadays agents write better code than most developers so it’s more about catching design or integration issues.
I think the bottleneck in the long term will be feature prioritization and specification, and testing. My role has already shifted a lot more into more CI checks to keep LLMs on guardrails
Maybe we should just review the prompt(s) the PR author used. Would be nice if this was automated and included in a PR. So code review is really checking that the inputs were valid, and testing the outputs.
I built a system that records numbered design decisions and numbered requirements as claude works in a Postgres database organised as a graph with links/interconnection between items.
It's an MCP server that Claude uses as we develop so there's a permanent record of everything and work is driven by requirements and decisions.
I've seen dozens of those, Claude even suggests building them, and they are all crap after a month. Just updating all the references of decisions becomes onerous. Make one change, it cascades everywhere, eventually all turns to utter crap. The gastown project failed in part due to agents inability to continuously carry context and it kept working on the information system instead of the problems. Good luck to you.
Mine seems fine. The key is that it must be structured as a graph database and anything can link to anything in the graph.
>> Just updating all the references of decisions becomes onerous.
LLMs are good at that sort of thing. Claude is instructed via MCP to ensure changes are linked when it adds or updates the database and to search the graph for relevant information fragments. Also I have a local GPU running a local LLM on an RTX3060/12GB that assists in keeping data quality and relationships relevant and maintained.
My system is an enormous help - in fact large scale development would be impossible without it.
I'm able to ask it the history of things, decisions supersede previous decisions, I can ask why the software behaves the way it does and why and when that decisions was made, I can pick up development threads that have spun off other threads and everything works real nice.
I've submitted the original paper (https://fion.ac/jellyfish.pdf), but others already have to little engagement, likely due to the paper's terrible title.
I really think ‘shifted the bottleneck to code review’ is misleading. It’s bad code. They write bad code. Yes if you want to salvage it you do a lot of code review, but that’s only because you’ve set your process up to pick spaghetti off the wall against which it has been flung. You could easily just push it all live without review and then pretend the bottleneck is SRE time, or set fire to yourself and claim the bottleneck is emergency service response time. But the root cause is that it’s bad code.
That’s a good take. How do we fix that? My own thought is agents trained on good codebases only. Not sprawling enterprise legacy applications which have long ago spread beyond their original design patterns. What are your thoughts?
I think it’s very hard with the current fine tuning infrastructure available to LLMs. Just like a real team, what you want is to immerse a new member in the domain, understanding that the code is a model of the world that both nourishes and is nourished by the team as you grow together. You want to reach a point where each contributor is intimately aligned on what you’re trying to achieve, and why and how. At that point all your code reviews should be left shifted anyway. But I have not had success putting this in a bunch of markdown files. There is very little infrastructure available for you to, for example, pair with an LLM for a period of time before setting it loose, and we’ve all become too impatient (voluntarily
or by compulsion) to value that anyway.
Software is always human scaled, then made to run at machine speed. Because the needs is ultimately human. And anything human does not change that much fundamentally. Physical laws are even more rigid.
A lot of AI practice fell flat because you can't point out the human needs in them. They powered by dreams and wishes and there nothing that ground them.
Vaguely reminiscent of the classic trope around 70's computer "effficiency gains" where a newly introduced computerised accounting system would end up taking longer than the original human-run system because once the computer was through doing the books the original human bookkeeper must still check the results manually.
And just because harvard researchers say something doesn't make it true either. Especially when it's like 6 months late and reviews are probably not the bottleneck anymore - testing, ci, parallelization are (imo).
I find funny that every time we discuss code made by AI people's reaction is to nurture some feeling of "someday, somehow they'll regret Claude Code and come back to me".
I suspect that the old assembly programmers had the same hope and feeling about the first higher level compilers for Ada, Pascal and C. And guess what? There are very few assembly programmers these days.
AI code is here to stay and will only grow. Deal with it. More than in any other area it is quite silly to be a Luddite in IT.
This just doesn't make sense. If AI is really smarter than humans we wouldn't need to review it. If it's not than why would the bottleneck shift, we shouldn't be doing this tradeoff between more reviews vs. writing code.
It's baffling the industry is choosing to move in the direction of more reviews which implies more things can fail.
Is their code reviewed by a junior, or another engineer on their level? That's the point, if AI is MORE intelligent/capable than humans, it follows we would have another AI review it.
It's just the latest attempt in bringing developer wages down.
There's decades worth of this: modelling, automatic code generation, low-code / no-code saas, "everyone should learn to code" and now llms.
Oh, and ageism of course. "we love your experience but we just don't feel it's a good cultural fit, so we're going to make the same mistakes you did a generation ago"
A lot of this university research on AI feels dated. This is something we all faced months ago. Many times they're also reaching conclusions using obsolete models, etc. Feels like things move too fast for traditional research cycles to catch up to AI dev
I think you are way overestimating the speed of diffusion into non-SV companies.
I was sitting with someone the other day who worked at a major ecommerce/media company (outside the US). They had access to Opus 4.5 via Co-pilot.
Its not resolved though, by any measure, except by removing human code review and coherence at the code level. Which is a pretty big tradeoff that has some at this point well-known downsides.
Not that I think most academic research is worth the time consumed producing it, but it feels dated because it actually takes time to do research instead of just rapidly changing direction based on feels, which is what developers are currently doing. It's the intellectual rigor of javascript frameworks, but with slop cannons.
In particular, code review is not solved. Pointing another error-prone bot at the output of your first error-prone bot, and then pressing "Enter" if the test suite passes, is not a solution. But you have to do it, because the next guy will beat you to prod if you don’t. And before too long, I'm sure that pressing Enter will be the bottleneck...
Right now what we have is a prisoner’s dilemma, where you can’t be slower than the guy next to you. So we rationalize slop and call it innovation.
Exactly. What a weird clickbait title to come out of a Tier 1 institution. You have Harvard PhD's concluding this!? This is what a PhD at that insitution is putting out? That is really sad. FTA:
The paper’s interviews show what that looks like on a team: a junior software engineer described a project where only one senior engineer could approve changes and could review the interviewee’s code only once a week.
So the issue is not AI. The issue is organizations and certain personalities who think that a human is going to be a better coder than an engineer. At this point rolling and writing your own code is akin to making up your own encryption scheme.
The problem is that when nobody writes code or feels ownership over it anymore, nobody will be qualified to or really interested in doing the code reviews. The way forward could involve dropping code reviews altogether. You could argue that the useful pingpong between coder and reviewers has been replaced by the similar pingpong between the prompter and the agent.
The compressed timelines which result when leadership sees how quickly code can be produced becomes pressure on the reviewer to not be a blocker.
Reviewers are the most burnt out right now
Seems like a way for the ones churning out the code to push responsibility onto reviewers.
In my company, people found out that if they use an agent that commits and do the pull/merge request, they can approve it themselves and bypass a second set of eyes.
Since they are the one doing the prompting, they quickly got lazy of actually reviewing the output from the LLM which means that nobody is reviewing anything.
It hasn't happened yet but I guess that is how you end up with major outages.
This is everywhere to my knowledge, and engineers are burning out left and right with stress because of how little anyone understands the code, and how much of a mountain of nobody knowing anything they have to handle. We are turning our critical infrastructure mechanisms into stochastic grey goo.
I find the engineers currently thriving are the ones who never gave a fuck about anything but their own advancement and are happy to drop weeks of work on others and take the credit.
Understanding is overrated, application code (user-space) is anyway just a way to make money. If you're making more money over time, you're doing it right. We're not writing kernel code or safety critical code, we can let AI do the heavy lifting of putting characters into a file.
There are plenty of apps that while not safety critical, are financially critical to people's lives. When you're working on an app that processes billions of dollars in payments per month you can't just yolo it.
It’s funny that anyone still thinks that these practices aren’t being applied to safety critical code in order to make money.
> The way forward could involve dropping code reviews altogether.
Yes, if we stop doing quality controls, we don't have the problem of being hung up on quality controls. You do of course have SOC-2 compliance which is a legally auditable attestation, so if you claim you do code review but don't you're committing fraud unless you drop that.
That would be great, IF there were valuable ping-pong between the prompter and the agent. For skilled engineers the game between themselves and the agent is enough. But most people are submitting senseless slop grenades for review.
Claude always comes up with more things to add to the code that are pure bloat and become inbred bombs over several iterations, and some people can't distinguish what suggestions to reject and go along with everything Claude suggests to the detriment of the program #garrytangate
I'm having good luck with the approach of building a thorough requirements specs that are a few pages long, are quite reviewable, and are human written/AI reviewed to prevent it from becoming slop. I call it layered spec driven development, where the one layer is AI read only, and the other layer is the slop layer that AI can play in. The problem is that it requires slowing down just a smidge, which to management these days is more unacceptable than no one having any clue how anything works.
From real-world experience, I have to disagree. It always has been and continues to be a faulty, up-front specification problem. The tail don't wag the dog.
Very true. Which makes the conclusion reached be this study even worse: we are stuck reviewing endless PRs that solve the wrong problems.
I don't fully disagree with you. That is a real big problem.
But both things can be true. One problem doesn't exclude the other.
> The tail don't wag the dog
If I increased the output of my pump jack and identified the refinery as the new bottleneck would you say the same pithy thing? NB a single pump jack can pump different grades of crude oil at the same time.
You may need to check the pipeline so that the refinery isn't stuck filtering random stuff from the oil.
AKA, it would be nice to have nice patches like this one to review.
https://cgit.freebsd.org/src/commit/?id=09d62ba33de113d01e88...
you're avoiding the question: If I increased the output of my pump jack and identified the refinery as the new bottleneck would you say i was wrong?
I’m not avoiding. But you have to state your assumption that everything before the pipeline is working well and it’s just the refinery that is struggling.
Then your analogy would break down as we all know that the PR machine is a slop producer.
With all respect to your personal experience, I could not possibly disagree more strongly with this.
In my own real world experience, the bottleneck has never been crafting the perfect specification up front. In fact, I have never, not even once, seen the initial specification for any real world software project of any mentionable size or scale survive unscathed to delivery. In reality, it is much like the old wartime maxim. "No Plan Survives First Contact With the Enemy".
The real world is messy and unpredictable. Every single software project of significance that I have been a part of has always involved testing, validating, and reworking the assumptions in the initial design and spec, refining them to arrive at the final product.
Furthermore, this iterative process (pre-AI) used to involve one or more humans following the development story and understanding what design changes and corresponding trade-offs were happening along the way (and why they were being made). By the time there was a finished deliverable ready for review, the person involved was privy to the story of why the development shaped up the way it did. They were able to account for the big picture, adapt accordingly along the way, and be able to explain why things ended up the way they did.
IME, with fully-agentic PRs, AI tends to view the specification as gospel and to view every point of friction between the specification and the real world as something to be worked around or minimized. Where a real human would pause, zoom out, and consider the system and bigger picture, considering what design changes might be needed, the AI will assume that its job is to make the specification a reality at all costs.
You can of course mitigate this by staying more closely involved with the work as it's being done, but this is of course slower than just throwing a spec over the wall and letting an agent run with it. It seems that people are far too eager to optimize for raw speed, and not nearly concerned enough with correctness or human understanding.
A sister comment mentioned a plethora of PRs that don't solve the right problem. If you are swamped with PRs solving the wrong problem, the root cause is not that you didn't foresee every future possibility or undiscovered devil in the details in order to create the right spec, the problem is that you thought you could come up with the right prompt at the outset and that an agent living in a clean digital world could make the journey all the way to a product fit for the messy real world on its own.
I don't think they're talking about "the perfect specification" they're talking about correctly describing what the program should do at all, formally or informally.
So the solution is: write a perfect spec and then you'll have perfect code, no need for reviews?
What world do y'all live in where this is your reality?
Here’s the thing: a human engineer is embedded in their company. They sit in meetings, they know whose opinions are more important and whose they can disregard. They know the history of how people at the company like to do things. They use all of this context to feed more precise specifications into a coding agent, and as a result, they get an output that better suits the environment of their company. Ergo, they spend less time reviewing agent output which was built to solve the wrong problems in the wrong way, and which ultimately must be discarded or significantly rewritten.
This feels obvious to me, but it's interesting how so many people feel the opposite.
I guess it depends a lot on how a company manages their SDLC/bureaucracy
If someone doesn't think code review is the bottleneck, chances are they've got AI doing their code reviews too.
Harvard researchers found a tree in a forest
This again increases the demand for good programmers. Write only programmers are dime on dozen. Those who can read code written by others (and themselves) and analyze it are harder to find.
There's no increase in demand until management who would hire them realizes how much trouble they're in.
You would hope, though we'll have to see how management actually sifts the chaff from the wheat in practice.
Code reviews have been notoriously neglected. In the most charitable cases, this was the result of being too bogged down with work or some kind of implementation minutia or time-consuming plumbing to have enough time to actually review code with enough care. That and you aren't necessarily rewarded for code reviews by mediocre companies, so there are perverse economic incentives. Which is disconcerting, because it creates a compulsion to produce without understanding, and understanding is more valuable than artifact.
Reading code is indeed more difficult, because the code's intent, composition, and architecture must be pieced together and inferred with the help of context (formalized languages are infamously bad at communicating intent and most code in the world is full of mixed and half-baked abstractions that further serve to obfuscate that intent, and intent is the measure). So a good code review of anything substantial involves thoughtful effort.
Now that LLMs can catch all sorts of trivial anti-patterns, the need for good judgement and reasoning is hopefully made more conspicuous, because as we know, vibe coders will continue to brush off code reviews, and mediocre programmers will continue to perform them poorly.
No, it is quite the opposite. In the scale that AI produces code, it isn't feasible to have humans reviewing it. And, frankly, reviewing code sucks big time.
The solution is to find ways to automate the process of code approval: more thorough tests and reviews, both made by AI.
If reviewing code "sucks big time", then I don't know what to tell you. Reading code is more important than writing it. LLMs can help out with reviews, but it cannot replace human judgement.
Maybe 6-12 months ago when everyone was reviewing in the same way we did pre AI. Nowadays agents write better code than most developers so it’s more about catching design or integration issues.
I think the bottleneck in the long term will be feature prioritization and specification, and testing. My role has already shifted a lot more into more CI checks to keep LLMs on guardrails
Maybe we should just review the prompt(s) the PR author used. Would be nice if this was automated and included in a PR. So code review is really checking that the inputs were valid, and testing the outputs.
For a given feature my list of prompts looks like:
- very well defined initial description of problem plus asking exactly what i want
- fix X
- why do you claim that Y?
- fix Y
- …
I don’t think it would be useful to review or commit prompts
Test don't review.
That bottlenecks too, in startup ahead of the adoption curve the whole cicd pipeline down to how to merge is being rethought
What about design decisions or code quality? Tests don't cover those.
I built a system that records numbered design decisions and numbered requirements as claude works in a Postgres database organised as a graph with links/interconnection between items.
It's an MCP server that Claude uses as we develop so there's a permanent record of everything and work is driven by requirements and decisions.
I've seen dozens of those, Claude even suggests building them, and they are all crap after a month. Just updating all the references of decisions becomes onerous. Make one change, it cascades everywhere, eventually all turns to utter crap. The gastown project failed in part due to agents inability to continuously carry context and it kept working on the information system instead of the problems. Good luck to you.
>> they are all crap after a month
Mine seems fine. The key is that it must be structured as a graph database and anything can link to anything in the graph.
>> Just updating all the references of decisions becomes onerous.
LLMs are good at that sort of thing. Claude is instructed via MCP to ensure changes are linked when it adds or updates the database and to search the graph for relevant information fragments. Also I have a local GPU running a local LLM on an RTX3060/12GB that assists in keeping data quality and relationships relevant and maintained.
My system is an enormous help - in fact large scale development would be impossible without it.
I'm able to ask it the history of things, decisions supersede previous decisions, I can ask why the software behaves the way it does and why and when that decisions was made, I can pick up development threads that have spun off other threads and everything works real nice.
>> Good luck to you.
Thank you!
They're the same picture
Equally time consuming and mind-numbing, IMO.
No pull request in my company get any human review comments anymore. No one is reading.
It's also hard to blame reviewers, when often that phase is preceded by the supposed "authors" not truly reading the reviews.
That's sort of old news. Code isn't manually reviewed anymore much.
I've submitted the original paper (https://fion.ac/jellyfish.pdf), but others already have to little engagement, likely due to the paper's terrible title.
I really think ‘shifted the bottleneck to code review’ is misleading. It’s bad code. They write bad code. Yes if you want to salvage it you do a lot of code review, but that’s only because you’ve set your process up to pick spaghetti off the wall against which it has been flung. You could easily just push it all live without review and then pretend the bottleneck is SRE time, or set fire to yourself and claim the bottleneck is emergency service response time. But the root cause is that it’s bad code.
That’s a good take. How do we fix that? My own thought is agents trained on good codebases only. Not sprawling enterprise legacy applications which have long ago spread beyond their original design patterns. What are your thoughts?
I think it’s very hard with the current fine tuning infrastructure available to LLMs. Just like a real team, what you want is to immerse a new member in the domain, understanding that the code is a model of the world that both nourishes and is nourished by the team as you grow together. You want to reach a point where each contributor is intimately aligned on what you’re trying to achieve, and why and how. At that point all your code reviews should be left shifted anyway. But I have not had success putting this in a bunch of markdown files. There is very little infrastructure available for you to, for example, pair with an LLM for a period of time before setting it loose, and we’ve all become too impatient (voluntarily or by compulsion) to value that anyway.
Software is always human scaled, then made to run at machine speed. Because the needs is ultimately human. And anything human does not change that much fundamentally. Physical laws are even more rigid.
A lot of AI practice fell flat because you can't point out the human needs in them. They powered by dreams and wishes and there nothing that ground them.
Vaguely reminiscent of the classic trope around 70's computer "effficiency gains" where a newly introduced computerised accounting system would end up taking longer than the original human-run system because once the computer was through doing the books the original human bookkeeper must still check the results manually.
Breaking news: Step A has been automated, so step B is now the bottleneck
We needed a research paper to tell us this?
Just because it is obviously true to you, it doesn't mean it is obviously true to others.
And just because it is obviously true to you, it doesn't mean that it is true, either.
And just because harvard researchers say something doesn't make it true either. Especially when it's like 6 months late and reviews are probably not the bottleneck anymore - testing, ci, parallelization are (imo).
Paraphrasing one of my favorite movie quotes: "Well ain't this a temporal anomaly. 6 months from any criticism!"
A Havard research I'll have you know! /s
do you even have a PhD?
(/s)
"code is cheap"...if you don't need to understand, review, and maintain it as part of a team.
I'm being told the answer is simply to let AI review and merge the code. Problem solved.
This, but unironically.
Good luck when you get woken up at the middle of the night for an outage!
Did this never happen before AI? Are we sure it happens more now, especially relative to the number of features being shipped?
I find funny that every time we discuss code made by AI people's reaction is to nurture some feeling of "someday, somehow they'll regret Claude Code and come back to me".
I suspect that the old assembly programmers had the same hope and feeling about the first higher level compilers for Ada, Pascal and C. And guess what? There are very few assembly programmers these days.
AI code is here to stay and will only grow. Deal with it. More than in any other area it is quite silly to be a Luddite in IT.
Harward researcher read lovable blog post
In other news, Dartmouth researchers find the sky is blue.
But people keep saying "coding was never the bottleneck!".
This just doesn't make sense. If AI is really smarter than humans we wouldn't need to review it. If it's not than why would the bottleneck shift, we shouldn't be doing this tradeoff between more reviews vs. writing code.
It's baffling the industry is choosing to move in the direction of more reviews which implies more things can fail.
"If AI is really smarter than humans we wouldn't need to review it."
Does the smartest person on your team not have their code reviewed?
Is their code reviewed by a junior, or another engineer on their level? That's the point, if AI is MORE intelligent/capable than humans, it follows we would have another AI review it.
No one said it's smarter than humans, and AI currently is better at code review than I am. It finds all kinds of edge cases I would have missed.
Still, humans are needed for the final sign-off for testing on real devices and making sure it works within the greater cultural context.
Alright, so if AI is better at code review than you, it follows you are not reviewing anymore, right? Otherwise that's the gap I'm pointing at.
It's just the latest attempt in bringing developer wages down.
There's decades worth of this: modelling, automatic code generation, low-code / no-code saas, "everyone should learn to code" and now llms.
Oh, and ageism of course. "we love your experience but we just don't feel it's a good cultural fit, so we're going to make the same mistakes you did a generation ago"