To be fair the peer review model has been falling apart long before AI generated text was a thing, mostly due to the exploitative nature of academic journals leading to incentive structures that don't align with producing good science.
Reviewers don't get paid and have zero incentive to do a good job. You can in many cases just submit a paper over and over to different journals until it encounters a reviewer that can't be bothered reading it, upon which it gets rubber stamped through. The result is that time and time again, incredibly shoddy and too often downright fraudulent science has gotten through peer review.
I have several papers published and this has been my experience too. One paper got submitted to dozens of different journals for several months as-is until it went through. Other times reviewers clearly did not read the paper in depth, and their critiques would have been answered if they read more than 2 pages in.
More importantly, I've never felt that the review process actually added anything, especially for engineering journals. No reviewer ever sees the actual prototype, algorithm, raw data, etc. The only thing they can critique is your methods and literature review. No one checks if the thing actually works as you're claiming it does, which is 80-90% of the hard part.
Certain fields, like mathematics, peer review is absolutely critical. But we've copy/pasted this process onto other fields where it's way less useful, which is why many fields are moving towards Arxiv instead of waiting for long, useless peer review process.
> One paper got submitted to dozens of different journals for several months as-is until it went through.
I hope you're exaggerating for the sake of an HN comment, but multiple concurrent submissions of the same work is absolutely grounds for desk rejection. It borders on an academic integrity violation, as most venues have an either implicit (e.g. through the ACM SIG they are a part of) or explicit policy forbidding concurrent submissions.
If you did this, I'm glad your work is rejected without being read. That's some shit that degrades the progress.
I've reviewed at least 20 papers in the last 3 months or so, probably closer to 40 just this year. What you're describing does not match my experience in any of the venues I have reviewed for, except for maybe low quality journals. If you're submitting to those venues, then your experience isn't typical.
Major conferences do care about review quality. Community standards are improving, as there is constant reflection on the process and how it's going.
AI is definitely causing new stresses on the system, but this comment makes the whole endeavor sound like it's a crap show, and it isn't.
> which is why many fields are moving towards Arxiv instead of waiting for long, useless peer review process.
ArXiv is also dropping the ball by subjecting papers to long moderation holds. I have a submission that has been on hold for over a month - approaching the point where I'll get the conference reviews back before the arXiv "review".
I understand that they are dealing with a flood of papers by and about AI, but it is annoying to get caught up in that for a paper in a non-AI field (cs.DS) that is entirely human-written except for a "find typos" pass.
I wonder if engineering or math is weird in this case. Papers are better if they are entirely verifiable from the claims in the paper, so, ideally it should be possible to fully verify the claims just based on what is said in the paper. For math that’s perfect because math is entirely made of arguments.
For engineering… I mean, sometimes you want to publish a paper about an algorithm that is more amenable to actual hardware and/or easier to tune. The reviewer would need access to an HPC cluster to verify your claim, nobody is going to burn their credits to verify your paper.
AI reviewers could be most helpful by… running the algorithm instead of their inference stacks!
Reviewers don't get paid but that doesn't mean they have zero incentive to do a good job.
First, reviewers do get compensated, just not with money but with reviews from other researchers when they submit their papers. Not every review is good, but I've gotten enough helpful reviews from highly competent researchers whose time is very valuable which I didn't have to pay a cent for, that I'm incentivized to do the same for others. Also I want the system to work, so that's also an incentive.
> You can in many cases just submit a paper over and over to different journals until it encounters a reviewer that can't be bothered reading it, upon which it gets rubber stamped through.
This is not how to build a career in research, only high impact journals and conferences matter.
> The result is that time and time again, incredibly shoddy and too often downright fraudulent science has gotten through peer review.
Peer review isn't supposed to detect fraud, it's not designed to do that, and the people reviewing aren't trained in fraud detection, nor should they be.
The system is supposed to detect fraud through reproducible studies, but if you hop over to this thread (https://news.ycombinator.com/item?id=50538303) you'll see people complaining all over that the results are obvious so what was the point of the research. Then you might appreciate how hard it is to make a career off reproducing results. If we think fraud in research is such a problem, we have to actually allocate funds toward reproducing results.
But to your point, peer review does stop plenty of shoddy research, just not all of it. It's not reasonable to expect peer review to stop all shoddy research, because we might be working on different and conflicting definitions of shoddy.
Well in the scenario you're a fraud or a hack, you're likely aware that you're a fraud or a hack, so you're unlikely to pursue a career in the traditional high impact journal sense. Though you may still wish to build credibility via volume of work, or to perpetuate some agenda, perhaps manufacturing a scientific basis for the magnetized water spiral business you're running.
We know replication doesn't prevent fraud either. Like in many fields most studies simply don't replicate. It's a whole thing.
It's very difficult to get institutions to work at scale when they assume every participant is acting in good faith. That may have worked 150 years ago when scientists were wealthy and doing the work for the love of the game (and also ironically not subject to peer review), but it doesn't work in the human centipede of fraud era of society.
> First, reviewers do get compensated, just not with money but with reviews from other researchers when they submit their papers.
It would make sense if publishing process was free and not extremely expensive. Publishers are sitting on pile of gold made from exploiting scientists and funneling public funds.
Abstract, for those not interested in reading TFA:
"Large language models now write papers and referee reports in economics. The field must develop best practices for handling AI-generated text in the publication process. Otherwise, peer review is in serious danger of collapsing. We propose that AI detectors should become a standard tool of the profession. Our experiment shows that they flag referee reports written by AI, not reports that AI merely proofread or helped edit.
Aided by a simple model, we argue that a paper flagged as AI-generated should trigger a closer reading and can be rejected at the first mistake, and that an AI-generated referee report adds nothing that editors could not get by running the prompt themselves. Such reports are unacceptable: they damage authors and journals and should be treated like plagiarism. Individual restraint has already shown its limits, and the profession needs to develop norms to solve this collective action problem. We close with a practical guide for authors, referees, and editors."
The computer security research community is strongly against using AI detectors, in part because the evidence has come against their being at all useful at scale [1]. Far too many false positives and negatives that the whole thing effectively becomes a witch-hunt test.
> Our evaluation not only finds that performance varies wildly across AIGT detectors (e.g., FPRs between 0.05 % and 68.6 % and FNRs ranging between 0.3 % and 99.6 %).
So it appears that there are good detectors that can be used and bad ones that shouldn’t.
That set of statistics does not tell you whether there are good detectors or not.
If the low FPR and the low FNR are the same detector, that's good. But it's more likely one detector has a high FNR and a different detector a high FPR, and the texts causing errors may be correlated across detectors making a "best of both worlds" multipass approach difficult or impossible.
I cannot read the entirety of this article, but nowhere in the abstract does it mention Pangram, which has repeatedly been shown in many independent studies to have a false positive rate of less than 1 in 10,000; by some accounts it is now down to 1 in 20,000.
Obviously there are many bad detectors that shouldn't be used, but it's so bizarre when studies like this leave out extremely high-quality detectors. For some reason there's just a segment of people who are dead-set on letting the world get polluted with bad writing.
Why do we invest 100s of Billions in the belief AGI is possible, yet an AI detector which protects the integretity of critical research is beyond our grasp?
Because it's an impossible problem. AI generates just words, and both AI and people use the same words. In fact both learn from each other. AI learns from people and eventually people will pick up patterns from AI.
Also, in many cases, a false positive hurts the innocent more than the cheaters. Many cheaters are already taking failure into account, while those innocent do not.
The more I get serious about this, the more I want to see Machines produce Machine-native output. I don't want crossover or doubt. I want high confidence that machine output:
> Aided by a simple model, we argue that a paper flagged as AI-generated should trigger a closer reading and can be rejected at the first mistake
“We propose a witch hunt fueled by unreliable, unaccountable, proprietary detection technology that’s roughly equivalent to tea leaf reading and can be fooled by trivial rewrite templates that anyone can find with a 5-second web search.”
The issue with double blind reviews is that it really depends on the problem domain.
In many areas, the pool of people who could have written a paper on a given topic and the pool of people who are subject matter experts, capable of reviewing a paper on said topic, are often very small and easily deduced.
But I agree that post-disclosure would add more of a skin in the game dynamics for reviewers.
IMO the solution is to just post pre-prints and be done with it. The market will naturally select on what is useful/important/correct.
(Totally open to everything public, I sign my peer reviews, but I don't see much point in people reading my peer reviews over just reading the paper themselves!)
Am not in academia, so the following might be naive, but it's not plausible that we go back to days before LLM writing and LLM reviewing papers. The amount of LLM output is also unlikely to ever decrease (and from what I understand the number of papers written even in pre-LLM days was already overwhelming the system). So:
1. Papers could be self-published on the web (Arxiv or Github or wherever else).
2a. Papers could be announced/advertised primarily by informal means (X and other social media, word of mouth, etc)
2b. In addition, there could be formal periodic announcements by academic institutions or industry entities that the authors belong to. E.g. a university might publish a list of top 10 (for your preferred value of 10) papers from each department, as selected by that department. The inclusion of a paper in these "top 10" lists would not necessarily mean that the paper has been peer-reviewed in any way. It would just signal that this paper may be worth checking out.
3. The purpose of journals/editors could shift from reviewing/publishing papers to aggregating all these informal announcements as well as all the "top 10" lists to identify most interesting/promising new papers in respective fields, with their URLs. Then it'd be up to the members of community interested in those topics to check out or not check out the papers in question.
I’m going to say a class of opinion that has historically been overwhelmingly rejected by this crowd:
Peer review will no longer matter in about 2 years and it will feel silly that we were ever asking ourselves these questions.
Despite being proven time and time again that performance trends are incredibly stable and if anything accelerating slightly, that pretraining scaling laws and their analogs for post training are reliable enough for this to be mathematically uncontroversial, it is just as reliable that these clearly accurate projections are so incredible that people still don’t seem to accept it. This sentence has been true and could have been said at any point in time over the last 4-5 years (and it has, many times).
The AI 2027 scenario while hilarious when posted is alarmingly not that far off from where we are today. As Dwarkesh reports from his conversations with Anthropic researchers and those at other labs, we are on track to solve RSI sometime in 2027 (again a relatively easy thing to project if you know a little bit about performance trends and how model performance is measured).
The world will not implode like a dying star, and I expect the experience to be more or less what we’re going through today, though some capabilities will experience a much higher rate of improvement, and since there are unavoidable bottlenecks that AI cannot get around ever and some that will take years to figure out how to break down. But “oh no we don’t have humans to peer review” is incredible to me: peer review is a quality filter, and humans are notoriously bad at it. Models are already better in many ways at peer review and will become so obviously better that having a human in the loop at all will be like hailing a horse and buggy ride from the airport.
> conversations with Anthropic researchers and those at other labs
The trouble is that most researchers at frontier labs were AGI/ASI/RSI-pilled a long time ago, likely when it was a distinctly minority position. It's part of their worldview, and as human beings, they're unlikely to say that it's not happening without very strong evidence that it's not happening.
The catch is that such evidence has not really appeared! The trends might just keep going! But I am skeptical that the predictions of the frontier lab folks are really valuable beyond the couple-of-months look into the future that their existing internal models provide.
You are probably responding to the ASI component. Today Opus 5.5 is definitely better at reviewing complex papers and grant applications in my field (neurogenetics and geroscience) than most human reviewers. It is already “no contest” in my experience.
1. Verification (is this paper's methods/conclusion valid)
2. Assign academic credit
3. Winnow down the set of all papers enough that a researcher can keep up with the new ones in their field.
This article mostly talks about #1, arguing that AI-generated papers are likely to fail verification while consuming a lot of review time.
However, I'm more interested in #3: as a researcher, how do I (externally) narrow down the new papers/ideas/libraries in my field enough that I can then manually choose what to read? E.g., give me a curated list of abstracts, short enough that I can read all of those abstracts and sub-select papers to read fully.
Peer-reviewed conferences do this by capping the number of accepted submissions (low enough that each paper can reserve ~20 minutes in the same building over several days), but #2 made that difficult even before AI. Unlimited AI-generated papers break the simple filter "looks like a LaTeX paper and matches my keywords", which worked fine for smaller fields.
Filtering out high AI % papers is an easy way to upper bound the number of papers you'll see by `(researchers in your field) * (time per researcher) / (time to write a polished LaTex paper by hand)`.
Conferences are now capping authors to a maximum of 8 papers which of course screws up university departments where everyone adds their name. This ETH professor says there should be a deposit paid to avoid AI spam https://www.linkedin.com/posts/omutlu_international-symposiu...
Same professor was rejected twice by peer review for what later became landmark papers one of which even got some Intel industry award. He did not change the paper and merely resubmitted that's how broken peer review already is.
I looked into this a little and found that it doesn't quite fit the narrative that I inferred (which may not be the one you were implying) from this thread. The authors are apparently from many teams all working for the pharma company Merck & Co. and the companies press release[1] suggests that part of the value of this paper is the coordination of the entire production chain. The number of authors seems to be a consequence of the purpose of the paper, Merck & Co.'s policies on attribution for publications, and the relationship between pharma and scientific publication (which I think has something to do with patents). This doesn't seem to have anything to do with university departments.
I'm not saying that it's reasonable for a paper to list 140 authors. In fact, I find the "author contribution" statement at the end to be comically vague. But I think the story is more complicated than CV padding from people who didn't make a substantial contribution.
I believe refundable tolls are going to be a must to avoid being flooded by AI. Whether it's reviewing papers or PRs, the gatekeepers should be able to charge for spam.
Clearly it's not worth wasting human attention on AI-generated text that doesn't reflect considerable human contribution.
Is the same true for AI-generated text that helps a human contributor contribute more than they otherwise would be able to? (like ESL, or someone who's just not great at writing) As in, the thoughts are human but the text is AI.
>AI-generated referee report adds nothing that editors could not get by running the prompt themselves.
This is false. Writing a good prompt to review a paper requires substantial background in the field and also requires substantial skill to evaluate the AI’s responses.
I routinely use Claude Opus to review my own papers and to point out my errors. If I am lucky I can also point out a few of Opus’ errors.
Over the last 6 months Opus has gotten to the point that it is as good a neurogeneticist as I am (Opus 5.5), and I have had the benefit of 45 years of training. I already rank Opus reviews of my grant applications to NIH as much superior to those that i receive from study section members who neither read the application fully nor do they read my recent publications. And sad to say: reviewers rarely know nearly as much quantitative genetics as Opus.
Most of the premises and results in the soft sciences are unverifiable and unfalsifiable, so no loss there.
In the harder sciences, publications as a product are dead. Perhaps "pure" university basic research is dead. I can imagine all the STEM funding sources drying up. Maybe research belongs in research consortia or for-profit companies?
Why should taxpayers pay for all those pointy-headed Poindexters?
I was tech chair for ICCV 2021 (6k+ submissions), so I saw the amount of bad behaviour even before LLMs.
Peer review will be redesigned from the ground up around LLMs. Patches are fine (AI detectors have a place, some bans can be good), but this won't be enough.
I'm surprised I don't see the proposed stronger solution: AI-based review. A separate model can provide an assessment of the paper, flag errors, inconsistencies. Then people can review the top scoring candidates.
Exactly! Of course the authors of papers and grant applications should already be doing this themselves. I would not submit a paper even to bioRxiv without a series of AI reviews (Opus 5.5).
Competent editors should do the same before begging for a good review by a human reviewer.
Computer Science professor here currently being flooded with review requests.
I think that academic conferences and journals should start charging a large deposit or fee for reviewing papers. If your paper is deemed to be of very low quality or AI slop your fee is kept and distributed evenly to all reviewers for that venue. If the paper meets some sort of threshold then the deposit is returned.
I hate for this to happen but it's the only way I see to stop the aslopolypse in academic writing.
Peer review will need to evolve; but that isn't necessarily a bad thing. The issue is not so much AI but people wielding it taking responsibility for how they wield it. Scientific reputation is tied to what you publish and where you publish it and how you conduct yourself doing so. And peer review is an important part of the process.
So, if you are say flooding a lot of channels with low quality slop that shouldn't be great for your reputation. And if your publication publishes a lot of slop, that should discourage scientists from risking their reputation by submitting articles. Peer review works both ways. If filters out low quality articles from high quality publications. And it makes authors of high quality articles prefer publications that have good reputations and good peer review. I take it as a given that people on both sides are going to be using AI. It's how they do that, that matters.
That's fine, as long as it's labeled accordingly. I opt for organic, sustainable options in everything where possible. Maybe journals just need a "Certified Free-Range Human" seal so readers know what they're getting.
It's amazingly offensive to LLM-review a paper that someone has put months or years of their life into writing, for report that will partly determine whether it can be published in a high quality journal. And the career consequences that come for it. For those unaware, tenure decisions in economics can rest on 3-5 top-level publications. The number of pubs is much smaller than in adjacent fields, like OR.
There should be strong, public, and embarrassing consequences for reviewers who do this. And the same applies to authors who use AI-generated text.
I just can't believe that people feel that they can inflict AI text on other humans as though it were their own. We can't normalize this, it's incredibly offensive. I recently received an AI communication from a doctor. Did the doctor even think about it, or did they just press enter? How much credence do I give their decision on the question I asked? Certainly much less than if they had written it themselves.
That said, the marketplace for ideas is going to look a lot different in the future, and that's probably ok. There needs to be a role for agentic or hybrid communication and discovery, and we need to find new ways to ferret out the truth in the midst of all of that. The old gatekeepers will still play some role, but it will be different. But feeding colleagues AI writing as your own needs to be a hard no.
The opposite (spending hours reviewing an AI-generated paper) is bad too. I think it comes down to a fundamental asymmetry in the effort put in on one side versus the other.
In the past month I've reviewed at least one paper that seemed entirely written by Claude. It was incredible in that in no way, shape or form, did Claude seem interested in communicating to an actual human. Even more incredible is that the human that took said artifact and submitted it to a research venue was fine with this. The paper might have been only one prompt away from being readable. "Ok Claude, please rewrite this paper so that a human can understand it. Instead of dozens of bizarre asides and parenthetical references to experiments you did in the background of our manic coding session, could you present this as a research paper that contextualizes its contribution in the context of related work? Make sure to state the problem and its importance up front. Produce motivation and tension that is resolved through the solution. Make this brain dump into a research paper."
I had half a mind to take said paper and just prompt Claude myself to do that, and then review that. But I think that isn't my job, now is it?
It almost definitely cannot, because it takes 1 minute to generate a research paper and it takes much more time to review it and make sure it's trash AI slop.
An AI model fixes a simple bug and it produces a 1800 LOC diff, a human does it with a diff of 14 lines. An AI model writes a blog post and it's 14 pages long, meandering, repetitive, full of claudese linguistic cliches, exaggerating random aspects of the message like they're the discovery of the century while simultaneously burying the lede. A human can get the message through in a handful of paragraphs.
Why are we thinking it's the human review bandwidth that is the problem in this picture? Fix the models so they can express ideas better.
To be fair the peer review model has been falling apart long before AI generated text was a thing, mostly due to the exploitative nature of academic journals leading to incentive structures that don't align with producing good science.
Reviewers don't get paid and have zero incentive to do a good job. You can in many cases just submit a paper over and over to different journals until it encounters a reviewer that can't be bothered reading it, upon which it gets rubber stamped through. The result is that time and time again, incredibly shoddy and too often downright fraudulent science has gotten through peer review.
I have several papers published and this has been my experience too. One paper got submitted to dozens of different journals for several months as-is until it went through. Other times reviewers clearly did not read the paper in depth, and their critiques would have been answered if they read more than 2 pages in.
More importantly, I've never felt that the review process actually added anything, especially for engineering journals. No reviewer ever sees the actual prototype, algorithm, raw data, etc. The only thing they can critique is your methods and literature review. No one checks if the thing actually works as you're claiming it does, which is 80-90% of the hard part.
Certain fields, like mathematics, peer review is absolutely critical. But we've copy/pasted this process onto other fields where it's way less useful, which is why many fields are moving towards Arxiv instead of waiting for long, useless peer review process.
> One paper got submitted to dozens of different journals for several months as-is until it went through.
I hope you're exaggerating for the sake of an HN comment, but multiple concurrent submissions of the same work is absolutely grounds for desk rejection. It borders on an academic integrity violation, as most venues have an either implicit (e.g. through the ACM SIG they are a part of) or explicit policy forbidding concurrent submissions.
If you did this, I'm glad your work is rejected without being read. That's some shit that degrades the progress.
I've reviewed at least 20 papers in the last 3 months or so, probably closer to 40 just this year. What you're describing does not match my experience in any of the venues I have reviewed for, except for maybe low quality journals. If you're submitting to those venues, then your experience isn't typical.
Major conferences do care about review quality. Community standards are improving, as there is constant reflection on the process and how it's going.
AI is definitely causing new stresses on the system, but this comment makes the whole endeavor sound like it's a crap show, and it isn't.
They never said the submissions were concurrent.
Dozens of different journals over several months implies there was concurrency.
> which is why many fields are moving towards Arxiv instead of waiting for long, useless peer review process.
ArXiv is also dropping the ball by subjecting papers to long moderation holds. I have a submission that has been on hold for over a month - approaching the point where I'll get the conference reviews back before the arXiv "review".
I understand that they are dealing with a flood of papers by and about AI, but it is annoying to get caught up in that for a paper in a non-AI field (cs.DS) that is entirely human-written except for a "find typos" pass.
I wonder if engineering or math is weird in this case. Papers are better if they are entirely verifiable from the claims in the paper, so, ideally it should be possible to fully verify the claims just based on what is said in the paper. For math that’s perfect because math is entirely made of arguments.
For engineering… I mean, sometimes you want to publish a paper about an algorithm that is more amenable to actual hardware and/or easier to tune. The reviewer would need access to an HPC cluster to verify your claim, nobody is going to burn their credits to verify your paper.
AI reviewers could be most helpful by… running the algorithm instead of their inference stacks!
Reviewers don't get paid but that doesn't mean they have zero incentive to do a good job.
First, reviewers do get compensated, just not with money but with reviews from other researchers when they submit their papers. Not every review is good, but I've gotten enough helpful reviews from highly competent researchers whose time is very valuable which I didn't have to pay a cent for, that I'm incentivized to do the same for others. Also I want the system to work, so that's also an incentive.
> You can in many cases just submit a paper over and over to different journals until it encounters a reviewer that can't be bothered reading it, upon which it gets rubber stamped through.
This is not how to build a career in research, only high impact journals and conferences matter.
> The result is that time and time again, incredibly shoddy and too often downright fraudulent science has gotten through peer review.
Peer review isn't supposed to detect fraud, it's not designed to do that, and the people reviewing aren't trained in fraud detection, nor should they be.
The system is supposed to detect fraud through reproducible studies, but if you hop over to this thread (https://news.ycombinator.com/item?id=50538303) you'll see people complaining all over that the results are obvious so what was the point of the research. Then you might appreciate how hard it is to make a career off reproducing results. If we think fraud in research is such a problem, we have to actually allocate funds toward reproducing results.
But to your point, peer review does stop plenty of shoddy research, just not all of it. It's not reasonable to expect peer review to stop all shoddy research, because we might be working on different and conflicting definitions of shoddy.
Well in the scenario you're a fraud or a hack, you're likely aware that you're a fraud or a hack, so you're unlikely to pursue a career in the traditional high impact journal sense. Though you may still wish to build credibility via volume of work, or to perpetuate some agenda, perhaps manufacturing a scientific basis for the magnetized water spiral business you're running.
We know replication doesn't prevent fraud either. Like in many fields most studies simply don't replicate. It's a whole thing.
It's very difficult to get institutions to work at scale when they assume every participant is acting in good faith. That may have worked 150 years ago when scientists were wealthy and doing the work for the love of the game (and also ironically not subject to peer review), but it doesn't work in the human centipede of fraud era of society.
We get compensated with a line item on our NSF synergistic activities document.
> First, reviewers do get compensated, just not with money but with reviews from other researchers when they submit their papers.
It would make sense if publishing process was free and not extremely expensive. Publishers are sitting on pile of gold made from exploiting scientists and funneling public funds.
Sounds like AIs should do the reviews!
That sounds like pull request review that is now totally useless.
Abstract, for those not interested in reading TFA:
"Large language models now write papers and referee reports in economics. The field must develop best practices for handling AI-generated text in the publication process. Otherwise, peer review is in serious danger of collapsing. We propose that AI detectors should become a standard tool of the profession. Our experiment shows that they flag referee reports written by AI, not reports that AI merely proofread or helped edit.
Aided by a simple model, we argue that a paper flagged as AI-generated should trigger a closer reading and can be rejected at the first mistake, and that an AI-generated referee report adds nothing that editors could not get by running the prompt themselves. Such reports are unacceptable: they damage authors and journals and should be treated like plagiarism. Individual restraint has already shown its limits, and the profession needs to develop norms to solve this collective action problem. We close with a practical guide for authors, referees, and editors."
The computer security research community is strongly against using AI detectors, in part because the evidence has come against their being at all useful at scale [1]. Far too many false positives and negatives that the whole thing effectively becomes a witch-hunt test.
[1] https://ieeexplore.ieee.org/stampPDF/getPDF.jsp?tp=&arnumber...
South Park perfectly depicted the nature AI detectors:
https://youtu.be/kVzUGRaWVT4
> Our evaluation not only finds that performance varies wildly across AIGT detectors (e.g., FPRs between 0.05 % and 68.6 % and FNRs ranging between 0.3 % and 99.6 %).
So it appears that there are good detectors that can be used and bad ones that shouldn’t.
That set of statistics does not tell you whether there are good detectors or not.
If the low FPR and the low FNR are the same detector, that's good. But it's more likely one detector has a high FNR and a different detector a high FPR, and the texts causing errors may be correlated across detectors making a "best of both worlds" multipass approach difficult or impossible.
Low fpr with avg fnr is still very useful.
I cannot read the entirety of this article, but nowhere in the abstract does it mention Pangram, which has repeatedly been shown in many independent studies to have a false positive rate of less than 1 in 10,000; by some accounts it is now down to 1 in 20,000.
Obviously there are many bad detectors that shouldn't be used, but it's so bizarre when studies like this leave out extremely high-quality detectors. For some reason there's just a segment of people who are dead-set on letting the world get polluted with bad writing.
Why do we invest 100s of Billions in the belief AGI is possible, yet an AI detector which protects the integretity of critical research is beyond our grasp?
Quitter.
Because it's an impossible problem. AI generates just words, and both AI and people use the same words. In fact both learn from each other. AI learns from people and eventually people will pick up patterns from AI.
Also, in many cases, a false positive hurts the innocent more than the cheaters. Many cheaters are already taking failure into account, while those innocent do not.
The more I get serious about this, the more I want to see Machines produce Machine-native output. I don't want crossover or doubt. I want high confidence that machine output:
- is optimized for machine input
- is convertible to human-native symbols, context
- includes metadata and proof-of-custody/origin
- leans into its role as an evolving language
- is necessarily terse and token-efficient
Basically: `unzaip output.zaip.gpg.sig --lang=EN-us --include-prompt`
`zip` => `zAIp`
It gets around some of my general complaints about "checking generated artifacts into version control", really interesting idea!
> Aided by a simple model, we argue that a paper flagged as AI-generated should trigger a closer reading and can be rejected at the first mistake
“We propose a witch hunt fueled by unreliable, unaccountable, proprietary detection technology that’s roughly equivalent to tea leaf reading and can be fooled by trivial rewrite templates that anyone can find with a 5-second web search.”
I found it more ironic that in order to save a burdened system we'll add a step that demands a heavier burden to the system.
Just throw more compute at it is the answer for everything for the AI pilled.
I am an occasional reviewer.
Peer review has been dying for a long time.
IMHO the solution was simple before AI and it is simple with AI:
Double-blind review, but after the submission process both the article and the reviews are made public, with full attribution.
The issue with double blind reviews is that it really depends on the problem domain.
In many areas, the pool of people who could have written a paper on a given topic and the pool of people who are subject matter experts, capable of reviewing a paper on said topic, are often very small and easily deduced.
But I agree that post-disclosure would add more of a skin in the game dynamics for reviewers.
IMO the solution is to just post pre-prints and be done with it. The market will naturally select on what is useful/important/correct.
(Totally open to everything public, I sign my peer reviews, but I don't see much point in people reading my peer reviews over just reading the paper themselves!)
Yeah cause openreview fixed allll the problems of publishing in ML or NLP.
Am not in academia, so the following might be naive, but it's not plausible that we go back to days before LLM writing and LLM reviewing papers. The amount of LLM output is also unlikely to ever decrease (and from what I understand the number of papers written even in pre-LLM days was already overwhelming the system). So:
1. Papers could be self-published on the web (Arxiv or Github or wherever else).
2a. Papers could be announced/advertised primarily by informal means (X and other social media, word of mouth, etc)
2b. In addition, there could be formal periodic announcements by academic institutions or industry entities that the authors belong to. E.g. a university might publish a list of top 10 (for your preferred value of 10) papers from each department, as selected by that department. The inclusion of a paper in these "top 10" lists would not necessarily mean that the paper has been peer-reviewed in any way. It would just signal that this paper may be worth checking out.
3. The purpose of journals/editors could shift from reviewing/publishing papers to aggregating all these informal announcements as well as all the "top 10" lists to identify most interesting/promising new papers in respective fields, with their URLs. Then it'd be up to the members of community interested in those topics to check out or not check out the papers in question.
I’m going to say a class of opinion that has historically been overwhelmingly rejected by this crowd:
Peer review will no longer matter in about 2 years and it will feel silly that we were ever asking ourselves these questions.
Despite being proven time and time again that performance trends are incredibly stable and if anything accelerating slightly, that pretraining scaling laws and their analogs for post training are reliable enough for this to be mathematically uncontroversial, it is just as reliable that these clearly accurate projections are so incredible that people still don’t seem to accept it. This sentence has been true and could have been said at any point in time over the last 4-5 years (and it has, many times).
The AI 2027 scenario while hilarious when posted is alarmingly not that far off from where we are today. As Dwarkesh reports from his conversations with Anthropic researchers and those at other labs, we are on track to solve RSI sometime in 2027 (again a relatively easy thing to project if you know a little bit about performance trends and how model performance is measured).
The world will not implode like a dying star, and I expect the experience to be more or less what we’re going through today, though some capabilities will experience a much higher rate of improvement, and since there are unavoidable bottlenecks that AI cannot get around ever and some that will take years to figure out how to break down. But “oh no we don’t have humans to peer review” is incredible to me: peer review is a quality filter, and humans are notoriously bad at it. Models are already better in many ways at peer review and will become so obviously better that having a human in the loop at all will be like hailing a horse and buggy ride from the airport.
> conversations with Anthropic researchers and those at other labs
The trouble is that most researchers at frontier labs were AGI/ASI/RSI-pilled a long time ago, likely when it was a distinctly minority position. It's part of their worldview, and as human beings, they're unlikely to say that it's not happening without very strong evidence that it's not happening.
The catch is that such evidence has not really appeared! The trends might just keep going! But I am skeptical that the predictions of the frontier lab folks are really valuable beyond the couple-of-months look into the future that their existing internal models provide.
You are probably responding to the ASI component. Today Opus 5.5 is definitely better at reviewing complex papers and grant applications in my field (neurogenetics and geroscience) than most human reviewers. It is already “no contest” in my experience.
Peer review serves a few purposes:
1. Verification (is this paper's methods/conclusion valid)
2. Assign academic credit
3. Winnow down the set of all papers enough that a researcher can keep up with the new ones in their field.
This article mostly talks about #1, arguing that AI-generated papers are likely to fail verification while consuming a lot of review time.
However, I'm more interested in #3: as a researcher, how do I (externally) narrow down the new papers/ideas/libraries in my field enough that I can then manually choose what to read? E.g., give me a curated list of abstracts, short enough that I can read all of those abstracts and sub-select papers to read fully.
Peer-reviewed conferences do this by capping the number of accepted submissions (low enough that each paper can reserve ~20 minutes in the same building over several days), but #2 made that difficult even before AI. Unlimited AI-generated papers break the simple filter "looks like a LaTeX paper and matches my keywords", which worked fine for smaller fields.
Filtering out high AI % papers is an easy way to upper bound the number of papers you'll see by `(researchers in your field) * (time per researcher) / (time to write a polished LaTex paper by hand)`.
Conferences are now capping authors to a maximum of 8 papers which of course screws up university departments where everyone adds their name. This ETH professor says there should be a deposit paid to avoid AI spam https://www.linkedin.com/posts/omutlu_international-symposiu...
Same professor was rejected twice by peer review for what later became landmark papers one of which even got some Intel industry award. He did not change the paper and merely resubmitted that's how broken peer review already is.
> which of course screws up university departments where everyone adds their name
Good. This practice is academic dishonesty and muddies the water terribly when it comes to evaluating a researcher's productivity and impact.
This publication had 140 authors: https://www.science.org/doi/10.1126/science.aed8713
I looked into this a little and found that it doesn't quite fit the narrative that I inferred (which may not be the one you were implying) from this thread. The authors are apparently from many teams all working for the pharma company Merck & Co. and the companies press release[1] suggests that part of the value of this paper is the coordination of the entire production chain. The number of authors seems to be a consequence of the purpose of the paper, Merck & Co.'s policies on attribution for publications, and the relationship between pharma and scientific publication (which I think has something to do with patents). This doesn't seem to have anything to do with university departments.
I'm not saying that it's reasonable for a paper to list 140 authors. In fact, I find the "author contribution" statement at the end to be comically vague. But I think the story is more complicated than CV padding from people who didn't make a substantial contribution.
[1] https://www.merck.com/news/merck-scientists-publish-landmark...
I believe refundable tolls are going to be a must to avoid being flooded by AI. Whether it's reviewing papers or PRs, the gatekeepers should be able to charge for spam.
Clearly it's not worth wasting human attention on AI-generated text that doesn't reflect considerable human contribution.
Is the same true for AI-generated text that helps a human contributor contribute more than they otherwise would be able to? (like ESL, or someone who's just not great at writing) As in, the thoughts are human but the text is AI.
How are we supposed to gauge human contribution?
I wished that the trend would be opting for clearer, less verbose papers. Text length and complex language are no proxy for quality.
>AI-generated referee report adds nothing that editors could not get by running the prompt themselves.
This is false. Writing a good prompt to review a paper requires substantial background in the field and also requires substantial skill to evaluate the AI’s responses.
I routinely use Claude Opus to review my own papers and to point out my errors. If I am lucky I can also point out a few of Opus’ errors.
Over the last 6 months Opus has gotten to the point that it is as good a neurogeneticist as I am (Opus 5.5), and I have had the benefit of 45 years of training. I already rank Opus reviews of my grant applications to NIH as much superior to those that i receive from study section members who neither read the application fully nor do they read my recent publications. And sad to say: reviewers rarely know nearly as much quantitative genetics as Opus.
I'm interested, do you not feel a bit uneasy about feeding all your research ideas to Claude?
IMHO..
Most of the premises and results in the soft sciences are unverifiable and unfalsifiable, so no loss there.
In the harder sciences, publications as a product are dead. Perhaps "pure" university basic research is dead. I can imagine all the STEM funding sources drying up. Maybe research belongs in research consortia or for-profit companies? Why should taxpayers pay for all those pointy-headed Poindexters?
I was tech chair for ICCV 2021 (6k+ submissions), so I saw the amount of bad behaviour even before LLMs.
Peer review will be redesigned from the ground up around LLMs. Patches are fine (AI detectors have a place, some bans can be good), but this won't be enough.
I'm surprised I don't see the proposed stronger solution: AI-based review. A separate model can provide an assessment of the paper, flag errors, inconsistencies. Then people can review the top scoring candidates.
Exactly! Of course the authors of papers and grant applications should already be doing this themselves. I would not submit a paper even to bioRxiv without a series of AI reviews (Opus 5.5).
Competent editors should do the same before begging for a good review by a human reviewer.
Reading span attention in most of humans that use AI and/or tiktok similar services is reduced to seconds. So is thinking.
Computer Science professor here currently being flooded with review requests.
I think that academic conferences and journals should start charging a large deposit or fee for reviewing papers. If your paper is deemed to be of very low quality or AI slop your fee is kept and distributed evenly to all reviewers for that venue. If the paper meets some sort of threshold then the deposit is returned.
I hate for this to happen but it's the only way I see to stop the aslopolypse in academic writing.
If yes, which I doubt already, can peer reviewers withstand a DoS attack by sheer volume of AI generated articles?
Peer review will need to evolve; but that isn't necessarily a bad thing. The issue is not so much AI but people wielding it taking responsibility for how they wield it. Scientific reputation is tied to what you publish and where you publish it and how you conduct yourself doing so. And peer review is an important part of the process.
So, if you are say flooding a lot of channels with low quality slop that shouldn't be great for your reputation. And if your publication publishes a lot of slop, that should discourage scientists from risking their reputation by submitting articles. Peer review works both ways. If filters out low quality articles from high quality publications. And it makes authors of high quality articles prefer publications that have good reputations and good peer review. I take it as a given that people on both sides are going to be using AI. It's how they do that, that matters.
Peer review will just be done by other AI, which is technically appropriate because they are peers to the AI authors.
That's fine, as long as it's labeled accordingly. I opt for organic, sustainable options in everything where possible. Maybe journals just need a "Certified Free-Range Human" seal so readers know what they're getting.
Thanks for this humorous comment in an otherwise intense thread!
It's amazingly offensive to LLM-review a paper that someone has put months or years of their life into writing, for report that will partly determine whether it can be published in a high quality journal. And the career consequences that come for it. For those unaware, tenure decisions in economics can rest on 3-5 top-level publications. The number of pubs is much smaller than in adjacent fields, like OR.
There should be strong, public, and embarrassing consequences for reviewers who do this. And the same applies to authors who use AI-generated text.
I just can't believe that people feel that they can inflict AI text on other humans as though it were their own. We can't normalize this, it's incredibly offensive. I recently received an AI communication from a doctor. Did the doctor even think about it, or did they just press enter? How much credence do I give their decision on the question I asked? Certainly much less than if they had written it themselves.
That said, the marketplace for ideas is going to look a lot different in the future, and that's probably ok. There needs to be a role for agentic or hybrid communication and discovery, and we need to find new ways to ferret out the truth in the midst of all of that. The old gatekeepers will still play some role, but it will be different. But feeding colleagues AI writing as your own needs to be a hard no.
The opposite (spending hours reviewing an AI-generated paper) is bad too. I think it comes down to a fundamental asymmetry in the effort put in on one side versus the other.
Exactly. It's responding to actual work with non-work, from either end.
In the past month I've reviewed at least one paper that seemed entirely written by Claude. It was incredible in that in no way, shape or form, did Claude seem interested in communicating to an actual human. Even more incredible is that the human that took said artifact and submitted it to a research venue was fine with this. The paper might have been only one prompt away from being readable. "Ok Claude, please rewrite this paper so that a human can understand it. Instead of dozens of bizarre asides and parenthetical references to experiments you did in the background of our manic coding session, could you present this as a research paper that contextualizes its contribution in the context of related work? Make sure to state the problem and its importance up front. Produce motivation and tension that is resolved through the solution. Make this brain dump into a research paper."
I had half a mind to take said paper and just prompt Claude myself to do that, and then review that. But I think that isn't my job, now is it?
> We are happy to acknowledge that we used LLMs to help with this paper.
Framed another way: AIs are arguing with AIs about the merits of AIs, while the Board of Governors of the Federal Reserve System is nodding.
Half of academics are the worst of all. Obsessed with their crappy Claude websites, tinkering with ChatBots and producing nothing of value.
Any academic using AI should be fired.
Sure, if by peer you mean other AI bot
It almost definitely cannot, because it takes 1 minute to generate a research paper and it takes much more time to review it and make sure it's trash AI slop.
Another valid question is - “Can AI-Generated text/code survive Peer reviews?”
So far we’ve been using volume and time as constraints to justify why reviewing so much AI-generated text and code is humanly impossible.
But if we solve time and cognitive constraints for subject matter experts, Can AI-generated text and code survive peer reviews?
An AI model fixes a simple bug and it produces a 1800 LOC diff, a human does it with a diff of 14 lines. An AI model writes a blog post and it's 14 pages long, meandering, repetitive, full of claudese linguistic cliches, exaggerating random aspects of the message like they're the discovery of the century while simultaneously burying the lede. A human can get the message through in a handful of paragraphs.
Why are we thinking it's the human review bandwidth that is the problem in this picture? Fix the models so they can express ideas better.
Well if we're talking science then we might be forced to slow down and actually examine our evidence. Oh poo