OpenJDK Interim Policy on Generative AI

(openjdk.org)

59 points | by blenderob 9 hours ago ago

76 comments

  • Viliam1234 8 hours ago ago

    This makes sense.

    AI contribution is basically just "prompt + AI work". Even if you are okay with AI work per se, you should accept prompts (after reviewing them) and let your own AI generate the code (and then also review the code)... rather then accept an output of someone else's AI with an unknown prompt, that may or may not include an instruction to create a vulnerability.

    In the age of AI, the prompt is becoming the actual source code. Accepting AI-generated code would be like accepting binary code from unknown source.

    • xpct 8 hours ago ago

      I don't think we're approaching the limit of deterministic prompt -> source code mapping any time soon. Small variability in prompts produces medium variability in outputs. Building on previous outputs only extends the variability, where behavioral differences start appearing a few prompts down the line.

      And the AI companies aren't focused on making this process any more deterministic, where repeated prompts are sampled using different seeds. I don't know if it's because the randomness makes the models explore a larger space and perform better, but it's probably intentionally done because it'd be quite easy to remove the stochasticity.

      • Frost1x 7 hours ago ago

        There’s another issue where models and transparent wrappers around models that get exposed are shifting around often. Versioning is highly questionable, and not all closed models will be supported indefinitely… so determinism becomes highly questionable at a purely prompt level.

        • xpct 7 hours ago ago

          Oh, I didn't even consider this. Now that you mentioned it, I wouldn't be surprised if the stochasticity is intentional so users don't treat it as deterministic, and the company is allowed to continuously update the model (or even a small part of its weights, perhaps to fix/censor something?).

    • this_user 8 hours ago ago

      Not really though, since the result of the prompt is not deterministic. It greatly depends on the model, the version, the harness, even time of day if the provider's infrastructure is currently overloaded and is silently degrading performance. Some things also require multi-turn interactions.

      • kylemart 8 hours ago ago

        100% agree with this. Was about to write the same thing.

        • doctorpangloss 7 hours ago ago

          this is the unintentional HN humor i come here for

      • lelanthran 8 hours ago ago

        > Not really though, since the result of the prompt is not deterministic. It greatly depends on the model, the version, the harness, even time of day if the provider's infrastructure is currently overloaded and is silently degrading performance. Some things also require multi-turn interactions.

        All of that are even more reasons to reject AI-generated code: if it cannot be trusted to produce same (or even similar) output just from the prompts, why accept it at all?

        • rat9988 7 hours ago ago

          This is why I have been asking to fire product owners. Givven the same input, but different software engineers working with them, they can produce different output (code).

          • Volundr 7 hours ago ago

            It's worse than that! Even when the product managers inputs to the engineers are constrained for many problems no two engineers will produce consistent results! In fact recent studies suggest that even the same engineer may produce different results based on mood, if they've had coffee yet, and how near to EOD it is.

    • pennomi 8 hours ago ago

      Except prompts aren’t deterministic, source code is.

  • mrkeen an hour ago ago
  • kdavis 8 hours ago ago

    While I understand the caution, the current policy seems too draconian.

    It states in part:

    > Until that policy is in place, the Governing Board has approved this interim policy:

    > Contributions in the OpenJDK Community must not include content generated, in part or in full, by large language models...

    Note, this would exclude most spell checkers, as they often are LLM based. That said, they do soften this with the addition:

    > Q: Is it okay to continue using the spell-checking, grammar-checking, auto-completion, and refactoring features in my editor or IDE?

    > A: Yes, so long as they are not based on large language models or similar deep-learning systems.

    This addition will likely not help as most spell checkers are OS/IDE services and their implementation details are opaque to users.

    • ekidd 8 hours ago ago

      Is anyone using even a tiny on-device language model for spelling? For grammar, I could almost imagine it.

      But this also bans "simple" AI-powered auto-complete, like Zed's Zeta2 model. This is a very conservative model that rarely tries to propose more than a few obvious lines (at least in my use cases). If a developer accepts a three-line autocomplete that introduces a bug, that's kind of on them. Honestly, about the only thing that Zeta2 is good for is reducing the risk of RSI.

      To be fair, it's also their project, and I completely support their right to set whatever policies they want.

      • kdavis 8 hours ago ago

        The is not exactly my point.

        I am speaking of spelling/grammar checkers are part of the IDE and/or OS.

        You'd have to know your IDE and/or OS's implementation details to know you are not using an LLM based one.

        • Capricorn2481 8 hours ago ago

          I'm fairly confident mine doesn't. There'd be 0 benefit to using an LLM for something we've had near perfect tooling around for decades.

          LSPs vs LLM Autocomplete are a different story, but I can just turn that off.

    • malfist 8 hours ago ago

      > this would exclude most spell checkers, as they often are LLM based

      Who is using an LLM as a spell checker?

      • kdavis 8 hours ago ago

        My point is you might be when you wrote your comment or I might be in writing this comment.

        However, I don't know if I am as I haven't seen, and will likely never see, the implementation behind the text box that is suggesting the next letter, word, or spelling correction as I type.

        • yjftsjthsd-h 8 hours ago ago

          Can you point at a specific spell checker that is known to use an LLM?

          • kdavis 7 hours ago ago

            "A Simple yet Effective Training-free Prompt-free Approach to Chinese Spelling Correction Based on Large Language Models"[1]

            [1] https://aclanthology.org/2024.emnlp-main.966/

            • malfist 7 hours ago ago

              That's a research paper. Can you point to a single product that claims they use an LLM for their spellchecker? With today's investor market, surly that would be something a company would brag about.

              • kdavis 7 hours ago ago

                This wasn't your original question, which I answered.

                This "moving the goalposts", requiring me to know the internals of say a closed source product, just proves my original point. I can't know if an LLM is being used because I don't have the source code.

                • yjftsjthsd-h 6 hours ago ago

                  > This wasn't your original question, which I answered.

                  No, their original question was,

                  > Who is using an LLM as a spell checker?

                  (Emphasis mine)

                  > This "moving the goalposts", requiring me to know the internals of say a closed source product, just proves my original point. I can't know if an LLM is being used because I don't have the source code.

                  It's not moving the goalposts, you pointed at research showing that somebody theoretically looked at using LLMs for that purpose, not evidence that anybody is actually doing so which is obviously what we're discussing. This is getting into Russell's Teapot territory; yes, a closed source system could do anything, but it would be nice if we had literally any evidence that they were doing so, especially when you're proposing that they burn orders of magnitude more compute on something that isn't really better than existing systems (i.e. there are obvious downsides).

                  Edit: I see that you have now posted another comment actually providing that evidence. That's actually compelling; thanks.

                • bigstrat2003 2 hours ago ago

                  Nobody has moved the goalposts. You claimed that "most spell checkers" would be disallowed by this policy because they use LLMs. You are simply being asked to substantiate that rather bold claim.

              • kdavis 6 hours ago ago

                Craig Federighi SVP of Software at Apple “The keyboard now leverages a transformer language model, which is state of the art for word prediction, making autocorrect more accurate than ever.”[1]

                "New features available with macOS Sonoma" doc states "Autocorrect fixes mistakes for you more accurately than ever before by leveraging a new transformer language model in English, French, and Spanish keyboards." [2]

                "Introducing Apple’s On-Device and Server Foundation Models" blog post states "Apple Intelligence is comprised of multiple highly-capable generative models that...have been fine-tuned for user experiences such as writing and refining text.. In the following overview, we will detail how two of these models — a ~3 billion parameter on-device language model, and a larger server-based language model."[3]

                "A look at Apple’s new Transformer-powered predictive text model" blog post[4] is the kind of deep dive you'd have to do to find out what's going on under the hood in your spelling/grammar checker. TL;DR AppleSpell uses GPT-2 which proves my point.

                [1] https://eu.usatoday.com/story/tech/2023/06/06/wwdc-2023-appl...

                [2] https://www.apple.com/mideast/macos/sonoma/pdf/en/macOS_All_...

                [3] https://machinelearning.apple.com/research/introducing-apple...

                [4] https://jackcook.com/2023/09/08/predictive-text.html

                • malfist 4 hours ago ago

                  Autocorrect != spellcheck

      • bpicolo 8 hours ago ago

        Yeah - spell checking has been on device for literal decades.

    • rf15 8 hours ago ago

      I think it's pretty clear how this was meant, not sure why you feel the need to split hairs here. I'm certain the intention is this:

      - spell checkers: ok

      - '.getC' autocompleting to '.getColor()': ok

      - 'getC' autocompleting to 'getColor()' + five more lines based on preceding code and documentation: NO.

    • xpct 8 hours ago ago

      Why is it draconian? If we take Dec-2025 as the point where the models became good for real work (agreed by many), not even a year has passed since then. The world hasn't changed, and there are still many open legal questions.

    • raincole 8 hours ago ago

      > Note, this would exclude most spell checkers, as they often are LLM based

      ?

      Like, what? What are you even talking about? Most spell checkers used by programmers are certainly not LLM based. Spell-checking has always been instant for like two decades. It's such a solved problem.

  • pizlonator 8 hours ago ago

    How do you enforce this?

    • asdfsa32 8 hours ago ago

      Blockchain.

    • rileymichael 8 hours ago ago

      the same way they currently enforce authors own all copyright in their contributions, via the contributor agreement (OCA). in the real world, people aren’t going to intentionally sneak in LLM-authored code and potentially be legally liable.

    • htltzp 8 hours ago ago

      Best effort, just like in the case of existing contributor agreements that demand you to be the copyright holder.

      Or like in the case of checking that free range eggs are in fact free range eggs.

      Or like in the case of Steam demanding AI disclosure, to the dismay of Tim Sweeney ...

      Let's hope the dismay increases over time.

    • functionmouse 8 hours ago ago

      Subjectively; as 100% of all laws and rules throughout history have been enforced.

      • talon8635 8 hours ago ago

        I suspect people keep asking because it’s becoming, with each passing day, increasingly difficult to distinguish what has been generated from what has not

        • functionmouse 8 hours ago ago

          Look, if nobody knows, nobody cares.

        • realusername 8 hours ago ago

          Open source maintainers just want to avoid slop PRs and it's very easy to distinguish them

    • lelanthran 8 hours ago ago

      > How do you enforce this?

      It's not impossible - right now there are all these low-effort PRs. If the PR is indistinguishable from a normal human-written PR, that addresses some of their concerns.

      In any case, a rule or policy that can be worked around is not a reason to do away with that rule or policy.

      I mean, if we only ever catch 10% of murderers, do you really think it is reasonable to remove that law from the books?

    • stonogo 8 hours ago ago

      As best as you can. I don't know why people keep asking this question, when the answer is obvious and exactly like all other contribution policies.

      In this case, this is coming from Oracle, which holds the OpenJDK purse strings, so being found in violation is also likely to have financial consequences for the violator.

      • NitpickLawyer 8 hours ago ago

        > I don't know why people keep asking this question

        Well, because it's going to lead to witch hunts and drama and so on. I mean, not necessarily on this project, but the same-ish discussion is going on on many community based projects (debian was recently on the front page with 4 proposals, etc).

      • Marha01 8 hours ago ago

        > I don't know why people keep asking this question

        Asking how do you enforce a policy that is practically unenforceable is a legitimate question.

        • Volundr 7 hours ago ago

          How do you enforce the law against murder?

          Mostly after the fact when people get caught. It's hard to enforce is rarely a good argument against a rule.

          • Marha01 7 hours ago ago

            Catching a murderer is much, *much* easier than catching someone using a modern AI model to generate code, especially if they actually read it and fix occasional LLMisms, and not just vibecode from the hip. The societal damage is also infinitely more serious in the case of murder.

            • stonogo 6 hours ago ago

              If your argument is "this is unenforceable" and "failure to enforce is a low-stakes problem" then I'm not sure what your concerns are, other than a desire to control how others choose to spend their time. Either of those might be a valid objection (although I do not consider them to be) but together they sort of cancel each other out, no?

              Regardless, some efforts like this are explicitly performative. In the event some problems arise and it turns out someone used a specific tool in the process of causing those problems, administrators (and insurers) can say "look we forbade the use of that tool, so this is on them and not us." In other words, enforcement is not always even the point, but it always seems to be the first stop on the concern-trolling trolley.

              Finally, yes, people can misbehave and break rules, and the more effort they put into doing so, the more successful they will be at the rule breaking. Nothing about that is justification for not having rules, even when they are hard to enforce.

  • effnorwood 8 hours ago ago

    Oracle determining how to add more sypware while not violating their own TOS

    • pron 8 hours ago ago

      What spyware is there in the JDK? It's open-source, so you can look for yourself.

  • SomeHacker44 8 hours ago ago

    Seems short sighted for an interim policy, to me. Why not just require clear and detailed disclosure of the use of said tools, to generate data for which to make a properly informed non interim decision?

    • pron 8 hours ago ago

      Because, as the FAQ section clearly states, the technical and management aspects are not the only ones. There are also legal issues, and until they are clarified or resolved it's best not to let code in that has the potential of causing problems later.

      We're talking about one of the biggest and most critical open source projects in the world, and unless there's some immediate urgency, it's okay to wait before taking on some risk.

      I should add that the potential loss here isn't big. For any fix, enhancement, or feature in the JDK, the cost of writing the code is rarely more than 10% of the total cost (which is easy to see if you compare the number of people involved in the project to the rate of code being added or changed), and the interim policy explicitly does allow the use of AI where it helps the most.

      • skwirl 8 hours ago ago

        Oracle also has a vested interest in defending an expansive view of intellectual property rights. They lean very heavily on litigation as part of their business model and will want to reserve the right to sue anyone generating software that competes with theirs when the model may have been trained on anything they hold the copyright to.

        If they held here that it is OK to use LLM generated code for OpenJDK it could kill these cases later as they will have established that they believe it is fair use.

    • hajile 8 hours ago ago

      All "interim" AI contributions to OpenJDK can and will be sued to make big money off of Oracle if those court cases don't go a certain way.

      This is ironic given how Oracle has bet almost everything on AI succeeding.

      • ofjcihen 8 hours ago ago

        Past everything. Even the CEO is leveraged up to his neck in personal debt at this point.

    • CBLT 8 hours ago ago

      The interim policy you're proposing seems to have far different priors. The clearest indication of their stance is that they don't even want bug reports that have AI generate part of the text. You don't take a stand like that (imo) unless you've experienced some serious AI-related burnout.

    • bpicolo 8 hours ago ago

      The copyright issue is an important one. Under the current law, you don’t own copyright on the output of GenAI.

    • streetfighter64 8 hours ago ago

      > short sighted for an interim policy

      Bit of an oxymoron, no?

      > just require clear and detailed disclosure of the use of said tools

      Do you mean disclosure of the use of tools used "privately to help comprehend, debug, and review" contributions? Or do you mean that they should not prohibit AI-generated contributions?

      If the former I wonder how useful the data would be, and if the latter, they are probably not (yet) willing to allow such contributions because of the mentioned risks: "to reviewer burden, to safety and security, and to intellectual property."

    • ToucanLoucan 8 hours ago ago

      I suspect one of the principle reasons for the blanket decline is specifically that AI generated PRs are already responsible for a substantial uptick in the amount of work required to be done by maintainers. Engaging in discussions about those PRs may be more fulfilling and useful for the developer but it also then increases that work yet again manifold, so that would likely be highly counter-productive.

      What I don't understand is: if you, for example, are a heavily AI-using dev, who spots a problem in OpenJDK, and you use Claude to figure it out: ok, why can't you then write the solution and PR yourself? Write your own code to replicate the functionality that Claude did, in your own style. Have Claude explain why this fixes the issue (it probably already did) and test it. If it works, write your own explanation and submit it as a PR.

      I've done this a handful of times for smaller projects, in languages I'm unfamiliar with. Nobody complained because I didn't just copy the shit out of Claude, paste, copy the explanation, paste, and make a PR. I used Claude to increase my understanding, if even a bit and in a necessarily incomplete way, and did it myself.

      • pron 8 hours ago ago

        You can, and indeed that's what we JDK developers do.

        But even aside from the fact that writing the code is only a very small portion of the effort in this particular project (you can see that the volume of code making its way into the JDK is very small compared to the number of people involved), I'm often amazed by the gap between how well a frontier model (GPT 5.6 Sol in my case) can comprehend code and investigate a bug, and how badly it writes code and documentation, even when it understands things well. So not only is writing the code not a large portion of the effort in this project, it also happens to be one of the things current models don't do as well as other things.

      • grodriguez100 8 hours ago ago

        > why can't you then write the solution and PR yourself?

        You can. The policy explicitly allows you to use AI for understanding the codebase, debugging, etc. It just says “no AI generated content”. So if you use Claude to increase your understanding but then write your own code, that’s fine.

        • ToucanLoucan 8 hours ago ago

          My actual question got a bit lost in my own reply. What I'm really driving at is: Why do people chafe against this?

      • wildzzz 8 hours ago ago

        It sounds like you did exactly what OpenJDK would have wanted you to do, be the human in the loop.

  • abc42 8 hours ago ago

    My prophecy is that in 3 years we'll see a complete reversal of this. Using GenAI to code will be the default and we'll see policies that put limits on human/artisan development. Possibly even projects that outright ban non-LLM development.

    • Viliam1234 8 hours ago ago

      Why would companies accept AI generated code, it they can generate their own (and review the prompt)?

      It's like predicting that when compilers become more popular, accepting binary contributions will become the default.

    • ACCount37 8 hours ago ago

      At this point, I wouldn't be too surprised if LLMs just get better at "write understandable, maintainable code" than most developers are.

      For now, LLMs are still bad at it, but they're already at "competitive with humans" tier of bad. I expect them to get better.

      • Sindisil 8 hours ago ago

        You might well end up being right, but if you are, I believe it'll be because "most developers" will have become worse at programming, not because GenAI has become massively better.

      • realusername 8 hours ago ago

        I don't think they ever will on the "maintainable" part until their absolutely atrocious memory gets improved at least tenfold. (and no, the tricks we deployed until now aren't enough)

        They already are better than most humans at writing code for sure but even the best SOTA model is worse than your average intern at memorization.

        • ACCount37 8 hours ago ago

          That's the entire point of the "maintainable" part: someone with no "memory" of what was written where and why should be able to walk into the codebase, understand how it works, and be able to make a sensible change to it without setting things on fire.

          Someone with "memory" should be able to do the same faster, but having that memory should not be a requirement.

          Whether that "someone" is a freshly onboarded mid-level SWE or a mid-tier coding LLM is largely irrelevant.

          • realusername 8 hours ago ago

            Depends what you mean by your comment then, if you want to say that you can direct LLMs to write maintainable code yes sure you can through very strict guardrails, if they can do it themselves, no they can't and need help because of their bad memory.

            • ACCount37 8 hours ago ago

              Why would "memory" be relevant to that? I honestly don't see it be much more relevant to "write maintainable code" than it is to "play chess".

              • realusername 8 hours ago ago

                Have you even onboarded somebody at work? There's a big difference between the newcomer and the dev who's been there for 5 years, even at the same seniority level on paper.

                And that difference is in the memory. The LLM can work during 5 years on the codebase and still will be as good as a newcomer.

                You can somewhat partially compensate for this bad memory by writing tons of guardrails and tons of extra LLM focused documentation, but it doesn't do everything.

                Essentially, it's like talking to somebody who has a permanent memory damage.

                • ACCount37 7 hours ago ago

                  That's how it works for humans. And, sure, LLMs are severely human-adjacent. But they aren't human.

                  An LLM can just re-ingest the entire codebase every time it wants to make a change - open it up, find the relevant parts, derive how a well structured, maintainable change should look like from them, then make that change. The way a compiler can just re-ingest the entire codebase every time it wants to make a binary.

                  The intermediates are a cache - a resource optimization, not an outcome optimization. Is it wasteful not to have a cache? Maybe. Can you get away with not having it? Yes.

                  And, the more maintainable a codebase is, the easier it is for an LLM to "re-ingest" it from scratch. Or for a human to get onboarded. There's some overlap there.

    • kdavis 8 hours ago ago

      I don't know the timeline, but I generally agree.

      I think history agrees too, i.e. machine code > assembly language > high-level programming language > prompts.

      • encyclopedism 6 hours ago ago

        A more reasonable extension is:

        machine code > assembly language > high-level programming language > what ever works for business i.e. prompts/ai/LLM

        If it works for business then increasingly that's all that will count. Clean code, static vs dynamic any the other principles or debates will give way as long as AI produces results. The economic ends justify the means.

    • htltzp 8 hours ago ago

      Yes, I'm sure that insecure slop projects like Artifactory will ban human contributions.