OK, I think I understand what this is about: the vulnerability that they reported (and Microsoft fixed) is that there was a trick you could use to run your own code with root privileges inside the container - when the system was designed to have you only execute code as a non-root user.
It turned out not to really matter, because the container itself was still secured - you couldn't make network requests from it and you couldn't break out of it, so really all you could do with root was mess up a container that only you had access to anyway.
I would give the one engineer the credit for doing things better, not Microsoft. Microsoft overall culture of security is terrible. Look at the CISA report.
Okay, so I give the team that put this together credit. Hopefully the parent company sees based on this that it's worth letting teams invest more in quality and security work, over features.
They're not. It's better to think of Copilot as a collaborative storytelling session with a text autocomplete system, which some other program is rudely hijacking to insert the result of running certain commands.
Sometimes the (completion randomly selected from the outputs of the) predictive text model goes "yes, and". Other times, it goes "no, because". As observed in the article, if it's autocompleting the result of many "yes, and"s, the story is probably going to have another "yes, and" next, but if a story starts off with a certain kind of demand, it's probably going to continue with a refusal.
funny how it sounds kind of the opposite of how people might work. Get enough 'no's from someone and they might finally cave in. get enough 'yes'es and they might get sick of doing everything you ask.
In the modern world vulnerabilities are stacks. Asserting that "the container itself was still secured" is just a statement that the attackers didn't find anything there. But container breakouts and VM breakouts are known things. All it takes is a few mistakes in configuration or a bug in a virtio driver or whatever. This is a real and notable result.
The problem is that you're encouraging people to keep stuff like this to themselves until they can use it to perform an exploit that they'd get paid for, which is the opposite of what Microsoft wants - they'd much rather you report it now so that if an exploit does get found that requires root they would potentially be protected.
The simple question for Microsoft to answer is - does it matter to them if attackers have root access on the container? If the answer is yes then the bug bounty for root access should at least pay something to encourage reporting. If the answer is no then this shouldn't have been marked as a vulnerability because root access is not considered a security issue.
But a $5 wrench isn't a critical security vulnerability just because someone somewhere might one day find the right person to apply it to to extract important credentials.
Not really the right metaphor. A $5 wrench isn't a "vulnerability" because it's $5! Tools that are accessible to everyone are part of the threat model, not something you can eliminate or avoid. This trick is novel and new.
Like, consider your personal cult was built around an "unopenable" bolt-tighted box. Then someone invents the wrench in an attempt to open it. That would be a clear "security vulnerability", right?
Not a serious one if all the wrench actually gets you is access to the room that contains the box that no known tool can open, which is a closer analogy to what happened.
And an exploit that breaks out of the sandbox is not really anything if it needs root to work... so if a hacker had those two MS wouldn't care about them selling those bugs because both of them are not serious. See, perfect security and it didn't cost them anything.
Again, though, you're taking "all that gets you" as a prior when (abandoning the metaphor) container and VM escapes are routine vulnerabilities. They just weren't the subject of this particular team who wanted to hack on AI. You don't do security analysis by presuming the absence of vulnerabilities!
Modern security is defense in depth. The AI pre-prompting setup was the first layer, and it was escaped. The UID separation inside the container was another, and it was broken. The container would have been next. And hopefully there are network firewalls and egress rules on top of that, etc... And all of those can and have failed in the past.
Microsoft have a bug bounty program which is credible and well run.
Suing people who responsibly disclose security issues to you is a disastrous thing to do. Word spreads instantly and now you won't get any responsibly disclosed bug reports in the future.
It's crazy to me that someone can write a post called "How We Rooted Copilot" when in reality they got root in an ephemeral python sandbox container that was locked down so much that they couldn't do anything.
I read "rooted copilot" and I think they got root on a vm that is core to copilot itself.
A much more accurate title would be "How We Rooted the Copilot Python Sandbox"
So am I just missing something or could you create a network connection to the "outside" world (clearly by finding your way around the local network? Start fuzzing the router endpoint, Etc. Or is Microsoft able to provide these containers where their customers can get root access to them without them having any risk of exfiltration or exploitation?
Back when openai released python interpretation it was trivial to do what they did there. There was no open network access, the only thing of interest was a little insight in to how their developers program. A couple of internal configuration files.
That repo, and its contributors are MS/Azure employees working on the service for running python code in a container. I don't know why it's under a personal account. Though it says it's a fork from an Office repo that I can't find.
Guys, chatbots are mostly token generators, they don't run programs e give you responses...it's not a simple shell program, it computes things in GPU and return tokens, in which are translated back to English.
Not really. You're referring to agents, but the model doesn't always require agents, and the public chatbot is not connected to a shell freely evaluating arbitrary commands.
Earlier LLMs used to be a goldmine for company secrets (when it learned documents that shouldn't be on public internet). Most of it seem to be scrubbed now.
> Earlier LLMs used to be a goldmine for company secrets (when it learned documents that shouldn't be on public internet).
Sounds fake. LLMs don't usually memorize things that appear once in their training set anyway, nor have I heard about major issues accidentally training on a bunch of non-public data.
I can see how someone would believe it to be true though, since LLMs can easily hallucinate in a way that looks like this is true.
This reminds me of that one time after working at a company for 4 months they informed me they were in a middle of an IP lawsuit which is part of the reason they hired me to rewrite the front end without knowing that was going on. That was f*(ked for reasons.
Whatever the case, the only time people look at your social media history is to look for attacks and the only reason they will look at a company's slack messages and emails are to look for attacks during discovery.
I would argue that company secrets are mostly useless for the company but very, very useful to other companies. For this reason, there should be retention policy of a day or two for almost all communication unless it is important, required by law, or documentation. And, definitely do not share that information with the public without good reason.
The bigger issue is around "material non‑public information" in stock market terms - things like unreported sales figures which someone could use to make trading decisions.
Using that information for trading is illegal, but so is exposing that information outside of approved channels.
Because its hard to define the parts that are really sensitive. At our work people must classify every document but a lot of people choose public for everything because it doesn't enforce any restrictions. So they can just dump it in a folder and share it with the whole company. This is not what we want them to do obviously but people are lazy, don't like to create access lists. But anyway it means we can't rely on the classification. And indicator detection like credit card and social security numbers is far from perfect. A lot of sensitive info will just be text, like about new products being developed. 3D models, code, strategy emails.
Also, if people start rooting around in everything they can take things out of context. If I send a message to my boss that I think that something we're doing is stupid, if that were public it could make some waves even though internally it's inconsequential because I'm a nobody. Also, many documents might have one or two bits that hint to really important information and having them can help finding those
As you probably know, there's tons of information in a multinational and the hardest part is finding the right stuff. This is one of the main tasks I use Copilot for. Also because outlook and SharePoint search are really terrible though. If those actually worked I wouldn't need copilot so much.
Because "mostly" does a lot of work in that sentence. Companies, like militaries, keep secret a lot of information that would be safe to release because they don't know which bits are highly unsafe.
At most of the companies I've worked, low-grade managers love to hoard secrets. It makes them feel powerful. Someone gets promoted from Lower Level Manager Grade 4 to Lower Level Manager Grade 5 and they feel all "Oooh! Look at the new things I know!"
My mother-in-law is like this with knowing what various relatives are doing. Being the gatekeeper of knowledge gives her imagined power. I guess it's just part of the human condition.
I know sysadmins and programmers who behave exactly they same way. They could give you permission or a script to do the thing you need to do but they'd rather have you come to them and ask them to do it. Gives them a sense of purpose, I guess.
Being such a person that fixes lots of stuff for other people nothing I do is secret but learning to do it seems too hard for most. What I do is try to delegate if I find people that do want to learn.
If someone shows me they are good at something they are going to have to expect being sent trickier problems.
Sometimes it might seem like I keep things a secret. I am probably just having a bad day.
That has an awful lot to do with what "the thing" is. I'm sure there are a few people out there doing it just to feel more important, but often there's a good reason for denying someone access - either it's just a terrible idea to begin with or they don't know you well enough to trust you without someone else (i.e. their boss) specifically requesting it.
I could be off base here about your experience, but I know that some people made the same comments about me when I pushed back on sharing dangerous credentials with inexperienced coworkers. Damned if you do, damned if you don't.
It may depend on what the script is for or the system being used. Segregation of duties is a risk mitigation principle of ISO 27001 to reduce fraud, waste, and error.
That's why corporate espionage is a really lucrative industry?
Of course it depends what secrets. 99% will just be internal process drivel and inter departmental bickering but there's some real important stuff in there too.
I looked for an alleged case of an LLM apparently reproducing email signatures—but couldn’t find it exactly, and of course many email signatures have been published over the years, especially on newsgroups. (Maybe it was conspiratorial kind of thinking from web commenters assuming ChatGPT was training on emails users were feeding it, which as mentioned certainly doesn’t need to be the case.)
When companies (non-"tech") started adopting them they also had no "guardrails" for content outside what the intent of such products were (dunno what the standard term for this is).
There was a boba tea company that had a free, no-sign-in required LLM that I used to generate a few bash scripts before ChatGPT free-tier started.
> We reported the vulnerability to Microsoft in April and they have since fixed it as a moderate severity vulnerability. As only important and critical vulnerabilities qualify for a bounty award, we did not receive anything, except for an acknowledgement on the Security Researcher Acknowledgments for Microsoft Online Services webpage.
I guess it makes sense that a poor little indie company like Microsoft can't pay bug bounties. Surely no bad things will come out of this.
> Now what have we gained with root access to the container?
> Absolutely nothing!
> We can now use this access to explore parts of the container that were previously inaccessible to us. We explored the filesystem, but there were no files in /root, no interesting logging to find, and a container breakout looked out of the question as every possible known breakout had been patched.
I'm sure there are more ways to acquire root. If Microsoft pays out for one, they have to pay out for all, and it seems pretty silly to do that for something that's slightly unintended but not dangerous.
It is hard to answer that since the stack is so convoluted. Some parts are forced on the user. Copilot is built into Microsoft Office workplace applications.
If you break out of a container, do you have access to the same system that serves these applications? Who knows, it looks like a gigantic mess.
Severity is based on impact. What was the impact here beyond single container and that specific user instance? Feels like moderate was okay, or even too high.
It's still good for reputation. This is by a researcher at a company, so a benefit for both of them. Plus if we didn't have bug bounty programs, they'd have to willingly work at Microsoft to do this research.
This could have turned badly in terms of reputation if they had tried to complain that the vulnerability should be critical, e.g. or using other ways to seek attention for not getting bounty, but current way was rather neutral way.
It's why I don't understand why people believe in "open source". Why would I contribute free dev work to a billion dollar corporation?
I do believe in "Free Software" which is contributing free dev work to my fellow man for the benefit of all man mankind.
Free software and open source are two ideologies for the same thing. Free Software is the ideology of developing the software for the benefit of mankind (it's sometimes termed a "political" stance but I see it as an ethical stance). Open source is the ideology of saving money at a corporation by not paying the developers. Sure open source can benefit mankind but will only develop corporate software for money. When developing on my own time, I will focus on software that either personally benefits me or benefits other regular people.
You need to think it in a different manner. When you have AGPL code, then it benefits mankind more than corporations. There's a Harvard report on value of open source to society based on how much money corporations put in.
Today linux is working nicely on desktops (even though it's not the year of linux) and is heavily dominated by corporations. The parts where linux doesn't do well are exactly parts without corporate support.
Software is becoming complex enough that it's not possible for a single company to just even maintain a compiler let alone an office suite. Its perfect ground for either one company having monopoly or an free software (not open source) being a base for masses.
Lichess, the gazillion of self-hosting software. There are many examples of free software that are exclusively (or let's say predominantly) used in noncommercial environments.
In any case, I agree with the commenter, and I think that developing a software which is also used by companies is different from looking for vulnerabilities in the context and scope of a bug bounty program for a specific company. Yes, you could argue that users of said company are going to be more secure, but it's evidence t like even in this case the company is the direct beneficiary.
> Why would I contribute free dev work to a billion dollar corporation?
The billion dollars company contributed more to your startup than you do to them. Microsoft provides:
- VSCode,
- Hosts all NPM repositories. You know, the ones small startups are too lazy to cache (also because it’s much harder to cache NPM repositories than Maven) and then you re-download them at each build,
Meh it depends whether you use those things of course. There's other IDEs, other languages. And Microsoft isn't doing this out of charity. A lot of the really useful plugins are not working on the open source version, so people that use them provide telemetry which is probably valuable. Or they use it as a gateway to their services like GitHub Copilot.
If a mega corporation gives you something for free it's always more beneficial to them otherwise they wouldn't do it in the first place.
So, no OSS contribution is valid unless you are using this very library?
Did Microsoft contribute more to the OSS world, or did the OSS world contribute more to Microsoft? I pardon Microsoft because they have donated Typescript, which is a true civilizational progress. You could say the OSS world has contributed to Microsoft because they’ve given them a real OS, which they didn’t have inner expertise to develop. We’re even.
Now you sound like you have a beef against large companies and would find any argument against them. Some guy once told me that I didn’t increase my employees by 30% out of benevolence, but because I must be an awful employer. See, why else would I increase employees.
This behavior is actively harmful to the rest of the world. You are depriving good actions from a “thank you” and hence you are depriving recipients of good actions from more of them. With this attitude, the world becomes exactly like you project it to be: Shitty.
The open source ecosystem was perfect before Microsoft tried to meddle, assimilate and destroy.
Microsoft has destroyed several open source projects by infiltrating them with mediocre MSFT employees.
Microsoft bought the GitHub monopoly in order to control open source further. Microsoft then stole and violated the copyright by training "AI" on the GitHub open source.
Microsoft finances influential open source organizations like OSI in order to make them more compliant and business friendly.
The useful projects are tiny compared to the entire open source stack. Paying for NPM repositories is a goodwill gesture and another power grab.
> So, no OSS contribution is valid unless you are using this very library?
You said Microsoft contributes to my start-up. That's only true if we actually use it.
> Now you sound like you have a beef against large companies and would find any argument against them.
I certainly have beef with Microsoft in particular yes. And most big tech. I work a lot with Microsoft people and they're always trying to get us to do things that benefits them and not us (and I hate the attitude of a mere supplier trying to tell us what to do). Always trying to get us to evangelize their stuff which is mostly mediocre, dumping constant rebranding campaigns on us etc.
I'm not looking for arguments but I do hate the mega corporations and I don't believe in any benevolence on their side. I think the world would be much better off without them. They have way too much influence on the world. They should have none, after all they are not people and can't vote.
I also don't appreciate their contributions to eg Linux and OpenStreetMap. There's always ulterior motives. Like giving running on their cloud a step up, embedding their own IP like RedHat/IBM do (and Canonical always tries but fails at). Most of the contributions are from big tech now. I don't believe in a 'win/win' scenario involving corporations.
But I'm very much against unbridled capitalism and neoliberalism yes. I think it causes most of what's wrong with this world, from unequal distribution of wealth, extreme pollution, wars (influenced by the MIC) etc. Even the heavy political polarisation. The feud between the democrats and republicans is really just a proxy war for big corporate interests. Running a campaign requires so much trouble that it's no longer possible with a real grassroots movement.
But anyway this is my opinion. Take it as it is or don't. You have the right to you own opinions of course! I'm aware my opinion isn't very nuanced.
> This behavior is actively harmful to the rest of the world. You are depriving good actions from a “thank you” and hence you are depriving recipients of good actions from more of them.
Nah. Microsoft doesn't care what I think. I'm nothing but an ant on the floor to them.
Besides, they are doing this for reasons. The thank you isn't one of them. Hosting npm is peanuts for a big cloud provider, just advertising really. And it gives them a lot of metrics about the usage of libraries and from where. And VS Code, I'm sure they had a discussion about "what's in it for us in the long term" with some big envisioned benefits. You don't start a big project without that.
With most of their other products it's more clear. Like edge, they clearly made this to lock corporate customers further into their ecosystem (it can be deeply locked down which corporate IT loves because they enjoy playing BOFH) and for customers for upselling to their services. It's not better than Google's, they just replaced Google's online services with their own.
I think the argument is that when big companies make use of stuff, it gets more scrutiny and occasionally they contribute back improvements, and the occasional unicorn gets actual man hours paid for improving it. So if your project gets big enough, it's beneficial. But you have to have a MIT/BSD license usually, because companies will normally stay away from GPL.
I know maintainers of projects have been hired directly by companies using their code as it is the most expedient way forward. Others might just offer up enough money to get the maintainer to take up a few of their specific issues/requests in a way that makes it worth their while. Just because someone is working on a project that is open source does not mean that money cannot be involved in the development. The company paying that money knows that the updates released as a normal part of the project will be available to anyone else using it as well.
No, we can't say. I'm not an asshole, it helps people, and companies shun GPL licenses. That's not a valid comparison. Microsoft can go fuck itself, people around me love my software and it improves their lives.
It's... 100% a valid comparison? The point is that doing free vulnerability research isn't irrational, not that doing open source work is bad. You're twisting yourself into a pretzel trying to keep the original argument alive.
It's called "I use the software, I already want to improve the software I'm using, so after I improve it I'll contribute the improvements I've already made to the broader community."
Granted, I myself have been guilty of not giving back to the open source community this way in the past, but I won't pretend that was reasonable or ethical of me!
edit: after reading some commemnts, i realize i may have meant to say "free software" instead of "open source"
People people who did bootcamps and thus are too risky to hire for most roles and cannot get into the standard CS hiring pipeline. Especially now that junior roles are drying up.
In professions like fashion, virtually everyone seems to at some point.
M$: If you're not going to send any money, send some swag. Make it cool and hackers will wear it, and now you have them advertising for you and possibly even want to work for you. Culture is a tool, and hackers have culture, so learn how to use it.
They didn't find anything they could do with it but that container isn't there for no reason. I agree with the rating but it's nonetheless worrying. You don't leave the house you bought unlocked because there's nothing in it to steal yet.
It's wild how easy this was. I feel like we're really in the wild west era of security with these AI tools -- reminds me of early Web 2.0 days, like when "samy is my hero" hit and Myspace didn't even have a security team. I anticipate many high-profile incidents before they figure out how to tame this beast.
I don't think there's really much "AI" involved in this; this is basically like breaking any hosted code IDE. I get that an LLM was the direct vector, but the underlying security issue is common to everything that runs remote code.
OK, I think I understand what this is about: the vulnerability that they reported (and Microsoft fixed) is that there was a trick you could use to run your own code with root privileges inside the container - when the system was designed to have you only execute code as a non-root user.
It turned out not to really matter, because the container itself was still secured - you couldn't make network requests from it and you couldn't break out of it, so really all you could do with root was mess up a container that only you had access to anyway.
I have to give Microsoft props here. Most companies don't bother to lock things down well enough, but they were thorough.
I would give the one engineer the credit for doing things better, not Microsoft. Microsoft overall culture of security is terrible. Look at the CISA report.
Okay, so I give the team that put this together credit. Hopefully the parent company sees based on this that it's worth letting teams invest more in quality and security work, over features.
Microsoft has islands of security excellence in what these days is a sea of mediocrity.
What CISA report?
Not OP, but guessing they were referencing this one:
https://www.cisa.gov/resources-tools/resources/CSRB-Review-S...
I’m guessing they mean this one:
https://www.cisa.gov/news-events/bulletins/sb25-167
> Microsoft--Microsoft 365 Copilot
> Description Ai command injection in M365 Copilot allows an unauthorized attacker to disclose information over a network.
> Published 2025-06-11
> CVSS Score 9.3
> Source Info CVE-2025-32711
https://www.cve.org/CVERecord?id=CVE-2025-32711
And maybe they are referring to this engineer from the linked advisory notes?
https://msrc.microsoft.com/update-guide/vulnerability/CVE-20...
> Acknowledgements
> Arantes (@es7evam on X) with Microsoft Aim Labs (Part of Aim Security)
I don’t know specifically how this container was implemented, but Microsoft has a standard way to do isolated Python sandboxes: https://learn.microsoft.com/en-us/azure/container-apps/sessi... Hopefully this feature is using that or something similar.
It seems weird to me that copilot sometimes refuses to execute code but sometimes allows it. What exactly are they aiming for?
They're not. It's better to think of Copilot as a collaborative storytelling session with a text autocomplete system, which some other program is rudely hijacking to insert the result of running certain commands.
Sometimes the (completion randomly selected from the outputs of the) predictive text model goes "yes, and". Other times, it goes "no, because". As observed in the article, if it's autocompleting the result of many "yes, and"s, the story is probably going to have another "yes, and" next, but if a story starts off with a certain kind of demand, it's probably going to continue with a refusal.
funny how it sounds kind of the opposite of how people might work. Get enough 'no's from someone and they might finally cave in. get enough 'yes'es and they might get sick of doing everything you ask.
In the modern world vulnerabilities are stacks. Asserting that "the container itself was still secured" is just a statement that the attackers didn't find anything there. But container breakouts and VM breakouts are known things. All it takes is a few mistakes in configuration or a bug in a virtio driver or whatever. This is a real and notable result.
If they had found and reported a container breakout I expect they would've got a bug bounty from it!
Are there any known unfixed container breakouts at the moment in the kind of systems Microsoft are likely to be using here?
The problem is that you're encouraging people to keep stuff like this to themselves until they can use it to perform an exploit that they'd get paid for, which is the opposite of what Microsoft wants - they'd much rather you report it now so that if an exploit does get found that requires root they would potentially be protected.
The simple question for Microsoft to answer is - does it matter to them if attackers have root access on the container? If the answer is yes then the bug bounty for root access should at least pay something to encourage reporting. If the answer is no then this shouldn't have been marked as a vulnerability because root access is not considered a security issue.
Presumably someone with mal-intent would sit on the root vulnerability waiting for a container breakout bug to come around.
But a $5 wrench isn't a critical security vulnerability just because someone somewhere might one day find the right person to apply it to to extract important credentials.
A container root exploit isn't a critical security vulnerability either, describing it as moderate seems fair, but it's a reasonable step towards one.
That is exactly what it is.
Propper security I depth means that when trusted actors betray the system, the damage is limited.
Not really the right metaphor. A $5 wrench isn't a "vulnerability" because it's $5! Tools that are accessible to everyone are part of the threat model, not something you can eliminate or avoid. This trick is novel and new.
Like, consider your personal cult was built around an "unopenable" bolt-tighted box. Then someone invents the wrench in an attempt to open it. That would be a clear "security vulnerability", right?
Not a serious one if all the wrench actually gets you is access to the room that contains the box that no known tool can open, which is a closer analogy to what happened.
And an exploit that breaks out of the sandbox is not really anything if it needs root to work... so if a hacker had those two MS wouldn't care about them selling those bugs because both of them are not serious. See, perfect security and it didn't cost them anything.
Again, though, you're taking "all that gets you" as a prior when (abandoning the metaphor) container and VM escapes are routine vulnerabilities. They just weren't the subject of this particular team who wanted to hack on AI. You don't do security analysis by presuming the absence of vulnerabilities!
Modern security is defense in depth. The AI pre-prompting setup was the first layer, and it was escaped. The UID separation inside the container was another, and it was broken. The container would have been next. And hopefully there are network firewalls and egress rules on top of that, etc... And all of those can and have failed in the past.
Sure, I guess, but a lot more is broken than Copilot if you assume arbitrary container escape. (I do!)
Almost certainly yes, since at that point all you're looking for is a Linux kernel LPE.
> they would've got a bug bounty from it!
Why do you think that, rather than get sued? I am curious
Microsoft have a bug bounty program which is credible and well run.
Suing people who responsibly disclose security issues to you is a disastrous thing to do. Word spreads instantly and now you won't get any responsibly disclosed bug reports in the future.
Microsoft are way too smart to make that mistake.
It's crazy to me that someone can write a post called "How We Rooted Copilot" when in reality they got root in an ephemeral python sandbox container that was locked down so much that they couldn't do anything.
I read "rooted copilot" and I think they got root on a vm that is core to copilot itself.
A much more accurate title would be "How We Rooted the Copilot Python Sandbox"
I read this as them breaking out of a Python sandbox into a container. That also squares with MSFT scoring this "moderate" severity.
So am I just missing something or could you create a network connection to the "outside" world (clearly by finding your way around the local network? Start fuzzing the router endpoint, Etc. Or is Microsoft able to provide these containers where their customers can get root access to them without them having any risk of exfiltration or exploitation?
Back when openai released python interpretation it was trivial to do what they did there. There was no open network access, the only thing of interest was a little insight in to how their developers program. A couple of internal configuration files.
This is literally the same.
How does he know that the response isn't just hallucinations?
I'm telling it because I work there and I don't recognize any of those processes.
In fact I found one script named keepAliveJupyterSvc.sh in a public repo: https://github.com/shivamkm07/code-interpreter/blob/load-tes...
That repo, and its contributors are MS/Azure employees working on the service for running python code in a container. I don't know why it's under a personal account. Though it says it's a fork from an Office repo that I can't find.
How do you figure? I don’t see anything that suggest they work for ms/azure?
It may not be a hallucination. Perhaps the Copilot code was generated from the GitHub training set?
Oh boy, this really seems to be hallucination.
Guys, chatbots are mostly token generators, they don't run programs e give you responses...it's not a simple shell program, it computes things in GPU and return tokens, in which are translated back to English.
This is very out of date. They now often trigger tooling and return the outputs of the tooling.
Not really. You're referring to agents, but the model doesn't always require agents, and the public chatbot is not connected to a shell freely evaluating arbitrary commands.
Earlier LLMs used to be a goldmine for company secrets (when it learned documents that shouldn't be on public internet). Most of it seem to be scrubbed now.
> Earlier LLMs used to be a goldmine for company secrets (when it learned documents that shouldn't be on public internet).
Sounds fake. LLMs don't usually memorize things that appear once in their training set anyway, nor have I heard about major issues accidentally training on a bunch of non-public data.
I can see how someone would believe it to be true though, since LLMs can easily hallucinate in a way that looks like this is true.
In my humble experience company secrets are mostly useless for other companies.
This reminds me of that one time after working at a company for 4 months they informed me they were in a middle of an IP lawsuit which is part of the reason they hired me to rewrite the front end without knowing that was going on. That was f*(ked for reasons.
Whatever the case, the only time people look at your social media history is to look for attacks and the only reason they will look at a company's slack messages and emails are to look for attacks during discovery.
I would argue that company secrets are mostly useless for the company but very, very useful to other companies. For this reason, there should be retention policy of a day or two for almost all communication unless it is important, required by law, or documentation. And, definitely do not share that information with the public without good reason.
The bigger issue is around "material non‑public information" in stock market terms - things like unreported sales figures which someone could use to make trading decisions.
Using that information for trading is illegal, but so is exposing that information outside of approved channels.
Then why are they secret?
Because its hard to define the parts that are really sensitive. At our work people must classify every document but a lot of people choose public for everything because it doesn't enforce any restrictions. So they can just dump it in a folder and share it with the whole company. This is not what we want them to do obviously but people are lazy, don't like to create access lists. But anyway it means we can't rely on the classification. And indicator detection like credit card and social security numbers is far from perfect. A lot of sensitive info will just be text, like about new products being developed. 3D models, code, strategy emails.
Also, if people start rooting around in everything they can take things out of context. If I send a message to my boss that I think that something we're doing is stupid, if that were public it could make some waves even though internally it's inconsequential because I'm a nobody. Also, many documents might have one or two bits that hint to really important information and having them can help finding those
As you probably know, there's tons of information in a multinational and the hardest part is finding the right stuff. This is one of the main tasks I use Copilot for. Also because outlook and SharePoint search are really terrible though. If those actually worked I wouldn't need copilot so much.
Because "mostly" does a lot of work in that sentence. Companies, like militaries, keep secret a lot of information that would be safe to release because they don't know which bits are highly unsafe.
Paranoia and not knowing which ones fall into "mostly" category :)
At most of the companies I've worked, low-grade managers love to hoard secrets. It makes them feel powerful. Someone gets promoted from Lower Level Manager Grade 4 to Lower Level Manager Grade 5 and they feel all "Oooh! Look at the new things I know!"
My mother-in-law is like this with knowing what various relatives are doing. Being the gatekeeper of knowledge gives her imagined power. I guess it's just part of the human condition.
Why limit it to low-grade managers?
I know sysadmins and programmers who behave exactly they same way. They could give you permission or a script to do the thing you need to do but they'd rather have you come to them and ask them to do it. Gives them a sense of purpose, I guess.
Being such a person that fixes lots of stuff for other people nothing I do is secret but learning to do it seems too hard for most. What I do is try to delegate if I find people that do want to learn.
If someone shows me they are good at something they are going to have to expect being sent trickier problems.
Sometimes it might seem like I keep things a secret. I am probably just having a bad day.
That has an awful lot to do with what "the thing" is. I'm sure there are a few people out there doing it just to feel more important, but often there's a good reason for denying someone access - either it's just a terrible idea to begin with or they don't know you well enough to trust you without someone else (i.e. their boss) specifically requesting it.
I could be off base here about your experience, but I know that some people made the same comments about me when I pushed back on sharing dangerous credentials with inexperienced coworkers. Damned if you do, damned if you don't.
It may depend on what the script is for or the system being used. Segregation of duties is a risk mitigation principle of ISO 27001 to reduce fraud, waste, and error.
That's why corporate espionage is a really lucrative industry?
Of course it depends what secrets. 99% will just be internal process drivel and inter departmental bickering but there's some real important stuff in there too.
Do you have any concrete examples of this? I have not seen any myself.
I looked for an alleged case of an LLM apparently reproducing email signatures—but couldn’t find it exactly, and of course many email signatures have been published over the years, especially on newsgroups. (Maybe it was conspiratorial kind of thinking from web commenters assuming ChatGPT was training on emails users were feeding it, which as mentioned certainly doesn’t need to be the case.)
Something like the top screenshot here, though:
https://www.zdnet.com/article/chatgpt-can-leak-source-data-v...
(not parent commenter but) tl;dr no
When companies (non-"tech") started adopting them they also had no "guardrails" for content outside what the intent of such products were (dunno what the standard term for this is).
There was a boba tea company that had a free, no-sign-in required LLM that I used to generate a few bash scripts before ChatGPT free-tier started.
Source?
Don't really seam to be a vulnerability?
The safety in the system is that the code is executed in a container.
Assuming the container was isolated. Which I'd assume it was.
Seems like they could have taken a shortcut by giving copilot a sudo binary to use as base64.
You would need to change ownership of the file to root also.
The important part:
I'm sure there are more ways to acquire root. If Microsoft pays out for one, they have to pay out for all, and it seems pretty silly to do that for something that's slightly unintended but not dangerous.Are you not concerned about all the other platforms that rely on containers as security boundaries between tenants? There are a lot of them.
It is hard to answer that since the stack is so convoluted. Some parts are forced on the user. Copilot is built into Microsoft Office workplace applications.
If you break out of a container, do you have access to the same system that serves these applications? Who knows, it looks like a gigantic mess.
I expect that they run their containers more isolated as virtual machines. So they have bigger problems of there is a breakout possible.
Severity is based on impact. What was the impact here beyond single container and that specific user instance? Feels like moderate was okay, or even too high.
Maybe this was their honeypot container.
I'll never understand why people do free dev work for multinational trillion dollar conglomerates.
It's still good for reputation. This is by a researcher at a company, so a benefit for both of them. Plus if we didn't have bug bounty programs, they'd have to willingly work at Microsoft to do this research.
This could have turned badly in terms of reputation if they had tried to complain that the vulnerability should be critical, e.g. or using other ways to seek attention for not getting bounty, but current way was rather neutral way.
Could say the same thing about open source software.
It's why I don't understand why people believe in "open source". Why would I contribute free dev work to a billion dollar corporation? I do believe in "Free Software" which is contributing free dev work to my fellow man for the benefit of all man mankind.
This may be a misconception. "Free software" (e.g. Linux) also benefits billion-dollar corporations and "open source" also benefits all mankind.
Free software and open source are two ideologies for the same thing. Free Software is the ideology of developing the software for the benefit of mankind (it's sometimes termed a "political" stance but I see it as an ethical stance). Open source is the ideology of saving money at a corporation by not paying the developers. Sure open source can benefit mankind but will only develop corporate software for money. When developing on my own time, I will focus on software that either personally benefits me or benefits other regular people.
I applaud your choice! I just can't think of any free software examples that don't also benefit corporations.
You need to think it in a different manner. When you have AGPL code, then it benefits mankind more than corporations. There's a Harvard report on value of open source to society based on how much money corporations put in.
Today linux is working nicely on desktops (even though it's not the year of linux) and is heavily dominated by corporations. The parts where linux doesn't do well are exactly parts without corporate support.
Software is becoming complex enough that it's not possible for a single company to just even maintain a compiler let alone an office suite. Its perfect ground for either one company having monopoly or an free software (not open source) being a base for masses.
That’s not an example of open source that doesn’t benefit corporations. Linux is amazing for corporations.
Lichess, the gazillion of self-hosting software. There are many examples of free software that are exclusively (or let's say predominantly) used in noncommercial environments.
In any case, I agree with the commenter, and I think that developing a software which is also used by companies is different from looking for vulnerabilities in the context and scope of a bug bounty program for a specific company. Yes, you could argue that users of said company are going to be more secure, but it's evidence t like even in this case the company is the direct beneficiary.
at least under some licenses like GPL/AGPL you get some code back.
Why do basic science which benefits everyone else for free?
> Why would I contribute free dev work to a billion dollar corporation?
The billion dollars company contributed more to your startup than you do to them. Microsoft provides:
- VSCode,
- Hosts all NPM repositories. You know, the ones small startups are too lazy to cache (also because it’s much harder to cache NPM repositories than Maven) and then you re-download them at each build,
- Typescript
Meh it depends whether you use those things of course. There's other IDEs, other languages. And Microsoft isn't doing this out of charity. A lot of the really useful plugins are not working on the open source version, so people that use them provide telemetry which is probably valuable. Or they use it as a gateway to their services like GitHub Copilot.
If a mega corporation gives you something for free it's always more beneficial to them otherwise they wouldn't do it in the first place.
So, no OSS contribution is valid unless you are using this very library?
Did Microsoft contribute more to the OSS world, or did the OSS world contribute more to Microsoft? I pardon Microsoft because they have donated Typescript, which is a true civilizational progress. You could say the OSS world has contributed to Microsoft because they’ve given them a real OS, which they didn’t have inner expertise to develop. We’re even.
Now you sound like you have a beef against large companies and would find any argument against them. Some guy once told me that I didn’t increase my employees by 30% out of benevolence, but because I must be an awful employer. See, why else would I increase employees.
This behavior is actively harmful to the rest of the world. You are depriving good actions from a “thank you” and hence you are depriving recipients of good actions from more of them. With this attitude, the world becomes exactly like you project it to be: Shitty.
The open source ecosystem was perfect before Microsoft tried to meddle, assimilate and destroy.
Microsoft has destroyed several open source projects by infiltrating them with mediocre MSFT employees.
Microsoft bought the GitHub monopoly in order to control open source further. Microsoft then stole and violated the copyright by training "AI" on the GitHub open source.
Microsoft finances influential open source organizations like OSI in order to make them more compliant and business friendly.
The useful projects are tiny compared to the entire open source stack. Paying for NPM repositories is a goodwill gesture and another power grab.
> So, no OSS contribution is valid unless you are using this very library?
You said Microsoft contributes to my start-up. That's only true if we actually use it.
> Now you sound like you have a beef against large companies and would find any argument against them.
I certainly have beef with Microsoft in particular yes. And most big tech. I work a lot with Microsoft people and they're always trying to get us to do things that benefits them and not us (and I hate the attitude of a mere supplier trying to tell us what to do). Always trying to get us to evangelize their stuff which is mostly mediocre, dumping constant rebranding campaigns on us etc.
I'm not looking for arguments but I do hate the mega corporations and I don't believe in any benevolence on their side. I think the world would be much better off without them. They have way too much influence on the world. They should have none, after all they are not people and can't vote.
I also don't appreciate their contributions to eg Linux and OpenStreetMap. There's always ulterior motives. Like giving running on their cloud a step up, embedding their own IP like RedHat/IBM do (and Canonical always tries but fails at). Most of the contributions are from big tech now. I don't believe in a 'win/win' scenario involving corporations.
But I'm very much against unbridled capitalism and neoliberalism yes. I think it causes most of what's wrong with this world, from unequal distribution of wealth, extreme pollution, wars (influenced by the MIC) etc. Even the heavy political polarisation. The feud between the democrats and republicans is really just a proxy war for big corporate interests. Running a campaign requires so much trouble that it's no longer possible with a real grassroots movement.
But anyway this is my opinion. Take it as it is or don't. You have the right to you own opinions of course! I'm aware my opinion isn't very nuanced.
> This behavior is actively harmful to the rest of the world. You are depriving good actions from a “thank you” and hence you are depriving recipients of good actions from more of them.
Nah. Microsoft doesn't care what I think. I'm nothing but an ant on the floor to them.
Besides, they are doing this for reasons. The thank you isn't one of them. Hosting npm is peanuts for a big cloud provider, just advertising really. And it gives them a lot of metrics about the usage of libraries and from where. And VS Code, I'm sure they had a discussion about "what's in it for us in the long term" with some big envisioned benefits. You don't start a big project without that.
With most of their other products it's more clear. Like edge, they clearly made this to lock corporate customers further into their ecosystem (it can be deeply locked down which corporate IT loves because they enjoy playing BOFH) and for customers for upselling to their services. It's not better than Google's, they just replaced Google's online services with their own.
I think the argument is that when big companies make use of stuff, it gets more scrutiny and occasionally they contribute back improvements, and the occasional unicorn gets actual man hours paid for improving it. So if your project gets big enough, it's beneficial. But you have to have a MIT/BSD license usually, because companies will normally stay away from GPL.
I know maintainers of projects have been hired directly by companies using their code as it is the most expedient way forward. Others might just offer up enough money to get the maintainer to take up a few of their specific issues/requests in a way that makes it worth their while. Just because someone is working on a project that is open source does not mean that money cannot be involved in the development. The company paying that money knows that the updates released as a normal part of the project will be available to anyone else using it as well.
No, we can't say. I'm not an asshole, it helps people, and companies shun GPL licenses. That's not a valid comparison. Microsoft can go fuck itself, people around me love my software and it improves their lives.
It's... 100% a valid comparison? The point is that doing free vulnerability research isn't irrational, not that doing open source work is bad. You're twisting yourself into a pretzel trying to keep the original argument alive.
It's called "I use the software, I already want to improve the software I'm using, so after I improve it I'll contribute the improvements I've already made to the broader community."
Granted, I myself have been guilty of not giving back to the open source community this way in the past, but I won't pretend that was reasonable or ethical of me!
edit: after reading some commemnts, i realize i may have meant to say "free software" instead of "open source"
Well a lot of people do this kind of work to be able to commit crimes.
It mostly pays in career benefits. Same reason why plenty intern for free.
Who is interning for free as a software engineer?
Me
People people who did bootcamps and thus are too risky to hire for most roles and cannot get into the standard CS hiring pipeline. Especially now that junior roles are drying up.
In professions like fashion, virtually everyone seems to at some point.
i don't think they did the work for them. they just reported it to them.
M$: If you're not going to send any money, send some swag. Make it cool and hackers will wear it, and now you have them advertising for you and possibly even want to work for you. Culture is a tool, and hackers have culture, so learn how to use it.
As you’ll see elsewhere, “root” got them literally nothing. They tried but there was nothing to be had.
They didn't find anything they could do with it but that container isn't there for no reason. I agree with the rating but it's nonetheless worrying. You don't leave the house you bought unlocked because there's nothing in it to steal yet.
More like leaving your front gate unlocked.
There was a time in programming that tried to avoid monstrosities like the Python scientific data stack combined with Copilot integration hacks.
That time produced qmail and postfix. We are back to the early 1990s.
It's wild how easy this was. I feel like we're really in the wild west era of security with these AI tools -- reminds me of early Web 2.0 days, like when "samy is my hero" hit and Myspace didn't even have a security team. I anticipate many high-profile incidents before they figure out how to tame this beast.
I don't think there's really much "AI" involved in this; this is basically like breaking any hosted code IDE. I get that an LLM was the direct vector, but the underlying security issue is common to everything that runs remote code.