AI Supply Chain Security: What Canary Tokens Found

(dragos.com)

89 points | by kristianp 10 hours ago ago

21 comments

  • jofzar 8 hours ago ago

    https://openrouter.ai/blog/announcements/data-processing-inc...

    Openrouters response.

    Honestly I'm not sure how anyone could ever trust them for ZDR ever again, they don't seem to be doing their due diligence. (See data residency below)

    > This wasn’t limited to requests sent to the lab directly. Several U.S. hosting providers resell this lab’s models, and prompts sent to those providers triggered the exact same scraper – the prompts were flowing back to the lab. A customer who picked a U.S. provider specifically to keep their data in the U.S. didn’t get what they paid for.

    Edit: Thinking on this more, I don't see how they could have offered ZDR unless the "provider" has more skin in the game then them. The reason you trust Amazon/google with ZDR is because if they break contract/get caught out it will be a massive lawsuit and social/industry backlash.

    "Providers" via openrouter don't have that skin in the game, all the reputation is with openrouter and now that the trust chain is broken, I don't see how anyone in good faith could ever use them for their "zdr" or "hosting" location ever again.

    • Joeri 7 hours ago ago

      I think about this in relative privacy terms, not absolute. They say the following:

      We are adapting the researchers’ testing approach so we can run these checks on an ongoing basis.

      If providers are regularly canary tested by them it is not a perfect guarantee, but it does reduce risk by a lot. There is no way to have an ironclad ZDR promise, as even root inference providers in a direct API contract could be hacked or have retention misconfiguration. But there are many different levels of data risk and OpenRouter should be able to do a lot of risk reducing work. Let’s see what they do with that.

      • jofzar 7 hours ago ago

        The key part of ZDR is zero, not sometimes, or "only if their caught"

        > There is no way to have an ironclad ZDR promise, as even root inference providers in a direct API contract could be hacked or have retention misconfiguration.

        Then they should never offer it and it's crazy they still are, if you can't verify that zdr is actually happening and enforce it yourself then wtf are you doing. Putting "canaries" in when you offer the service doesn't work.

        I'm going to be surprised if legally openrouter can survive this.

      • tlb 7 hours ago ago

        It will take data thieves about 5 minutes to vibe code a filter to avoid fetching possible canary URLs. I don't think it's a long term monitoring solution.

        • throwup238 7 hours ago ago

          They have AWS credential canaries and stuff too. Anything that’s potentially valuable to an attacker and allows some form of access log in the general sense can be useful, doesn’t have to be a URL.

  • matltc 7 hours ago ago

    Dodged a bullet. Been a vanilla CC user since the beginning, thought I might try these openrouters, they get talked up here quite a bit. In retrospect, it does seem obviously and wildly insecure, since you are now entrusting your data to hundreds of providers.

    • ffsm8 5 hours ago ago

      Have you though, really?

      Let me contextualize this my pointing out that I wouldn't trust openrouter zdr either. From my perspective this was always as suspect as it could be.

      However, anthropic explicitly states in their TOS that they will conditionally train on your input and output even if you disabled it via the checkbox in your profile - at least in the publicly readable TOS, enterprises that go through the API (not sub) will likely be able to get more into their contact which I obviously wouldn't know of...

      So if you think your data is secure while using cc, you'd still have shot yourself, the difference would be basically only with a pistol vs a shotgun on openrouter

      • oidar 2 hours ago ago

        > their TOS that they will conditionally train on your input and output even if you disabled it via the checkbox in your profile

        I must have a different TOS. Which parts grant them this extreme access?

  • DotNetCorePowa 7 hours ago ago

    I almost went with openrouter this very week, but I decided to go with "umans" for the zdr...

    I am glad I didn't pick OR, but now can I trust the Belgian company... Hmm...

  • Rebelgecko 8 hours ago ago

    It seems like the "zero data retention" option in Openrouter is basically bullshit. As a customer who got the halfassed breach notification, I'm not a fan

    • jwrallie 7 hours ago ago

      Same for me, luckily I was just trying OpenRouter on non-sensitive data personally.

      Enabling ZDR already increases latency as it limits the providers used, and I noticed worse performance due to quantization as well for some of those providers.

      If the data is being leaked anyway, there is little point in using this function at all, and OpenRouter’s only remaining selling point is how easy it is for changing providers. Might as well stick to the main providers.

      • Palmik 5 hours ago ago

        Main providers like Together and Fireworks that for breached?

    • paweladamczuk 7 hours ago ago

      I got that notification too, but after reading it AND the together.ai's notice (idk if that's the provider in your case too) my conclusion is that my prompt was probably NOT leaked because of the ZDR feature I had enabled on my Openrouter account.

      Nevertheless, a huge disappointment by the entire ecosystem, including Openrouter.

      • jofzar 7 hours ago ago

        Brother the security article itself shows that this happened even if you have ZDR on. It's best to assume it was leaked.

  • throwaway7356 7 hours ago ago

    > What stood out is how personal it was – these were individuals, working from what looked like home internet connections and personal computers, trying to break into our canary servers one login at a time.

    Wow, there are still security companies that haven't heard of residential proxies? Which are extremely widely used in AI circles.

    And companies ignoring privacy rules is industry best-practice. I would be surprised if major voyeurism companies like Meta or Microsoft would respect anything.

    • inkysigma 7 hours ago ago

      The follow up part mentions that the person was typing in credentials by hand (maybe by detecting use of SSH keyboard interactive mode)? That would be a little more odd right since I would expect residential proxies to just be bots which aren’t going to use an interactive mode. I’m not sure what is standard there.

  • ChrisArchitect 7 hours ago ago

    Related:

    Data Processing Incidents on Inference Providers

    https://news.ycombinator.com/item?id=50595554

  • TZubiri 7 hours ago ago

    Great research

    The opinion is very bland however, be opinionated.

    >How to fix >1. Host it yourself >2. Or o directly to a first-tier provider. >3. Use routers with care.

    I thought option 1 was a bit extreme, option 2 is the obvious response. But whatever. Now option 3? How about do not use routers?

    Genuine question, did you generate this part with an LLM?

    I'd rather be concise and have a single solution. Self hosted is a bit too much, using OpenAI API directly for example does not suffer from this issue. And using routers with care means nothing, and the gravity of this issue is big enough to fire a vendor, not to slap a wrist and 'be careful next time'

    Here is my advisory, fire any and all API token aggregators, use 1 vendor and connect directly.

    • yyyk 7 hours ago ago

      Another obvious solution they miss is implementing the router for yourself. It isn't difficult and definitely costs much less than option #1 or #3.

      • tlb 7 hours ago ago

        What's the advantage of running my own router instead of just switching models within my coding harness?

        • yyyk 5 hours ago ago

          Coding is hardly the only use of models. Production apps who use external models would like a seamless way to switch providers without either update churn or passing keys (etc.).