Black high school students sue after AI mistakes Doritos bag for gun

(lawyerscommittee.org)

90 points | by DeepLogin 9 hours ago ago

35 comments

  • ChrisRR 31 minutes ago ago

    The need for for gun detection AI at US schools is already massively concerning

  • npteljes 17 minutes ago ago

    AI is not the problem here.

    >The AI surveillance system’s error prompted the false alarm and set off a chain of human failures >armed police were dispatched anyway because school and police officials failed to have sufficient protocols in place to prevent cancelled alerts

    This is a mistaken setup, by humans. The human override (cancelling the false alert) hasn't worked.

  • thereader12 2 hours ago ago

    So, apparently Omnialert sells enabling computer vision on existing cameras and backs up mistakes with human review. They at least claim no facial recognition, just guns. Reviews on reddit say the system (and similar ones) has many false positives, and failed to detect real guns like rifles.

    Garbage in, garbage out. The system should not have been deployed because of it's incompetence if nothing else.

  • syradar 4 hours ago ago

    Doesn’t sound like AI was the problem. Multiple human errors and process errors led to the armed response.

    • thereader12 2 hours ago ago

      It sounds like AI was the problem. Omnialert's ai gave it a 90% probability. The company also had multiple failures before it got out but the alert is designed to be sent quickly.

      • npteljes 15 minutes ago ago

        False positives are not necessary a problem. A false positive is just a wrong suspicion.

        In this particular case, the automated alert was just the beginning of the chain of events. The chain could have been stopped well after, and it looks like that they even tried to stop is, but were unsuccessful. That's the problem.

  • ChrisArchitect 7 hours ago ago

    Some discussion on the October 2025 incident at the time:

    https://news.ycombinator.com/item?id=45684934

  • AngryData 7 hours ago ago

    Unsure if you got a problem or not? Call the cops and now you know you have atleast 1 problem.

    Fuck this dystopian shit.

  • snvzz 5 hours ago ago

    Students shouldn't be bringing bags of Doritos to class either.

    • aftbit 4 hours ago ago

      Students who choose to bring unapproved snacks to class shouldn't face detention and search by the police.

      • Someone 3 hours ago ago

        > Students who choose to bring unapproved snacks to class shouldn't face detention

        If there are clear rules that forbid it, why not, for example on repeat offences?

        Having said that, it doesn’t seem that was the case here. FTA: “No child in Baltimore County should face this kind of danger while waiting outside his own school for a ride home from his mother,” said Freida Winfield, who was on her way to pick up her son Keith and the students when they were all ordered to their knees, handcuffed, and searched.

        • watwut 3 hours ago ago

          Because it is a.) idiotic b.) adults doing it are commiting much worst offence then the original one.

          Power abuse is worst then unaproved snack. Ability to make rules is not carte blanche for mistreatment, even if you feel good and powerful when mistreating people.

        • ungut an hour ago ago

          I can't believe what I'm reading. Are you seriously suggesting there is an amount of Doritos that warrants use of force on school children? I'm vexed. What amount of Doritos do you think would be appropriate before calling the misnamed American death squad to come and shoot some black kids?

          • mirddes 4 minutes ago ago

            It would entertain me to no end suggesting any child possessing multiple shipping containers filled with Doritos should be investigated. But that might be considered silly.

    • eesmith 3 hours ago ago

      Do you often make up excuses as a form of post-hoc justification?

      They didn't bring the Doritos and it wasn't during class, nor in a school building, nor during normal school hours.

      From the linked-to complaint at https://www.lawyerscommittee.org/wp-content/uploads/2026/11/...

      ```37. On Monday, October 20, 2025, Plaintiffs attended football practice after school. Practice ended at approximately 6:30 PM. After practice, their coach passed out individual bags of Doritos to the team, as he occasionally did after practices. Taki took several.

      38. Plaintiffs then went outside the school to wait for Keith’s mom to pick them up, just as they did many days after practice.

      39. Plaintiffs convened near the picnic tables near the back entrance of the school. As they waited for their ride, they were talking and eating the Doritos their coach had given to them.```

    • pvaldes 2 hours ago ago

      You never know when you will need an extra of energy to run from a mass shooter, because politicians refused to do anything about it, literally, in centuries.

      ----------

      A better way to solve the problem would be weapons that call the police and sleep when entering some protected areas, or enter in self-blocked mode if manipulated to remove this feature.

      Your fridge can call South Korea all day, but nobody can built such type of weapons? for Pete's sake.

      The police would love to have the series number of the weapon that is being fired inside a school before to reach the scene. Much faster response to help the family also. (Often mass shooters shoot their family first. If you reach them sooner, many of this victims could survive instead to bleed to death).

      The gun makers would salivate about a gun that go to sleep automatically when entering into a school area.

      First because the invaluable publicity of being the safest family brand in the market (only maker that really had the fucking galls to do something about it). The parents would love to have the option to block a gun from being used in a configurable range of areas of their choice, like for example their own bedrooms.

      Second because the maker would avoid events that make they look bad or be sued. Their insurance company would love that also.

      But most of all, because that would increase the profits. A self-blocked gun must be bought again, or fixed by the maker for a price. And families would gladly want to pay this price. Still much better than seeing your confused boy rotting in jail after a temporary depression, an adolescent broken heart or a drug/booze mistake.

      Old brands will never change until forced to change, and politicians will not move a finger, so make a "Tesla for guns" and take all the market for yourself by assault. If you can't do it for the children, at least do it for capitalism.

      • snvzz an hour ago ago

        >weapons that call police and self-block if they enter into some protected areas, or that self-block if manipulated to remove this feature.

        And, in a life and death situation, you'd be dead because, unlike the one that the bad guy is carrying, your gun can't connect to the network to authorize a discharge.

        Or when it does, it's too late; latency was too high.

        • pvaldes an hour ago ago

          The myth of the child hero is that, just a myth. Weapons fail 200 times a year, each year, to stop children to shoot other children or their teachers in USA. School shootings rarely end with a "childhero" or a "janithero" killing the shooter before they can kill tens of people. At this moment is too late yet. Self blocking doors, is a realistic solution. Your children being put in the situation to kill somebuddy or being killed just because they went to school is not. Is unfair to ask them to manage this psychologically. Many times they will end shooting collateral innocent victims by mistake and left scarred for life, or will be mistaken by the aggressor by police, and killed.

          So your fix will never happen, because exactly zero adolescents should be allowed to bring a hidden gun at their school and because is very unfair to ask them to fix the problem, so you don't need to lose a dollar. Doors that detect guns are placed yet for some reason. The idea of allowing only the "good children" to carry guns inside, just because, is 120% nuts. Adolescents are hormonal imbalanced by definition.

          Why to spend money on this doors if you have guns available in the teacher's room? Every mass shooter would go directly there first.

  • readthenotes1 7 hours ago ago

    Not sure what the race of the Students has to do with the story, or is the implication that AI would not have identified the bag of Doritos as a weapon on white Students?

    • PeeMcGee 7 hours ago ago

      > is the implication that AI would not have identified the bag of Doritos as a weapon on white Students?

      Yes

      • what 6 hours ago ago

        Unhinged.

        • arvid-lind 5 hours ago ago

          You're uninformed if that's your take.

          • floro 3 hours ago ago

            If AI is unbiased against black students then there is no issue. If AI is biased, then there's a good reason for it

        • sky2224 5 hours ago ago

          How? The topic of not making your AI model racist has been around for a long time.

      • autoexec 6 hours ago ago

        The AI must have been trained on all the police reports and news articles where police mistook common everyday items for deadly firearms when black people hold them such as cordless phones, cellphones, wallets, canes, hair brushes, gas pump nozzles, bicycle parts, sun glasses, pill bottles, ipods, Wii remotes, toy trucks, cologne bottles, sandwiches, bibles, candy bars, shower rods, flashlights, broomsticks, underwear, beer bottles, wrenches, and vape pens. AI really is just a reflection of ourselves.

        • metalman 2 hours ago ago

          Tests for AI racial desrimination are not mandated, yet, but are an inevitability. We could work up a joke about driving while bieng AI, ok too egdy and wry to be very funny, but still.

    • johnpython 6 hours ago ago

      You should question why your immediate reaction is to feel threatened that race was mentioned in the story

      • drdaeman 5 hours ago ago

        They most likely don’t feel that way. Check out a few headlines from weirder parallel universes where identity-highlighting culture somehow evolved different:

        “Ginger high school students sue after AI mistakes…” - with an inherited body variation.

        “Goth high school students sue after AI mistakes…” - with a sociocultural group.

        Most likely those would raise some eyebrows as to “ok, but why a highlight, what importance does it serve here?”

        There’s likely no brunette or hippie fragility involved there. So, it’s possible to imagine there’s possibly some different idea rather than a feeling of threat here. Don’t so easily dismiss it to a convenient fragility excuse here. It’s surely not that simple.

        And if anything, explaining the importance (if you know it and don’t just conform) would do a lot more bridge-building than accusing of fear. For all it’s worth, the article does a really poor job at this, mentioning a single Black student with a bag of chips, and not covering how that expanded onto their peers.

        • eesmith 3 hours ago ago

          > “Goth high school students sue after AI mistakes…” - with a sociocultural group.

          That's pretty much our universe.

          After the Columbine massacre in 1999 there were a lot of rumors and headlines about how goth culture was a major contributing factor to blame, with Marilyn Manson as a scapegoat.

          A Goth kid wearing a trenchcoat was eyed suspiciously as being the next killer in hiding.

          There was a widespread satanic panic at the time, including training some police to recognize "satanic ritual abuse". Had there been AI the time you know that some of the training data would have included these so-called "indicators", leading to false notifications resulting in a AI mistakes.

          • drdaeman 26 minutes ago ago

            Yes, both examples are not entirely random groups.

            Problem is in the article itself being sloppy. Unlike your comment, it highlights a detail in the title then totally fails to provide any more details that connect to it. There’s that one story and then a statement that some communities are affected more, and that’s about it. And this explanation gap doesn’t feel right: the most important detail of the story (for why else would it get into the title) ultimately gets a couple relevant sentences in the whole text - that just cannot be right for some.

            And then people ask “why Black” and other people accuse of racism… and so goes the misunderstanding.

            Just one of many possible explanations, of course.

      • VladVladikoff 6 hours ago ago

        Where do they mention feeling threatened?

      • mitchdoogle 6 hours ago ago

        And you should question why you feel the need to imply others were threatened when they didn't mention that word or give any indication that they were.

    • Spivak 5 hours ago ago

      "Students of demographic historically and currently disproportionately harmed by police violence sue school for using shoddy AI to call the cops on them saying they have a gun."

      Being wrong has worse consequences for black students than white students so you can maybe imagine why they might be extra mad at such a system being implemented even if the false positive rate was equal.

      > The false alert eventually led to armed local police swarming the teenagers on campus just steps away from their school’s entrance. They were ordered to their knees, handcuffed, and searched, despite doing nothing wrong.

      Good thing there's no chance of a cop "fearing for their life" and someone ending up dead in a situation like this.

    • kyleee 7 hours ago ago

      Correct, because no self respecting white student would be eating Doritos; they prefer Takis