We got AI to Wombo Combo – collaborative agents in Melee

(frisson-labs.com)

5 points | by coatol5 9 hours ago ago

6 comments

  • goodmythical 8 hours ago ago

    This could be amazing for accessibility. Say that the only available input is voice.

    If a player could say "down-b, dash, okay dodge and shoot while I think, see if we can get in some shields" to where it really matters what and when the user is saying things as opposed to "okay win this match please", the player and the opponent could both have a sense of a fair fight.

    Like, compared to having a STT model that adds so much latency to where there's no shot, this could be actually useful.

    My only question is, it seems like Reyn1/2 are recieving the same voice inputs? They're each doing the "same" thing for every command, just right/left, up/down, taunt/dash dance. Like, it's not as if one is shooting while the other is edge-hopping or something.

    There are two instances where the commands differ (up-b/drill him, and I forget the other one), but I wonder if that's the model choosing different commands based on the single input since all of the others are seemingly mirrored.

    • coatol5 6 hours ago ago

      If you check out the video at the top of the article: https://www.frisson-labs.com/reyn

      We first show both agents receiving the "Go to the top platform" command. But the second command splits "go left" and "go right" across both agents. In short, each agent acts independently, and receives commands independently. You could have completely different voice inputs for each agent.

      We could also unify it into one command, but that wasn't the goal of this project.

      Accessibility is a great idea. I had half a mind to demonstrate a "pokemon battle" using voice commands.

      • goodmythical 4 hours ago ago

        I saw the commands, it just seemed that most of them were the same for each, like there's no 1 go left 2 shoot laser or 1 ledge jump 2 shield, it's always 1left2right/1up2down so it kind of seemed like they were both receiving the same command.

        Pokemon with voice commands is an excellent idea. I can see two streamers popping in in place of trainers shouting "I choose you, pikachu! Use Quick Attack!", or even outside the streaming aspect it adds a bit of nice immersion to be able to speak to games that don't natively support it..."where is my base"/"tell steve where this ravine I'm in is" chatting coords in minecraft...all sorts of stuff.

        It could also be combined with any of the sub-vocal "lip reader" setups that allow you to send text commands without actually speaking, and then AR glasses, and you could effectively game while walking around with your hands free to do whatever.

  • Computer0 9 hours ago ago

    Why not Godzilla destroy all monsters melee?

    • coatol5 8 hours ago ago

      Holy crap I haven't heard this game referenced since I was in middle school. Would love to do it. I love that game.

  • coatol5 9 hours ago ago

    We did an exploration into collaborative agents in Super Smash Bros Melee.

    We wanted the ability to talk to and coordinate with it in real time.

    We used a combination of imitation learning (we labeled thousands of hours of data using english language predicates) and RL to create an action policy for doubles in Melee.

    Full write up (and model weights) on our website: https://www.frisson-labs.com/reyn

    Built upon our previous work creating a competitive Melee agent: https://www.frisson-labs.com/faynt