OpenAI still has a WebRTC Problem

(moq.dev)

67 points | by surprisetalk 3 hours ago ago

20 comments

  • Sean-Der 2 hours ago ago

    WebRTC doesn't give up on late audio. This isn't factually correct. WebRTC has NACKs[0]. The receiver (AI Inference server) decides when to either accept the gap in RTP packets or if it sends the NACK and waits.

    This article doesn't describe how Voice AI systems I work on operate. For speech-to-speech I am working at 20ms/200ms chunks and doing inference. So I don't want buffers

    [0] https://webrtcforthecurious.com/docs/06-media-communication/...

    • Sean-Der 2 hours ago ago

      What I find frustrating about these sales pitches that conflate the different parts of WebRTC. WebRTC is protocol, sdk/software and a API in the browser. So if one part has an issue it is important to explain it with detail.

      It's like attacking HTTP by saying 'HTTP is bad because this server doesn't have Y setting'.

      • msdrigg 6 minutes ago ago

        But the default WebRTC client is the one that browsers use. If Y setting isn't in that client or even if it is in the client and not enabled in the browser, it's basically as if WebRTC didn't have that at all. It would be like saying 'SSH is bad b'c OpenSSH doesn't have Y setting' b/c the vast majority of clients/servers are OpenSSH.

    • cma 2 hours ago ago

      You can have opus in-band FEC as well, though he mentions it in the article too.

  • tonelord 20 minutes ago ago

    I've had some experience with audio on the web, back in the day I built a weird collab music streaming system in the browser and presented at NIME. Back then AudioWorklets were new and I didn't understand WebRTC. The initial design used Raspberry Pis and IceCast. However, the setup was not only difficult to me but beyond the abilities of my target audience. I was more familiar with web dev and figured why can't something similar work for the browser, truly naive. After lots of learning through trial and error and understanding how these types of streaming worked, I ended up with something in hindsight worked pretty simply and was beautifully naive.

    It took audio input, fed it into a wasm opus encoder, output audio in "segments" of audio, and uploaded via typical HTTP. No WebSockets or WebRTC or QUIC (I don't even think I used HTTP2). The only reason it wasn't just a full encoded recording was because I wanted to make it continuous and streamed. I basically learned this via HLS, but also didn't understand HLS nor was it easy to encode and stream data (in my mental model) from the browser to the server.

    On the server and streaming to the client, it just collected the segments and told clients when they were ready, via polling, similar to HLS. The client would download, wasm decode, and add the audio to a buffer. The thing is that because of the nature of the project it didn't really have much to do with one continuous stream, but it did take sequences of the segments from multiple user's uploads and combine them for the different audio channels.

    Most of the argument I think for what I'm seeing for voice AI is the code complexity, server complexity, interruptions, latency, drop outs, that kind of stuff, and I guess that's all good for companies with huge numbers of users and wanting to follow the work done at frontier labs. Also everyone wants the technology they sell to be the ones companies use.

    Just weighing in that there could be cases where you record some audio, encode it, upload it, transcribe it, feed to an LLM, TTS the response and download it. You can stream it in segments or not to reduce latency. You can interrupt it with UI/DSP/NN and detect the end of a message similarly. Maybe for most smaller companies that would be enough? It feels like reading these different takes is that they create problems while solving problems, and maybe make the adoption of voice AI harder (and therefore slower) than it could be. I'm not any less naive than I was before though, but often its people like me tasked to implement this stuff.

  • FLeXMurphy 2 hours ago ago

    Summary: Long-winded advertisement for MoQ. Author mentions that WebRTC has some limitations and Google needs to address them. Until then, anyone (including OpenAI) trying to use WebRTC for human<->bot communication may run into issues.

  • cklscikng 2 hours ago ago

    Quite interesting. I found the part about tokens with different priorities super cool.

  • InsidiousWombat 2 hours ago ago

    Native apps should probably just run local voice recognition AI

    • Sean-Der 2 hours ago ago

      Size is the tough part. These models require 256GB+ of vram, but other then that I think it's possible!

      • 14u2c an hour ago ago

        Transcription models certainly don't require that ram. Parakeet even runs the CPU while generally outperforming Whisper.

    • cush 2 hours ago ago

      This is for communicating with agents and inference, not just dictation.

  • tschellenbach an hour ago ago

    So their argument is that you can send voice generated by the AI faster than realtime and buffer it. This is not entirely true since you can interrupt the AI agent. So you would need some additional client side logic to handle that. Meh not sure if this tradeoff of fixing potential network issues, for more client side logic is worth it.

    Their other argument is that you can retry the sending of your voice to the AI. Which WebRTC already does. (NACK, FEC, Opus RED)

  • fidotron 2 hours ago ago

    The moq people, and they are loud and obnoxious, have this incredible ability to ignore the practical experience that led to the defaults of various systems being what they are while believing the solution is to expose every possible to dial to them so they can eventually rediscover the same results, which presumably they will smugly and obnoxiously broadcast as new discoveries to celebrate their brilliance.

    It's like that famous saying about a certain group of people: you can always rely on them to do the right thing, after they've tried every other possibility. They could at least exhibit some level of self awareness.

  • koolala 2 hours ago ago

    This seems like an ad for their voice SaaS?

  • paulez 2 hours ago ago

    Just ask "what is le capital of la France" no more confusion then.

  • bob1029 2 hours ago ago

    Unhinged indeed.

  • doctorpangloss 2 hours ago ago

    If I were "Sean-Der," I would not reply. The "being wrong on the Internet causes smart people to write smarter stuff that's interesting to read" discourse is over. You have to own how much of a waste of time it is.

    • kidq330 2 hours ago ago

      Sean is not the only developer who has used WebRTC, and conferencing is not the only thing MoQ can be used for. I'd really like to see more devs weighing in, especially those with WebRTC experience

    • Sean-Der 2 hours ago ago

      I don't understand, mind rephrasing?

      I respond because incorrect info will cause developers to make bad decisions/build the wrong thing.

      • doctorpangloss 25 minutes ago ago

        as someone who works at openai you already know that developers are not making any decisions anymore haha