Run a tiny shitty llm, one that is super fast. Or better yet try out a decision model now days.
Feed it the previous conversation turn and all the STT (ASR output) you have so far and just ask it for a probability that the user is done speaking.
Use that probability to shorten your VAD time.
It isn't as good as a true speech to speech model but this will cut another few hundred ms. Oh as a bonus it also lets you wait longer when the user isn't done talking yet.
Ugh this is the 3rd or so time I've posted this -
Run a tiny shitty llm, one that is super fast. Or better yet try out a decision model now days.
Feed it the previous conversation turn and all the STT (ASR output) you have so far and just ask it for a probability that the user is done speaking.
Use that probability to shorten your VAD time.
It isn't as good as a true speech to speech model but this will cut another few hundred ms. Oh as a bonus it also lets you wait longer when the user isn't done talking yet.