Author here. I’ve been using a proactive personal agent (Dr.Doodle, it's a Muse) for a few weeks, and the failures were mostly bad decisions about when to let me know about things, e.g. pinging me five times about a package I’d already picked up. That made me think the thing to evaluate isn’t the output but the decision to act (or not), which mostly happens where nobody looks. Curious how others building background or proactive agents are evaluating this.
Author here. I’ve been using a proactive personal agent (Dr.Doodle, it's a Muse) for a few weeks, and the failures were mostly bad decisions about when to let me know about things, e.g. pinging me five times about a package I’d already picked up. That made me think the thing to evaluate isn’t the output but the decision to act (or not), which mostly happens where nobody looks. Curious how others building background or proactive agents are evaluating this.