Yes, those sure are big red flags! If you're not seeing demos, at a minimim, within HOURS then that's a warning sign. Eval metrics should be the first step, before you build anything.For example, when I rebuilt my AI's memory architecture this weekend the very first thing we did was get good eval snapshots.
Here's how I built my own custom persistent-memory AI research assistant. Note the need for multiple orthogonal governance layers! If you don't have those then your system will be naturally unstable and apt to collapse into confabulation or dishonesty.
Yes, those sure are big red flags! If you're not seeing demos, at a minimim, within HOURS then that's a warning sign. Eval metrics should be the first step, before you build anything.For example, when I rebuilt my AI's memory architecture this weekend the very first thing we did was get good eval snapshots.
Here's how I built my own custom persistent-memory AI research assistant. Note the need for multiple orthogonal governance layers! If you don't have those then your system will be naturally unstable and apt to collapse into confabulation or dishonesty.
Here's what worked for us:
https://energyscholar.github.io/persistent-ai-collaboration/