{"id":47122431,"title":"Show HN: Attest – Test AI agents with 8-layer graduated assertions","points":1,"user":"tommathews","time":1771855259,"time_ago":"6 hours ago","type":"link","content":"

I built Attest because every team I've seen building AI agents ends up writing the same ad-hoc pytest scaffolding — checking if the right tools were called, if cost stayed under budget, if the output made semantic sense. It works until the agent gets complex, then it collapses.

60–70% of what makes an agent correct is fully deterministic: tool call schemas, execution order, cost budgets, content format. Routing all of this through an LLM judge is expensive, slow, and unnecessarily non-deterministic. Attest exhausts deterministic checks first and only escalates when necessary.

The 8 layers: schema validation → cost/perf constraints → trace structure (tool ordering, loop detection) → content validation → semantic similarity via local ONNX embeddings (no API key) → LLM-as-judge → simulation with fault injection → multi-agent trace tree evaluation.

Example:

  from attest import agent, expect\n  from attest.trace import TraceBuilder\n\n  @agent("support-agent")\n  def support_agent(builder: TraceBuilder, user_message: str):\n      builder.add_tool_call(name="lookup_user", args={"query": user_message}, result={...})\n      builder.add_tool_call(name="reset_password", args={"user_id": "U-123"}, result={...})\n      builder.set_metadata(total_tokens=150, cost_usd=0.005, latency_ms=1200)\n      return {"message": "Your temporary password is abc123."}\n\n  def test_support_agent(attest):\n      result = support_agent(user_message="Reset my password")\n      chain = (\n          expect(result)\n          .cost_under(0.05)\n          .tools_called_in_order(["lookup_user", "reset_password"])\n          .output_contains("temporary password")\n          .output_similar_to("password has been reset", threshold=0.8)\n      )\n      attest.evaluate(chain)\n

\nThe .output_similar_to() call runs locally via ONNX Runtime — no embeddings API key required. Layers 1–5 are free or near-free. The LLM judge is only invoked for genuinely subjective quality assessment.

Architecture: single Go binary engine (1.7ms cold start, <2ms for 100-step trace eval) with thin Python and TypeScript SDKs. All evaluation logic lives in the engine — both SDKs produce identical assertion results. 11 adapters covering OpenAI, Anthropic, Gemini, Ollama, LangChain, Google ADK, LlamaIndex, CrewAI, and OpenTelemetry.

v0.4.0 adds continuous evaluation with σ-based drift detection, a plugin system, result history, and CLI scaffolding. The engine and Python SDK are stable across four releases. The TypeScript SDK is newer — API is stable, hasn't been battle-tested at scale yet.

The simulation runtime is the part I'm most curious about feedback on. You can define persona-driven simulated users (friendly, confused, adversarial), inject faults (latency, errors, rate limits), and run your agent against all of them in a single test suite. Is this useful in practice for CI, or is it a solution looking for a problem?

Apache 2.0 licensed. No platform to self-host, no BSL, no infrastructure requirements.

GitHub: https://github.com/attest-framework/attest\nExamples: https://github.com/attest-framework/attest-examples\nWebsite: https://attest-framework.github.io/attest-website/\nInstall: pip install attest-ai / npm install @attest-ai/core","url":"item?id=47122431","comments":[],"comments_count":0}