LangWatch

LangWatch is a testing, evaluation, and observability platform for AI agents. It's built for engineering teams that have moved past simple chatbots and are running agents that take dozens of steps, call external tools, and can fail in ways that are hard to reproduce. The core problem it solves: agents have too many possible paths to test by hand, and bugs that slip through tend to surface in production at the worst time.

The platform centers on simulation-based testing, where synthetic users run multi-turn text or voice conversations against your agent before any code ships. You write scenarios in plain language, and the same tests run locally and in CI without extra setup. Adversarial red-teaming is built in, probing for jailbreaks, policy violations, and unsafe tool calls.

Evaluation goes beyond single-turn output scoring:

• LLM-as-a-judge reads the full trace, step by step, and returns a verdict with reasoning
• Pairwise comparison lets you pit two prompts, models, or versions head-to-head
• Online evaluation scores live production traffic in real time
• Multimodal scoring handles images and mixed media, not just text
• Evals run from a Jupyter notebook or the team UI, whichever fits the workflow

Observability is OpenTelemetry-native with full GenAI spec support, so traces from Cline, OpenHands, or any other agent framework plug in without a rewrite. Every token, tool call, and cost is tracked per span, and you can view runs as a waterfall, flame graph, topology, or sequence diagram.

A feature called Langy closes the loop between product and engineering: a PM writes a goal in plain English, Langy generates a full test plan and scenarios, runs them in parallel, scores the results against a rubric, and opens a pull request with a prompt revision when something regresses. The platform also includes a Prompt Registry so changes are versioned and reviewable.

For teams with compliance requirements, LangWatch is ISO 27001 certified and GDPR compliant, with RBAC, SSO, SCIM, audit logs, and custom data retention. It deploys as managed SaaS across EU, US, UK, and APAC regions, as a self-hosted Docker or Kubernetes install, or in a hybrid configuration where the data plane runs on your infrastructure. The source is open under Apache 2.