Sentinel: A Continuous Evaluation Service for Production Agentic Pipelines

Consider a support chatbot deployed at a financial services company. Traffic is healthy, error rates are flat, p99 latency looks fine. Three weeks in, a customer escalates. The bot has been quoting incorrect early redemption penalties – not consistently, not obviously, but often enough to matter.

Nobody had instrumented the outputs.

This is the structural problem with applying conventional observability to LLM systems. A traditional service either returns the right output or it doesn't. An LLM service operates in a probabilistic regime where correctness, tone, factual accuracy, and policy compliance are continuous, context-dependent properties that degrade gradually, inconsistently, and silently.

No error rate captures a sycophantic assistant that agrees with the user regardless of whether the user is right. No latency spike signals a response that contradicts the retrieved context it was given.

The question isn't whether to monitor. It's whether to find out before a user calls to complain.

The evaluation methodology literature has matured. LLM-as-a-Judge has been validated across a range of tasks. RAGAS introduced principled faithfulness metrics for RAG pipelines. DeepEval, LLM Guard, and Guardrails AI each contribute tools for specific parts of the problem.

But every one of these is a library. And that distinction matters more than it appears.

A team that adopts an evaluation library also inherits a second system: model hosting, inference hardware, scanner configuration, and the ongoing operational burden of keeping the evaluation infrastructure healthy alongside the application it monitors.

For most product teams, this is work that doesn't appear on any roadmap until something goes wrong. Engineers responsible for shipping a customer-facing LLM application shouldn't also need to become experts in operating an evaluation pipeline.

Sentinel exposes evaluation as a service – a call rather than a deployment. The consuming application fires an HTTP request. What happens after that is Sentinel’s problem.

The evaluation problems worth solving don't all look the same. Hallucination, factual inconsistency, policy compliance, and multi-dimensional quality scoring are properties that resist algorithmic approximation – they require a capable language model to reason about. These are the cases where LLM-as-a-Judge earns its place, and they represent the majority of what Sentinel does.

LLM-as-a-Judge is the primary evaluation mechanism across Sentinel’s scanner suite. The cost reduction and latency improvements described in the next section were achieved specifically here – by making this path as efficient as possible without sacrificing signal quality.

Not every evaluation problem belongs on the LLM path, though. Simpler checks – format validation, token budget enforcement, structural consistency – don't benefit from model reasoning and would only add unnecessary latency and cost if routed there. For these, Sentinel uses lightweight, purpose-built models that return results fast and cheaply.

One domain deserves a specific note. Refusal detection. Over-refusal is a real production quality problem that typically goes untracked because the model technically "behaved safely." Flagging incorrect refusals as a first-class failure mode is a deliberate choice.

LLM-as-a-Judge evaluation is expensive. A naive implementation of the full evaluation pipeline would require approximately 12 separate LLM API calls per interaction – one per metric. At current pricing, this cost compounds quickly against any non-trivial request volume.

Six composite scanners – covering safety, hallucination, response quality, tone and style, RAG quality, and general LLM metrics – each consolidate multiple evaluation dimensions into a single structured prompt.

A single call to the safety scanner returns simultaneous assessments of hate speech, harassment, violence, and self-harm. Each with its own pass/fail flag, normalised score on a 0–1 scale, and a natural language rationale. One call. Four metrics.

Bundling is not just an optimisation. A model evaluating related dimensions with full mutual context may produce more internally consistent scores than separate calls where each assessment lacks awareness of the others.

Those 12 calls reduce to approximately 2–3. The cost reduction is approximately 5×. The bundling is invisible to downstream consumers – results are parsed and stored per metric.

LLM calls, even bundled ones, have latency that's incompatible with blocking the response path in latency-sensitive applications. For cases where evaluation must gate a response rather than observe it asynchronously, a different set of scanners is required.

Sentinel’s "lightening" scanners are fine-tuned local models designed for exactly this regime. A prompt injection classifier. Embedding models for semantic relevance scoring. Natural language inference models for factual consistency checking.

Zero LLM API calls. Zero remote calls. The full lightweight path runs in under 500 milliseconds.

The relevance detection approach here is worth calling out specifically. Rather than asking a language model whether an output is relevant to a prompt, the system derives irrelevance from the geometric distance between prompt and output embeddings. Fast, cheap, and structurally different from asking a model to reason about its own outputs – which introduces a different set of biases.

The intended deployment pattern is layered: the lightweight local-model path as a synchronous pre-filter before the LLM call, combined with the full asynchronous evaluation pipeline after the response is returned. The two paths are separately configurable.

The interaction-level framing breaks down precisely where modern LLM deployments are growing fastest.

An agent that searches the web, calls APIs, delegates to sub-agents, and synthesises a final response doesn't have a single prompt/response pair to evaluate. It has a trace. Multiple LLM calls, each with their own potential failure modes. Tool invocations that may or may not have been necessary. Reasoning chains that may be internally coherent but factually wrong at a critical step.

Sentinel accepts OpenTelemetry traces as a first-class input. Each span representing an LLM invocation, a retrieval step, or a tool call becomes an independent evaluation target. The scanner configuration is attached to the agent definition rather than the individual request – every trace it emits is evaluated against that profile automatically.

What a coding agent should be evaluated on is fundamentally different from what a customer support agent or a research synthesiser should be evaluated on.

Sentinel lets teams configure evaluation profiles that reflect the actual risk surface of their specific deployment – not a generic quality rubric applied uniformly across all agents. The bundling logic keeps cost manageable: even a complex agentic trace with many LLM spans doesn’t require a proportional increase in evaluation API calls.

A single scan answers a point-in-time question. Longitudinal tracking answers the more important one: is this system getting worse?

A model version update shifts tone slightly. A prompt change introduces new ambiguities. A retrieval index refresh changes what the system has access to. None of these changes throw an error. All of them can meaningfully affect quality – and all of them are invisible without aggregated evaluation data over time.

A rising hallucination rate after a deployment. Drift in sentiment across a particular topic cluster. A creeping increase in incorrect refusals after a safety prompt was tightened. Individual scans produce data points. Aggregated metric trends over time make patterns legible.

This matters most in agentic deployments, where the number of variables in play is high. A change to one tool's output format can ripple through an agent's reasoning in ways that only manifest statistically, across many traces. Spot-checking individual interactions won't surface this.

The most expensive place to discover a quality regression is production. The second most expensive is right before a deployment, when something has already changed and the team needs to understand what.

Sentinel's evaluation pipeline can be run as a pre-promotion quality gate. A representative sample of interactions – drawn from production traffic or from a curated test set – is evaluated against the agent's registered scanner profile. Scores are compared against a defined baseline. If any metric falls outside acceptable bounds, promotion is blocked.

Quality becomes a property of a deployment the same way test coverage is – a condition that must be satisfied before code promotes, not a metric consulted after an incident.

Teams that have already defined scanner profiles for their agents get this capability without additional configuration. The same profile that drives production monitoring also drives the pre-production gate.

Instead of discovering that a prompt change degraded faithfulness three days after it reached production, you discover it before the deployment completes. The eval run becomes part of the deployment artefact – auditable evidence that quality was measured, not assumed.

The gap in LLM observability has never been the methodology. It has been the operationalisation.

Most teams don't run comprehensive evaluation in production not because they don't want to – but because standing up and maintaining an evaluation pipeline is expensive enough to keep getting deferred. Usually until something breaks.

There is a class of production LLM systems where the failure surface is large enough, the causal chains long enough, and the regressions subtle enough that continuous evaluation is no longer optional. Agentic systems are that class.

For those systems, evaluation should be something you call – not something you operate. The hard parts – scanner selection, profile configuration, cost management, longitudinal aggregation, deployment gating – should be solved once, at the service level, and reused across every agent that needs them.

Thorough, continuous evaluation should be the easy option. For agentic systems, it may also be the only reliable one.