An agent can return a successful status and still fail its user. It might claim a task is finished after ignoring a tool error, or answer a cancellation request with contradictory instructions. LangSmith's AI Search public beta, announced October 13, lets investigators describe such behavior rather than first guess an error code, tag or exact phrase.
That creates an exploratory step before a durable evaluator exists. It also puts another model between the investigator and the execution record. The feature's usefulness depends on the evidence that model actually receives, not just the fluency of its search interface.
Search now asks what happened, not only what was mentioned
LangChain's announcement describes queries for frustrated users, disclosure of implementation details and agents that did not finish assigned work. Search operates across individual runs, traces and threads: a trace records a request and nested operations, while a thread links traces in a conversation.
Existing structured filters first narrow candidates. Status, latency, tags, metadata, cost and feedback still belong in those filters; a natural-language description cannot infer run status, tags, metadata or trace structure. Full-text search remains useful for a distinctive identifier, phrase or error. Behavioral search adds an interpretation of the messages and actions inside the selected set.
A decision model reads candidate trajectories in parallel and streams matches as they appear. This is not merely an embedding search for a topic. Successful and failed billing conversations can have almost identical vocabulary and nearby vectors. Deciding whether a completion claim matches recorded actions requires a judgment about sequence and meaning.
Matches carry confidence scores and open in a trajectory view with available influential messages highlighted. Confidence is not ground truth. Sarcasm, repeated questions, quiet abandonment or polite rejection can produce disagreement even between human reviewers. Nor can every match be highlighted: repeated text and messages absent from the view can prevent evidence localization. A score without a complete rationale is not an auditable finding by itself.
Normalization removes duplication, not every coverage gap
Agent telemetry contains model calls, user messages, retrieved documents, shell commands, tool results, retries and handoffs. Sending a full history to each model call repeats older material throughout a naïve trace representation.
LangChain examined 150 randomly selected Open SWE coding-agent conversations. It reports a median of roughly 72,000 tokens, with only around 2,000 from user messages and final replies, and about 85% from tool output. Several conversations exceeded one million tokens. This is a company sample, not an independent industry benchmark or an AI Search accuracy evaluation.
The SmithDB data layer normalizes supported formats as traces arrive, presenting a session once through human messages, agent replies, tool calls and results. Supported sources include LangChain, LangGraph, the OpenAI and Anthropic SDKs, Vercel AI SDK and OpenTelemetry. A common trajectory makes investigation possible across formats without repeatedly presenting the same history.
The 8,000-token ordering determines what the judge misses
Each trajectory gets an 8,000-token context budget. User messages and direct agent replies have highest priority. Intermediate model messages, tool calls, tool results and system messages fill the remaining space. The next message is trimmed if it does not fit; later material falls outside the model's view after the budget is full.
That favors conversational evidence such as frustration, refusal, confusion and unsupported promises. It creates a predictable problem when the decisive contradiction lies in a tool result. An authorization failure can establish that no action completed, but higher-priority dialogue may consume the budget before the judge sees it. A confident match—or no match—therefore cannot establish complete execution coverage.
The representation also excludes subagent, middleware and compaction-message content. A parent agent can report success while a subagent records the actual failure outside the searchable material. Increasing the main conversation's budget would not, by itself, add those excluded categories.
LangChain says it is exploring query-dependent selection and signals for a second pass. Prioritizing tool evidence for an ignored-error query would address a different need from prioritizing dialogue for sentiment. Those are development directions, not capabilities established by the present fixed ordering.
Regional availability is not regional inference residency
The beta is available on LangSmith Cloud in Google Cloud's US, European and Asia-Pacific regions. An organization administrator enables it for all members, and no separate model-provider key is required.
Enabling AI Search permits trace data to be transferred to and processed in the United States through LangChain-managed inference and subprocessors. This applies to customers using the European region as well; selecting a regional LangSmith deployment must not be presented as a guarantee that AI Search inference remains there.
LangChain explicitly says not to use AI Search for protected health information, and the feature is not covered by any Business Associate Agreement signed with LangChain. General review for leaked personally identifiable information is not interchangeable with permission to process regulated health data. The search itself can send the sensitive trace through the judge.
Retention, access, residency and subprocessor requirements therefore need assessment before administrator enablement. Rich observability can capture prompts, customer records, retrieved documents and tool outputs; minimizing sensitive fields before collection reduces exposure rather than relying on a search feature to remove it afterward.
Exploration is not yet a repeatable monitoring control
The filter documentation describes important beta restrictions. AI Search descriptions cannot be saved as views or preserved in copied page links. They do not appear in recent-filter history, and the statistics panel is unavailable while a query is active. Rate limits apply per workspace, but the referenced documentation does not publish their threshold.
Teams can record exact wording elsewhere, but that adds manual work and does not guarantee identical classifications. Small wording changes can alter a subjective category. Saved searches, stable result sets and documented limits would be meaningful improvements for incident response, not details already solved by a natural-language box.
Public material supplies no precision, recall, latency distribution or independent comparison against embeddings and predefined evaluators. Plausible matches may coexist with rare, consequential misses. Evaluation should include known incidents, difficult negatives and different frameworks, checking both false positives and false negatives.
The durable value lies in what follows a match
A support team could search for users repeating a request after the agent claimed resolution, then discover confusion between account closure and subscription cancellation. Reviewed examples can become a dataset and a tested rubric, with online evaluators measuring future traffic consistently. Search discovers the category; monitoring applies it after its definition stabilizes.
Competitors already combine tracing and model judgment. Datadog's trace-level evaluations put trace spans into a judge prompt to assess goal completion, tool order or conversation quality. Arize Phoenix's tracing workflow connects annotations, evaluations, datasets and experiments. LangSmith's distinction is query-time exploration before a team has formalized the failure, not the invention of model-based evaluation.
Making fragmented evidence retrievable also matters outside observability; the source's related knowledge-base discussion concerns searchable technical records. It is contextual reading, not a quality benchmark for AI Search.
Grouping recurring matches and adaptive evidence selection could make discovery more useful. LangChain describes those ambitions, while competition may shift toward auditable evidence, reproducibility and governance. For now, high-impact matches need confirmation against the underlying record, and sensitive-data searches cannot replace deterministic scanners or formal controls. The launch changes when teams can ask behavioral questions; it has not established that every answer is complete, reproducible or safe to enforce automatically.