## Definition **LLM observability** is the practice of instrumenting an LLM application so every step — prompts, retrieved context, tool calls, and the non-deterministic model outputs — is traced together with latency, token cost, and quality signals. It turns a black box of chained, stochastic calls into something you can debug, cost, and improve. ## What you trace - The full **trace** of a request as nested spans: prompt, retrieved chunks, tool / [[Function Calling|function]] calls, model responses, and each step of an agent's loop. - **Metrics** — latency, token usage and cost, error and retry rates. - **Quality signals** — user feedback, [[LLM as a Judge|LLM-judge]] scores, grounding / [[Hallucination]] flags. ## Why it differs from ordinary monitoring - Outputs are **non-deterministic** — you must capture the actual inputs and outputs, not just HTTP status codes, or you cannot reproduce a bad answer. - Chains and [[Multi-Agent System|agents]] are **multi-step** — a single user request fans out into many model and tool calls that only make sense as a nested trace. - Cost is **per-token** and easy to blow up — it has to be tracked explicitly per step. OpenTelemetry is increasingly the shared standard for emitting these spans (used by tools such as Langfuse and LangSmith), so LLM traces sit alongside the rest of a system's telemetry. ## Related - [[LLM as a Judge]] - [[Eval Pipeline Bootstrapping]] - [[AI Agent]] - [[Retrieval-Augmented Generation]] ## Sources - [[AI Engineering - Chip Huyen]]