Why Traditional Observability Fails for LLM Apps

Traditional monitoring treats applications as deterministic systems. You track request rates, error percentages, latency percentiles, and resource utilization. When these metrics degrade, you investigate and fix the underlying issue. This model breaks down for LLM applications because the most critical failures are semantic, not syntactic.

Consider a customer support chatbot that responds to product questions. Traditional monitoring shows all requests complete successfully with acceptable latency. What it misses: the model started recommending discontinued products after a knowledge base update, response quality degraded for technical questions, and average conversation length doubled because users need multiple exchanges to get useful answers. These issues manifest as business problems—increased support tickets, lower customer satisfaction—long before they appear in standard metrics.

LLM observability requires tracking fundamentally different signals: output quality scores, semantic similarity to expected responses, cost per interaction type, token usage patterns, prompt injection attempts, and the correlation between model outputs and downstream user behavior. These metrics are application-specific—what matters for a code generation tool differs completely from a content moderation system.

The Observability Requirements for Production LLMs

Production LLM applications need visibility into multiple layers: infrastructure (API health, rate limits), application (request flow, retry logic), model behavior (input/output quality, consistency), and business impact (cost, user satisfaction). Each layer requires different instrumentation approaches and generates different actionable insights.

Infrastructure monitoring is straightforward—track API latency, error rates, and quota consumption. Application monitoring requires tracing the full request path through your system, including prompt construction, model calls, output parsing, and result delivery. Model behavior monitoring is where specialized tools become necessary because you need to evaluate semantic properties of inputs and outputs that require domain knowledge. Business impact monitoring connects LLM behavior to metrics like conversion rates, session length, and revenue per interaction.

LangSmith: End-to-End LLM Application Debugging

LangSmith provides deep integration with LangChain applications but works with any LLM application through its SDK. The core value is comprehensive tracing of multi-step LLM interactions, including chains, agents, and tool calls. Every LLM invocation captures the prompt, completion, token counts, latency, and cost, organized into hierarchical traces that show how complex operations break down into individual model calls.

Core Capabilities

The tracing interface shows the complete execution tree for each request. For agent-based applications, you see each reasoning step, tool call, and result in sequence. This visibility is critical when debugging why an agent chose a specific action or why a chain produced unexpected output—you can inspect the exact prompt the model received at each step, not just your prompt template with variables unfilled.

// LangSmith instrumentation example
import { Client } from "langsmith";

const client = new Client({
  apiKey: process.env.LANGSMITH_API_KEY,
  projectName: "production-chatbot"
});

// Automatic tracing for LangChain components
import { ChatOpenAI } from "langchain/chat_models/openai";

const model = new ChatOpenAI({
  temperature: 0.7,
  callbacks: [client.getCallbacks()] // Auto-traces all calls
});

// Manual tracing for custom code
await client.runTree(
  {
    name: "generate_product_recommendation",
    inputs: { userId: user.id, context: userContext },
    run_type: "chain"
  },
  async () => {
    // Your LLM logic here
    const result = await model.call(messages);
    return { output: result };
  }
);

The dataset management features let you build test suites for LLM applications. Capture real production inputs that caused problems, create expected outputs, and run regression tests to ensure prompt changes don't break existing functionality. This testing approach is more practical than synthetic test cases because it uses actual user inputs that exposed real failure modes.

When LangSmith Excels

LangSmith is most valuable for teams building complex LLM applications with multiple chained operations or agent-based systems. If your application makes a single LLM call per request, simpler tools suffice. But when you're orchestrating five model calls to answer one user query—extracting intent, retrieving context, generating response, checking quality, formatting output—the hierarchical tracing becomes essential for understanding where issues originate.

The playground environment lets you replay production traces with modified prompts or parameters. You can take a failed interaction, adjust the system prompt, and re-run to see if the change fixes the issue. This rapid iteration eliminates the deploy-test-debug cycle when tuning prompts for edge cases discovered in production.

Limitations and Considerations

Cost tracking is basic—you see token counts and estimated costs, but there's no built-in budgeting, alerting on cost thresholds, or cost attribution to business entities like users or organizations. For applications where cost management is critical, you need supplementary tools. The platform focuses on debugging and improving LLM behavior, not managing the economics of LLM operations.

Data privacy requires careful configuration. By default, LangSmith captures full prompts and completions, which may include sensitive user data. You can configure filtering to redact PII, but this requires upfront effort and ongoing maintenance as your application evolves. Teams in regulated industries need to carefully evaluate whether sending production data to LangSmith's cloud complies with their security requirements.

Helicone: Cost-Focused Observability

Helicone positions itself as observability infrastructure specifically for managing LLM costs and usage. It operates as a proxy between your application and LLM APIs, capturing every request and response with zero code changes beyond updating the API endpoint. This architecture provides complete visibility without SDK integration or code instrumentation.

Proxy-Based Architecture

The proxy approach has specific advantages: it captures 100% of LLM traffic regardless of which library or framework your application uses, it works with any language, and it requires minimal application changes. You simply point your OpenAI client at Helicone's proxy URL instead of the OpenAI API directly.

// Before: Direct OpenAI call
const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY
});

// After: Route through Helicone
const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  baseURL: "https://oai.hconeai.com/v1",
  defaultHeaders: {
    "Helicone-Auth": `Bearer ${process.env.HELICONE_API_KEY}`,
    // Tag requests for cost attribution
    "Helicone-Property-User": userId,
    "Helicone-Property-Environment": "production"
  }
});

The cost tracking dashboard breaks down spending by user, feature, model, and time period. You can set budget alerts that trigger when specific users, API keys, or features exceed spending thresholds. This granularity helps identify cost optimization opportunities—you might discover 80% of your API costs come from 20% of users, or that a specific feature uses expensive models when cheaper alternatives would suffice.

Request Caching

Helicone's semantic caching reduces costs by detecting when your application sends functionally identical requests and returning cached responses instead of calling the LLM API. This works at the semantic level—"What's the weather?" and "How's the weather?" are different strings but semantically similar, so Helicone can return the same cached result.

The caching configuration lets you specify TTLs per request type and control cache key generation. For deterministic queries (like "What's the capital of France?"), long TTLs make sense. For time-sensitive queries, shorter TTLs or cache bypass rules prevent stale responses. The cache hit rate dashboard shows how much you're saving and helps tune cache policies.

Limitations and Trade-offs

The proxy architecture introduces a single point of failure and adds network latency. If Helicone's proxy is down or slow, all your LLM requests are affected. They provide high availability guarantees, but you're adding a dependency to your critical path. The latency overhead is typically 10-30ms, which is negligible compared to LLM inference time but becomes noticeable in latency-sensitive applications.

The focus on cost tracking means observability features beyond usage monitoring are limited. You get request logs and basic metrics, but not the deep tracing, quality evaluation, or debugging tools that platforms like LangSmith provide. Helicone excels at answering "How much are we spending and on what?" but doesn't help much with "Why is the model producing poor outputs?"

Weights & Biases LLM Tools

W&B extends their ML experiment tracking platform with LLM-specific features. The core strength is comprehensive experiment tracking—you can log every prompt variant, model configuration, and output for systematic A/B testing and performance comparison. This structured experimentation approach helps teams move from ad-hoc prompt tweaking to data-driven optimization.

Prompt Engineering Workflow

The platform treats prompt development as an iterative experiment process. You create prompt variants, test them against a dataset of inputs, evaluate outputs using automated metrics or human review, and track which variants perform best. This workflow prevents the common pattern where teams continuously modify prompts based on the most recent failure without understanding overall impact.

import wandb

# Initialize experiment tracking
wandb.init(project="customer-support-bot", name="prompt-optimization-v3")

# Log prompt variants
prompts = {
  "baseline": "You are a helpful customer support agent...",
  "concise": "You are a customer support agent. Be brief and direct...",
  "empathetic": "You are an empathetic customer support agent..."
}

# Test each variant
for variant_name, system_prompt in prompts.items():
  results = []

  for test_case in test_dataset:
    response = llm.generate(
      system=system_prompt,
      user=test_case.input
    )

    # Evaluate response
    score = evaluate_response(response, test_case.expected_output)

    results.append({
      "input": test_case.input,
      "output": response,
      "score": score,
      "cost": response.cost
    })

  # Log variant results
  wandb.log({
    f"{variant_name}/avg_score": average(r["score"] for r in results),
    f"{variant_name}/avg_cost": average(r["cost"] for r in results),
    f"{variant_name}/results": wandb.Table(dataframe=results)
  })

wandb.finish()

The comparison views show how different prompts perform across your test cases. You can filter to specific input types to understand which prompt works best for which scenarios. This granular analysis often reveals that no single prompt is universally best—you might need different prompts for technical questions versus billing inquiries.

Production Monitoring Integration

W&B Prompts (their LLM monitoring product) captures production traffic with similar tracing capabilities to LangSmith. The advantage is unified tooling—your experimentation data and production data live in the same platform, making it easier to trace production issues back to specific prompt versions or model configurations.

The evaluation framework lets you run automated quality checks on production outputs. You define evaluation functions (using LLMs as judges, rules-based checks, or custom logic) that score outputs for properties like correctness, safety, relevance, and conciseness. These scores create time-series metrics that reveal quality degradation before it becomes obvious in user behavior.

When W&B Makes Sense

Teams already using Weights & Biases for ML model training get significant value from extending their workflow to LLM applications. The learning curve is minimal, and the integrated experience from experimentation to production is smoother than stitching together separate tools. For teams not already in the W&B ecosystem, the value depends on how important structured experimentation is to your workflow.

The platform shines for teams treating prompt engineering as a scientific process rather than trial-and-error. If you're running systematic evaluations, comparing prompt versions across representative datasets, and making data-driven decisions about prompt changes, W&B provides the infrastructure to do this rigorously. For smaller teams doing simpler applications, it may be overkill.

Arize AI: Production Model Monitoring

Arize focuses on production monitoring for ML models and extends these capabilities to LLMs. The core differentiator is sophisticated drift detection—the platform identifies when model inputs, outputs, or behavior patterns shift from baseline distributions. For LLMs, this translates to detecting prompt distribution changes, output quality degradation, and the semantic drift that indicates your application behavior is changing in ways you didn't intend.

Drift Detection for LLM Applications

LLM applications drift in ways traditional ML models don't. Your users start asking different types of questions, external knowledge bases update, or model behavior changes after provider updates. Arize's drift detection captures these shifts by analyzing embedding distributions for inputs and outputs over time. Significant distribution shifts trigger alerts so you can investigate before the drift causes user-visible problems.

from arize.pandas.logger import Client, Schema
from arize.utils.types import ModelTypes, Environments

# Initialize Arize client
arize_client = Client(api_key=ARIZE_API_KEY)

# Log predictions with context
schema = Schema(
  prediction_id_column_name="request_id",
  prediction_label_column_name="response",
  prompt_column_names=["system_prompt", "user_message"],
  response_column_names=["assistant_response"],
  embedding_feature_column_names={
    "user_embedding": "user_message_embedding",
    "response_embedding": "response_embedding"
  },
  tag_column_names=["user_id", "feature_name"]
)

# Send batch of predictions
response = arize_client.log(
  dataframe=predictions_df,
  model_id="support-chatbot-v1",
  model_version="1.2.0",
  model_type=ModelTypes.GENERATIVE_LLM,
  environment=Environments.PRODUCTION,
  schema=schema
)

# Arize analyzes distributions and detects drift automatically

The platform uses embeddings to understand semantic content. By tracking how user query embeddings cluster over time, you can detect when users start asking about new topics or when the distribution of question types shifts. This visibility helps explain sudden changes in model performance or cost—if users start asking more complex questions, average tokens and latency will increase even if nothing changed in your code.

Evaluation and Human Feedback

Arize supports both automated evaluation (using LLM judges or custom scoring functions) and human feedback collection. You can sample production responses for manual review and track how human ratings correlate with automated metrics. This validation helps refine automated evaluation—if human reviewers consistently rate responses as poor when your automated score is high, your evaluation logic needs adjustment.

The feedback loop integration lets you collect thumbs up/down ratings from users and correlate this feedback with model behavior. You can analyze which prompt patterns, input types, or response characteristics correlate with negative feedback and prioritize those areas for improvement.

Limitations

Arize's ML heritage shows in the configuration complexity. Setting up proper monitoring requires understanding embedding feature engineering, schema definitions, and drift detection parameters. For teams without ML operations experience, the learning curve is steeper than simpler observability tools. The platform is powerful but not simple.

Cost tracking and management features are limited compared to tools like Helicone. Arize focuses on model quality and behavior monitoring, not economic optimization. You'll likely need a separate tool for cost management unless your primary concern is detecting quality issues.

OpenLLMetry: Open-Source Observability

OpenLLMetry provides open-source LLM observability using OpenTelemetry standards. The project auto-instruments popular LLM libraries (OpenAI SDK, LangChain, LlamaIndex) to emit telemetry data in OpenTelemetry format, which you can send to any compatible backend like Jaeger, Grafana, or Datadog. This approach gives you vendor-neutral observability data that works with your existing monitoring infrastructure.

Auto-Instrumentation Approach

The auto-instrumentation wraps LLM SDK calls to capture prompts, completions, token counts, latency, and errors as OpenTelemetry spans. You get distributed tracing for LLM calls without manual instrumentation, similar to how OpenTelemetry auto-instruments web frameworks and database clients.

from opentelemetry.instrumentation.openai import OpenAIInstrumentor

# Auto-instrument OpenAI SDK
OpenAIInstrumentor().instrument()

# Configure where to send telemetry
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider()
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317"))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

# Now all OpenAI calls are automatically traced
import openai
response = openai.ChatCompletion.create(
  model="gpt-4",
  messages=[{"role": "user", "content": "Hello"}]
)
# Trace data sent to your OTLP collector

The advantage is flexibility. You control where telemetry data goes, how long it's retained, and who has access. For regulated industries or teams with strict data residency requirements, self-hosted observability removes the third-party data sharing concerns of cloud platforms.

Integration with Existing Infrastructure

If you already run Prometheus for metrics, Grafana for dashboards, and Jaeger for distributed tracing, OpenLLMetry fits naturally into that stack. You don't need to learn new tools or consolidate observability on a new platform. The LLM telemetry appears in the same place as your application traces, making it easier to understand how LLM behavior affects overall system performance.

The downside is more assembly required. Commercial platforms provide purpose-built UIs for LLM debugging, prompt comparison, and quality analysis. With OpenLLMetry, you build these capabilities yourself using your observability backend's features. This works well for teams with strong DevOps practices but creates more work than turnkey solutions.

When to Choose Open Source

OpenLLMetry makes sense when data privacy, vendor lock-in concerns, or cost constraints outweigh the convenience of commercial platforms. You get basic observability—request tracing, error tracking, performance metrics—without ongoing subscription costs or data sharing. For basic production monitoring, this often suffices.

The open-source approach also makes sense for teams already invested in OpenTelemetry infrastructure. Adding LLM observability becomes a matter of enabling auto-instrumentation rather than integrating a new vendor. The telemetry data uses standard formats, so switching backends or adding analytics tools doesn't require reinstrumentation.

Building Custom Observability

For specific requirements that existing tools don't address, custom observability infrastructure may be necessary. This makes sense when you need domain-specific quality metrics, tight integration with internal systems, or observability for self-hosted models that third-party tools don't support well.

Core Components of Custom Observability

A minimal custom observability system needs request logging, metrics aggregation, and alerting. The logging layer captures every LLM interaction with inputs, outputs, metadata, and timing. The metrics layer aggregates this data into time-series metrics like requests per minute, average cost per request, error rates by type, and quality scores. The alerting layer monitors metrics for anomalies and notifies teams when thresholds are exceeded.

// Minimal observability middleware
class LLMObservability {
  async trackRequest(
    requestId: string,
    metadata: RequestMetadata,
    execute: () => Promise
  ): Promise {
    const startTime = Date.now();

    try {
      const response = await execute();

      // Log successful request
      await this.logRequest({
        requestId,
        timestamp: startTime,
        duration: Date.now() - startTime,
        userId: metadata.userId,
        feature: metadata.feature,
        model: response.model,
        inputTokens: response.usage.promptTokens,
        outputTokens: response.usage.completionTokens,
        cost: this.calculateCost(response),
        success: true,
        // Optionally log prompt/completion
        prompt: metadata.includeContent ? response.prompt : undefined,
        completion: metadata.includeContent ? response.completion : undefined
      });

      // Update metrics
      this.metrics.increment("llm.requests.total", {
        model: response.model,
        feature: metadata.feature
      });

      this.metrics.histogram("llm.latency", Date.now() - startTime, {
        model: response.model
      });

      this.metrics.histogram("llm.cost", this.calculateCost(response), {
        user: metadata.userId
      });

      return response;
    } catch (error) {
      // Log failed request
      await this.logRequest({
        requestId,
        timestamp: startTime,
        duration: Date.now() - startTime,
        error: error.message,
        success: false
      });

      this.metrics.increment("llm.requests.errors", {
        errorType: error.constructor.name
      });

      throw error;
    }
  }

  private calculateCost(response: LLMResponse): number {
    const pricing = this.modelPricing[response.model];
    return (
      (response.usage.promptTokens * pricing.input) +
      (response.usage.completionTokens * pricing.output)
    ) / 1000; // Pricing is per 1K tokens
  }
}

This basic wrapper provides visibility into costs, latency, error rates, and usage patterns. It's enough for many production applications and takes a few hours to implement versus days or weeks integrating and configuring commercial platforms.

When Custom Makes Sense

Custom observability is most justified when you have unique requirements that commercial tools don't address: proprietary quality metrics based on domain knowledge, integration with internal user behavior analytics, or observability for models running on internal infrastructure. The development effort pays off when the alternative is stitching together multiple partial solutions or when data privacy requirements make third-party tools non-viable.

The cost advantage emerges at scale. For applications making millions of LLM calls monthly, commercial platform costs can reach thousands of dollars per month. Custom infrastructure has upfront development cost but lower marginal cost per request, making it more economical above a certain volume threshold.

Choosing the Right Tool for Your Needs

The optimal observability approach depends on your application complexity, team expertise, and priorities. Simple applications with single-step LLM calls need less sophisticated tooling than complex agent-based systems. Teams prioritizing cost optimization have different requirements than teams focused on output quality.

Many teams use multiple tools: Helicone for cost management and caching, LangSmith for debugging complex executions, and custom metrics for business-specific quality tracking. This hybrid approach optimizes for different observability needs without depending entirely on any single vendor.

Implementation Best Practices

Sampling for High-Volume Applications

Logging every request works until you reach scale. At millions of requests monthly, storing full prompts and completions becomes expensive and often unnecessary. Implement sampling strategies that capture representative traffic without overwhelming storage systems.

Sample all errors and edge cases (high latency, high cost, quality scores below threshold) while sampling routine successful requests at lower rates. This ensures you have complete data for problematic requests while reducing storage costs for standard operations.

Privacy-Preserving Observability

LLM observability often requires capturing sensitive data—user messages, personal information, proprietary content. Implement redaction pipelines that remove or hash PII before logging. Use one-way hashes for user identifiers so you can track per-user patterns without storing actual user IDs in observability systems.

// Redaction middleware
function redactSensitiveData(text: string): string {
  // Redact email addresses
  text = text.replace(/[\w.+-]+@[\w.-]+\.[a-zA-Z]{2,}/g, "[EMAIL]");

  // Redact phone numbers
  text = text.replace(/\b\d{3}[-.]?\d{3}[-.]?\d{4}\b/g, "[PHONE]");

  // Redact credit cards
  text = text.replace(/\b\d{4}[-\s]?\d{4}[-\s]?\d{4}[-\s]?\d{4}\b/g, "[CARD]");

  // Redact SSNs
  text = text.replace(/\b\d{3}-\d{2}-\d{4}\b/g, "[SSN]");

  return text;
}

// Hash user IDs for privacy
function hashUserId(userId: string): string {
  return crypto.createHash("sha256")
    .update(userId + SALT)
    .digest("hex")
    .substring(0, 16);
}

Alerting on Meaningful Signals

Alert fatigue destroys observability value. Configure alerts for signals that require action, not just interesting metrics. An alert on "average cost increased 10%" is only useful if you'll investigate and optimize. Alerts should trigger when cost exceeds budget, error rate spikes above baseline, or quality metrics fall below acceptable thresholds.

FAQ

How much does LLM observability typically cost?

Commercial platforms typically charge based on request volume or data ingestion. LangSmith starts at $39/month for small teams, scaling to enterprise pricing at high volumes. Helicone is free for moderate usage with paid plans for advanced features. W&B Prompts pricing depends on your existing W&B contract. For applications with millions of monthly requests, expect costs from hundreds to thousands of dollars monthly. OpenLLMetry is free but requires infrastructure for storage and processing.

Do observability tools add latency to LLM requests?

Most tools use async logging that adds minimal latency. LangSmith and W&B typically add less than 10ms overhead because telemetry is sent after the response returns to the user. Proxy-based tools like Helicone add network round-trip latency (10-30ms typically) because requests route through their infrastructure. For applications where every millisecond matters, measure the actual latency impact in your environment before deploying to production.

Can I use observability tools with self-hosted models?

Tools with SDK-based instrumentation (LangSmith, W&B, OpenLLMetry) work with any model API including self-hosted. Proxy-based tools like Helicone require the target API to be compatible with the proxied provider's interface. For completely custom model APIs, OpenLLMetry or custom observability may be the only options unless you build adapter layers to make your API look like OpenAI's.

How should I handle observability for sensitive applications?

For applications handling regulated data (healthcare, finance), implement redaction before sending to observability platforms or use self-hosted solutions. OpenLLMetry with self-hosted backends keeps all data in your infrastructure. Some commercial platforms offer private deployment options but at significant cost premium. Always review vendor security certifications and data handling policies against your compliance requirements.

What metrics should I track for LLM applications?

Essential metrics include request count, error rate, latency (p50, p95, p99), cost per request, token usage, and model-specific metrics like cache hit rate. Application-specific quality metrics depend on your use case—for chatbots, track conversation length and resolution rate; for code generation, track compilation success rate. Start with infrastructure metrics, add cost tracking early, then layer in quality metrics as you understand what predicts user satisfaction.

How do I evaluate LLM output quality automatically?

Common approaches include LLM-as-judge (using another model to score outputs), rule-based checks (detecting banned content, format validation), similarity scoring (comparing outputs to reference answers), and user feedback collection. No single metric captures quality completely—use multiple signals. LLM judges work well for subjective qualities like helpfulness but can be expensive. Start with cheap rules-based checks and add sophistication where needed.

Should I log full prompts and completions?

Full logging is valuable for debugging but expensive at scale and raises privacy concerns. Log everything in development and initial production. As volume grows, sample aggressively for successful requests while capturing all failures and edge cases. Use hash-based sampling to ensure you can trace specific user journeys when needed without storing everything. Implement automatic data retention policies to delete old logs.

How do I detect when model quality degrades?

Track evaluation metrics over time and alert on degradation. Compare recent outputs against baseline distributions using embeddings to detect semantic drift. Monitor user feedback signals like thumbs down rates or support ticket creation following interactions. Set up regression test suites with known inputs and expected outputs, running them periodically to catch quality regressions before users report them.

What's the difference between observability and monitoring?

Monitoring tells you when something is broken through predefined metrics and alerts. Observability lets you investigate why it broke through rich data exploration. For LLMs, monitoring tracks request rates and error counts. Observability lets you examine the actual prompts and completions to understand why quality degraded or costs spiked. You need both—monitoring for detection, observability for diagnosis.

Can I switch observability tools later without disrupting production?

SDK-based tools are easier to swap than proxy-based. With SDKs, you change initialization code and redeploy. With proxies, you change the API endpoint and need to handle the migration carefully to avoid downtime. Using abstraction layers (wrapping observability calls in your own interfaces) makes switching easier but adds initial complexity. Consider vendor lock-in risk when choosing tools—open-source or tools with data export APIs reduce switching costs.

Conclusion

LLM observability is essential for production applications but the right approach depends on your specific requirements. Complex applications with multiple chained operations need sophisticated tracing tools like LangSmith. Cost-sensitive applications benefit from Helicone's optimization features. Teams with strict data governance should consider self-hosted options like OpenLLMetry or custom solutions.

Start with basic observability—request logging, cost tracking, error monitoring—then add specialized tools as needs emerge. The mistake many teams make is over-investing in observability infrastructure before understanding which signals actually predict problems. Build incrementally, prioritizing observability for the most critical paths in your application, and expand coverage based on where you encounter production issues that existing instrumentation can't diagnose.