OpenTelemetry GenAI Observability: A Production Guide

Published July 24, 2026 · Research window: July 24, 2026 (UTC) · Audience: AI and full-stack engineers

If an LLM application fails, “the model was slow” is not a diagnosis. The delay may be retrieval, queueing, the provider, streaming, tool execution, or a retry hidden inside an SDK. OpenTelemetry GenAI observability gives those steps a common telemetry shape, so one request can be followed from the HTTP edge through RAG, model calls, tools, and the exporter.

This guide shows how to build that view in a Node.js service without turning prompts into a permanent data leak. The design is based on OpenTelemetry’s current JavaScript documentation, the official GenAI semantic-conventions repository, the OpenTelemetry GenAI project note, and the W3C Trace Context specification. Where a convention is still marked Development, I label it as such rather than treating it as a settled contract.

The architecture: observe the request, not just the model

The useful unit is a request trace with nested spans. The root span represents the application operation. Child spans represent retrieval, prompt assembly, the provider call, and any tool or post-processing step. Metrics answer aggregate questions; events can preserve selected interaction details when policy permits.

Original diagram by Essa Mamdani. Conceptual architecture only; it contains no measured benchmark or vendor UI.

The diagram’s main design choice is the privacy gate between collection and storage. Redaction and sampling belong in the telemetry path, not in a dashboard query after sensitive data has already been copied into several systems. This is an architectural recommendation derived from the GenAI conventions’ opt-in treatment of message content and from the practical need to control data before export.

Courtesy: OpenTelemetry Community. Source: https://opentelemetry.io/docs/languages/js/. Accessed: 2026-07-24.

This visual proves the current status boundary for JavaScript: the official page lists traces and metrics as Stable, logs as Development, and browser client instrumentation as experimental and mostly unspecified. The article therefore focuses on server-side tracing and metrics, with logs/events treated as an opt-in design decision.

What the GenAI conventions standardize

The official GenAI semantic-conventions repository extends OpenTelemetry’s general conventions with GenAI spans, metrics, events, agent spans, MCP, and provider-specific material. Its human-readable documents are generated from YAML models, so the repository is the source to inspect when an attribute changes.

The current repository labels the main GenAI documents Development. That does not make them unusable; it changes how you adopt them. Pin your instrumentation version, keep provider adapters behind a small interface, and expect attribute names or requirements to move. Do not build billing or incident policy around an undocumented field without a compatibility test.

The useful fields fall into four groups:

The repository also documents gen_ai.data_source.id for agent and RAG workflows. Use a stable logical identifier such as support-kb-v3, not a raw connection string or a user’s document title. That lets you compare retrieval sources without putting secrets or personal data into every span.

A safe Node.js instrumentation boundary

OpenTelemetry JavaScript supports Node.js and browser environments. Its official status page lists server traces and metrics as Stable, while the repository’s GenAI semantic conventions remain Development. The code below uses stable OpenTelemetry APIs and adds the GenAI attribute names as ordinary span attributes. That separation keeps the application portable while the GenAI schema evolves.

Install the API and a Node SDK in the service that owns the model call:

1npm install @opentelemetry/api @opentelemetry/sdk-node \
2  @opentelemetry/exporter-trace-otlp-http

Initialize the SDK before importing application modules. The official OpenTelemetry JavaScript repository demonstrates the same early-registration pattern and uses a console exporter for a local smoke test.

1// instrumentation.mjs
2import { NodeSDK } from '@opentelemetry/sdk-node';
3import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
4import { resourceFromAttributes } from '@opentelemetry/resources';
5import { ATTR_SERVICE_NAME } from '@opentelemetry/semantic-conventions';
6
7const sdk = new NodeSDK({
8  resource: resourceFromAttributes({
9    [ATTR_SERVICE_NAME]: 'answer-api',
10  }),
11  traceExporter: new OTLPTraceExporter({
12    url: process.env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT,
13  }),
14});
15
16await sdk.start();
17
18const shutdown = () => sdk.shutdown().catch((error) => {
19  console.error('OpenTelemetry shutdown failed', error);
20});
21process.once('SIGTERM', shutdown);
22process.once('SIGINT', shutdown);

Now wrap the logical operation and model call. The example deliberately does not attach the prompt, completion, authorization header, or retrieved document text. It records a prompt template version and stable IDs instead.

1import { context, trace, SpanStatusCode } from '@opentelemetry/api';
2
3const tracer = trace.getTracer('answer-api', '0.1.0');
4
5export async function answerQuestion({ question, conversationId }) {
6  return tracer.startActiveSpan('answer', async (rootSpan) => {
7    rootSpan.setAttribute('gen_ai.conversation.id', conversationId);
8    rootSpan.setAttribute('gen_ai.prompt.name', 'support-answer');
9    rootSpan.setAttribute('gen_ai.prompt.version', '2026-07-24');
10
11    try {
12      const contextDocs = await tracer.startActiveSpan('rag.retrieve', async (span) => {
13        span.setAttribute('gen_ai.data_source.id', 'support-kb-v3');
14        const docs = await retrieveDocuments(question);
15        span.setAttribute('rag.documents.count', docs.length);
16        span.end();
17        return docs;
18      });
19
20      const result = await tracer.startActiveSpan('chat model call', async (span) => {
21        span.setAttribute('gen_ai.operation.name', 'chat');
22        span.setAttribute('gen_ai.provider.name', 'provider.example');
23        span.setAttribute('gen_ai.request.model', process.env.CHAT_MODEL);
24        span.setAttribute('gen_ai.request.stream', true);
25
26        try {
27          const response = await callModel({ question, contextDocs });
28          const usage = response.usage ?? {};
29          span.setAttribute('gen_ai.response.model', response.model ?? 'unknown');
30          span.setAttribute('gen_ai.response.finish_reasons', response.finishReasons ?? []);
31          if (Number.isInteger(usage.inputTokens)) {
32            span.setAttribute('gen_ai.usage.input_tokens', usage.inputTokens);
33          }
34          if (Number.isInteger(usage.outputTokens)) {
35            span.setAttribute('gen_ai.usage.output_tokens', usage.outputTokens);
36          }
37          return response.text;
38        } catch (error) {
39          span.recordException(error);
40          span.setStatus({ code: SpanStatusCode.ERROR });
41          throw error;
42        } finally {
43          span.end();
44        }
45      });
46
47      rootSpan.setStatus({ code: SpanStatusCode.OK });
48      return result;
49    } catch (error) {
50      rootSpan.recordException(error);
51      rootSpan.setStatus({ code: SpanStatusCode.ERROR });
52      throw error;
53    } finally {
54      rootSpan.end();
55    }
56  });
57}

There is one subtle implementation issue in nested asynchronous spans: every span must end exactly once, including retrieval failures. In a shared helper, put span.end() in finally; the shortened retrieval example above assumes retrieveDocuments succeeds. In production, make that helper own its lifecycle so a rejected promise cannot leave an open span.

Metrics: aggregate without destroying signal

Metrics are where high-cardinality mistakes become expensive. A histogram of model latency can be useful. A time series whose labels include the full prompt, conversation ID, or response ID can exhaust a metrics backend and expose user data.

Good dimensions are bounded and operational: service, environment, provider, requested model, operation name, and outcome. Keep conversation and request identifiers in traces, where they can be sampled and searched with access control. Track token usage as measurements sourced from the provider response, not as a local estimate based on character count. If the provider reports billed and model-consumed counts separately, store the billing count for cost analysis and document the distinction.

The official GenAI metrics document defines Development metrics for client token usage and operation duration. That is a useful map for naming and aggregation, not a promise that every backend or instrumentation library emits every metric today. Validate your exporter and backend with a fixture response from each provider.

Propagation: keeping a distributed trace intact

The W3C Trace Context recommendation standardizes the traceparent and tracestate headers. OpenTelemetry uses W3C Trace Context as a default propagation format in common SDK setups. If an API gateway, queue consumer, RAG worker, and model proxy each start a new root trace, you lose the causal chain even though each service has telemetry.

Use the SDK propagator at service boundaries. Do not manually copy a trace ID into a custom header unless you also have a documented migration plan. For messages that sit in a queue, attach context to message metadata and extract it when the consumer starts a processing span. Treat incoming context as untrusted input: validate it through the SDK and never use a trace identifier for authorization.

Privacy controls for prompts, tools, and RAG

The official GenAI material distinguishes operational attributes from message content. The latter is opt-in for a reason. Prompts can contain credentials, customer records, source code, health information, or documents retrieved from an internal index. A “debug mode” that captures everything should be a short-lived, access-controlled incident switch with a documented expiry.

Use this policy sequence:

1. Default to metadata-only spans: model, provider, operation, latency, status, token counts, and stable template or corpus identifiers.
2. Redact secrets before telemetry leaves the process. Do not rely on a backend processor to find every API key format.
3. Sample normal traffic and retain error traces at a higher rate, but apply the same redaction rules to both paths.
4. If content capture is necessary, store a reference to a separately protected object with a short retention period rather than copying raw content into every span.
5. Test tool arguments and retrieved chunks as data exfiltration paths. A model response is not automatically safe because it came from your own provider.

These are engineering controls, not legal advice. Data classification, retention, access review, and deletion workflows still belong to the service owner and the organization’s privacy program.

A practical rollout checklist

Start with one user journey and one provider. Before adding dashboards, verify the trace itself.

• Register the Node SDK before application imports.
• Create one root span for the user-visible operation.
• Add child spans for retrieval, model calls, tools, retries, and post-processing.
• Record requested and returned model names separately.
• Add provider-reported token usage only when the field is present.
• Keep prompts, completions, tool arguments, and retrieved text out by default.
• Test traceparent across the gateway, worker, and model proxy.
• Bound metric labels and review cardinality before production export.
• Pin the GenAI convention version and run a fixture test on dependency upgrades.
• Define who can enable content capture, why, for how long, and how it is deleted.

For adjacent architecture context, see my guides to AI agent stacks, MCP, ACP, memory, and workflows, the complete MCP production guide, and multi-model AI routing. Observability becomes more useful when those system boundaries are explicit.

FAQ

Is OpenTelemetry GenAI observability production-ready?

OpenTelemetry’s core JavaScript traces and metrics are listed as Stable, but the dedicated GenAI semantic-convention documents are currently marked Development. You can use them in production with version pinning, fixture tests, and a provider adapter that isolates schema changes. Treat the names as an evolving contract, not an immutable billing API.

Should I log prompts and model responses?

Not by default. Start with metadata-only traces and add content capture only for a defined debugging or evaluation purpose. Redact secrets before export, restrict access, set a short retention period, and make the capture switch auditable. Prompt content may contain personal data, credentials, proprietary code, or retrieved internal documents.

What should an LLM trace contain?

At minimum, connect the user operation to retrieval, model, tool, retry, and post-processing spans. Record provider, requested model, returned model when available, operation name, status, latency, stream mode, and provider-reported usage. Keep raw content and unbounded identifiers out unless an explicit policy permits them.

Do token counts prove cost?

No. Token counts are evidence used in a cost calculation, not the calculation itself. Providers can expose billed units, model-consumed units, cached tokens, or separate reasoning usage. Store the provider and model, record which response field supplied the count, and join it with the price schedule that applied at the time.

Is a trace ID an authorization token?

Never. A trace ID is a correlation value, not a permission. Accept and propagate trace context through the SDK, but authorize users with your normal identity and policy system. Avoid placing sensitive identifiers in URLs or returning internal trace metadata to untrusted clients.

Should I use a vendor-specific LLM observability product instead?

That is a deployment choice, not an either-or telemetry decision. OpenTelemetry gives you a portable instrumentation and propagation layer; a vendor can receive OTLP data and provide storage or analysis. Compare retention, content controls, provider coverage, query cost, and export options on a representative trace set before committing.

Conclusion

The production lesson is simple: trace the work around the model, standardize the metadata you can defend, and make content capture exceptional. OpenTelemetry GenAI observability is most valuable when it connects RAG, model calls, tools, retries, and user outcomes without turning every prompt into a permanent log entry. Start with one path, verify the trace across boundaries, and upgrade the developing GenAI conventions behind tests and a privacy gate.

Primary sources

• OpenTelemetry JavaScript documentation
• OpenTelemetry JavaScript repository
• OpenTelemetry GenAI semantic conventions
• OpenTelemetry for Generative AI
• W3C Trace Context

Author context: Essa Mamdani writes about AI systems, full-stack architecture, developer tooling, and production engineering. The recommended implementation is a reference design; verify SDK versions, provider fields, retention rules, and backend behavior in your own environment before rollout.