Part
Observability platform
You might say: run telemetry, execution visibility, tracing backend, logs, monitoring, dashboard
The optional place where a factory records run status, logs, traces, token use and cost, and shows the factory's metrics. The orchestrator's run page may be enough for a small factory. Software reads the signals first, and a person is called only when they need to act.
- What it does
- Makes failures, slow steps, tool calls, and spend visible across many runs.
- Ask
- What must a human learn from one failed run, and what must they compare across runs?
- Typical
- Choose no extra platform, LLM or agent tracing, general APM, or an LLM gateway with usage accounting.
- Trouble
- A run fails without a useful log, spans cannot join to the run, sensitive prompts leak into telemetry, or spend cannot be assigned to a task.
What to look for
- A run page with status, logs, artifacts, and timestamps
- A trace tree linking model calls and tool calls
- A dashboard for token use, cost, latency, or error rate
Parts
- Run log: The ordered record of a run's status changes, commands, tool results, warnings, and final outcome.
- Agent trace: A trace links one run's model requests, tool calls, and hooks into a parent-child timeline.
- Token cost: The money spent by one run on model input, model output, cached tokens, and related calls.
- Factory metrics: The measures that show whether automated work flows, waits, fails, and recovers in a useful way.
Catalogue: Platform type
- None: The orchestrator's own run records serve as the telemetry. A person can view each run and its logs, and re-run or cancel it, with no other platform. The run records answer the factory's operational questions without a separate telemetry platform.
- LLM or agent tracing: A tracing product records LLM calls, tool calls, retrieval steps, and other observations. It shows the tool path inside an LLM run.
- General APM: A telemetry backend collects signals such as traces, metrics, and logs for the runner and its services. It covers signal types beyond model-specific events.
- LLM gateway: A model gateway records usage and spend for requests. It attributes usage centrally on the model request path.
Sources
- Anthropic, Monitoring, Monitoring
Track Claude Code usage, costs, and tool activity across your organization by exporting telemetry data through OpenTelemetry (OTel).
- OpenTelemetry, Signals, Signals
The purpose of OpenTelemetry is to collect, process, and export signals.
- Arize Phoenix, Overview: Tracing, Overview: Tracing
LLM tracing records the paths taken by requests as they propagate through multiple steps or components of an LLM application.
- Betsy Beyer et al., Site Reliability Engineering, Chapter 1, Introduction
Monitoring is one of the primary means by which service owners keep track of a system’s health and availability.
- Betsy Beyer et al., Site Reliability Engineering, Chapter 1, Introduction
Monitoring should never require a human to interpret any part of the alerting domain. Instead, software should do the interpreting, and humans should be notified only when they need to take action.
- Matt Pocock, software-factory skill (pull request #1316), Services
Where runners send logs, traces and status, so a human can see what happened inside each run.
Use it on your project
teardown-apply: hold "Observability platform" (observability) from the teardown at https://teardownatlas.com/software-factory/ against my project
Paste it into a coding agent that has the teardown skills.
An AI agent wrote this map with the teardown skill. It can contain errors: check the sources before you rely on a part. About and legal