Marketing Foundation
Compact two-skill starter: clarify positioning and choose a lead magnet. Use Marketing Launch for the broader eight-skill go-to-market workflow.
The tunnel uses the same cards as the catalogue. Browse only as deep as needed — or load a broad bundle immediately.
SEO, Sales, Agents or another broad area → one bundle call → work.
Read-only access to published skills. Default 8, maximum 10 skills / 120,000 characters.
Compact two-skill starter: clarify positioning and choose a lead magnet. Use Marketing Launch for the broader eight-skill go-to-market workflow.
Build an evidence-led marketing plan from ICP and competition through positioning, campaigns, growth and measurement.
Diagnose architecture and context, plan agent-team responsibilities, then organize project context and session handoffs. Memory and cost-runtime reviews remain outside this pack.
Review the journey from landing page and lead capture through registration, first value and transparent upgrades.
Plan a campaign, draft its channel content and review the work against actual brand guidance.
Prioritize an editorial roadmap and plan how to launch and distribute it across suitable channels.
Understand customer needs, compare competitors and plan a community around real member value.
Choose a relevant lead magnet, then draft a permission-based nurture journey with entry, suppression and exit rules.
Define the API contract, then plan how to observe its latency, failures and retries. Guidance and checklist; no production changes.
Profile a dataset, choose and interpret statistical methods, then validate calculations and conclusions before sharing.
Plan logs, metrics, traces and actionable runbooks for an existing service; use API Contracts first when the missing piece is interface behavior rather than runtime evidence.
---
name: observability-and-instrumentation
description: Instruments code so production behavior is visible and diagnosable. Use when adding logging, metrics, tracing, or alerting. Use when shipping any feature that runs in production and you need evidence it works. Use when production issues are reported but you can't tell what happened from the available data.
---
# Observability and Instrumentation
## Overview
Code you can't observe is code you can't operate. Observability is the ability to answer "what is the system doing and why?" from the outside, using the telemetry the code emits. Instrumentation is not a post-launch add-on — it's written alongside the feature, the same way tests are. If a feature ships without telemetry, the first user-reported bug becomes archaeology instead of a query.
## When to Use
- Building any feature that will run in production
- Adding a new service, endpoint, background job, or external integration
- A production incident took too long to diagnose ("we couldn't tell what happened")
- Setting up or reviewing alerting rules
- Reviewing a PR that adds I/O, retries, queues, or cross-service calls
**NOT for:**
- Diagnosing a failure happening right now — use the `debugging-and-error-recovery` skill (observability is what makes that skill fast next time)
- Profiling and optimizing measured slowness — use the `performance-optimization` skill
- Launch-day monitoring checklists and rollback triggers — see the `shipping-and-launch` skill; this skill covers the instrumentation that feeds them
## Process
### 1. Define "working" before instrumenting
Telemetry without a question is noise. Before adding any instrumentation, write down 2–4 questions an on-call engineer will ask about this feature:
```
FEATURE: checkout payment retry
QUESTIONS ON-CALL WILL ASK:
1. What fraction of payments succeed on first attempt vs after retry?
2. When a payment fails permanently, why? (provider error? timeout? validation?)
3. Is the payment provider slower than usual?
→ Every signal below must help answer one of these.
```
If you can't name the questions, you're not ready to instrument — you'll log everything and learn nothing.
### 2. Pick the right signal for each question
| Signal | Answers | Cost profile | Example |
|---|---|---|---|
| **Structured log** | "What happened in this specific case?" | Per-event; grows with traffic | `payment_failed` with provider error code |
| **Metric** | "How often / how fast, in aggregate?" | Fixed per series; cheap to query | p99 latency of provider calls |
| **Trace** | "Where did time go across services?" | Per-request; usually sampled | One slow checkout, broken down by hop |
Rule of thumb: metrics tell you **that** something is wrong, traces tell you **where**, logs tell you **why**.
### 3. Structured logging
Log events, not prose. Every log line is a JSON object with a stable event name and machine-readable fields:
```typescript
// BAD: string interpolation — unqueryable, inconsistent
logger.info(`Payment ${id} failed for user ${userId} after ${n} retries`);
// GOOD: stable event name + structured fields
logger.warn({
event: 'payment_failed',
paymentId: id,
provider: 'stripe',
errorCode: err.code,
attempt: n,
}, 'payment failed');
```
**Log levels — use them consistently:**
| Level | Meaning | On-call action |
|---|---|---|
| `error` | Invariant broken; someone may need to act | Investigate |
| `warn` | Degraded but handled (retry succeeded, fallback used) | Watch for trends |
| `info` | Significant business event (order placed, job finished) | None |
| `debug` | Diagnostic detail | Off in production by default |
**Correlation IDs are mandatory.** Generate (or accept) a request ID at the system boundary and attach it to every log line, span, and outbound call. Without it, you cannot reconstruct a single request from interleaved logs:
```typescript
// Express: child logger per request, ID propagated downstream
app.use((req, res, next) => {
req.id = req.headers['x-request-id'] ?? crypto.randomUUID();
req.log = logger.child({ requestId: req.id });
res.setHeader('x-request-id', req.id);
next();
});
```
**When several entry points write to one log, name the entry point.** A correlation ID identifies a run; it does not say which code path started it. The same job reached by a scheduler, by a replay endpoint, and by a manual CLI run produces interchangeable lines in one sink, so attributing a line falls back to elimination — cross-reading the scheduler's history, the process table, a deploy log — and that argument holds only as long as those external records happen to still exist. Stamp the entry point where the run starts, next to the correlation ID, and propagate both the same way:
```typescript
// One helper for every entry point: the run's own logger carries both fields.
// `entryPoint`, not `source` — ECS reserves `source.*` for network fields.
export const runLog = (entryPoint: 'scheduler' | 'replay_endpoint' | 'cli', runId: string) =>
logger.child({ entryPoint, requestId: runId });
// scheduler tick -> runLog('scheduler', crypto.randomUUID())
// POST /jobs/:id/replay -> runLog('replay_endpoint', req.id)
// CLI invocation -> runLog('cli', process.env.RUN_ID ?? crypto.randomUUID())
```
Both fields have to cross the same boundaries as the correlation ID — queue metadata, HTTP headers — or a worker re-derives the entry point and guesses. A field that merely correlates with an entry point is a hint, not an attribution: anything that can invoke the job can reproduce it.
**Never log secrets, tokens, passwords, or full PII.** This is a hard rule from the `security-and-hardening` skill — telemetry pipelines are a classic data-leak path. Allowlist fields; don't log whole request bodies.
### 4. Metrics
For request-driven services, instrument **RED** on every endpoint and every external dependency: **R**ate (requests/sec), **E**rrors (failure rate), **D**uration (latency histogram, not average). For resources (queues, pools, hosts), use **USE**: **U**tilization, **S**aturation, **E**rrors.
As with tracing, the vendor-neutral path is the OpenTelemetry metrics API (same SDK and context as step 5). The example below uses Prometheus' `prom-client` — one common backend choice, not the only one; the RED/USE and cardinality rules are identical either way.
```typescript
import { Histogram } from 'prom-client';
const httpDuration = new Histogram({
name: 'http_request_duration_seconds',
help: 'HTTP request duration',
labelNames: ['method', 'route', 'status_class'], // '2xx', not '200'
buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
});
```
**Cardinality is the failure mode.** Every unique label combination is a separate time series. Labels must come from small, fixed sets (route template, status class, provider name). Never use user IDs, raw URLs, error messages, or other unbounded values as labels — that belongs in logs and traces.
```
OK as label: route="/api/tasks/:id" status_class="5xx" provider="stripe"
NEVER a label: user_id, email, request_id, full URL, error message text
```
Track averages never, percentiles always: an average hides the 1% of users having a terrible time. Use histograms and read p50/p95/p99.
### 5. Distributed tracing
Use OpenTelemetry — it's the vendor-neutral standard, and auto-instrumentation covers HTTP, gRPC, and common DB clients with near-zero code:
```typescript
// tracing.ts — must be imported before anything else
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
const sdk = new NodeSDK({
serviceName: 'checkout-service',
instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();
```
Add manual spans only around meaningful internal units of work (e.g., `applyDiscounts`, `chargeProvider`) and attach the attributes on-call will filter by. Propagate context across every async boundary — HTTP headers, queue message metadata — or the trace dies at the gap. Sample head-based at a low rate by default; keep 100% of errors if your backend supports tail sampling.
### 6. Alerting
Alert on **symptoms users feel**, not on causes:
```
SYMPTOM (page-worthy): CAUSE (dashboard, not a page):
error rate > 1% for 5 min CPU at 85%
p99 latency > 2s one pod restarted
queue age > 10 min disk at 70%
```
Cause-based alerts fire when nothing is wrong and miss failures you didn't predict. Symptom-based alerts fire exactly when users are hurt, regardless of the cause.
Rules for every alert you create:
1. **It must be actionable.** If the response is "ignore it, it self-heals", delete the alert.
2. **It links to a runbook** — even three lines: what it means, first query to run, escalation path.
3. **It has a threshold and duration** justified by the SLO or by historical data, not by a guess.
4. Use two severities only: **page** (user-facing, act now) and **ticket** (degradation, act this week). A third tier becomes noise that trains people to ignore everything.
#### Writing Runbooks
Rule 2 above requires every alert to link to a runbook. A runbook's job is to answer three questions without requiring the reader to think: what is happening, what to check first, and who to call if that doesn't resolve it. Store in `docs/runbooks/` named after the alert.
**Minimum viable runbook (three lines):**
```markdown
# Runbook: High Error Rate on /api/tasks
**Means:** DB connection pool likely exhausted, or a bad deploy.
**First check:** `SELECT count(*) FROM pg_stat_activity WHERE backend_type = 'client backend';`
— if count > pool limit, see Step 2. (Swap in the equivalent for your database.)
**Escalate to:** #db-oncall or engineering on-call rotation.
```
**When to expand beyond three lines:** add steps only when the first check alone isn't enough to decide. A five-step runbook that covers the three most common causes is better than a twenty-step document that covers every edge case and gets skimmed.
**Keep runbooks current.** Update the runbook as part of closing every incident it was used in — a stale runbook builds false confidence. If a step was wrong or missing, fix it before marking the incident resolved.
### 7. Verify the telemetry itself
Instrumentation is code; it can be wrong. Before calling the work done, trigger the paths and look at the actual output:
- Force an error in staging → find it in the logs by `requestId`, confirm fields are structured (not `[object Object]`)
- Send test traffic → confirm metric series appear with the expected labels and sane values
- Follow one request across services in the tracing UI → no broken spans
- Fire each new alert once (lower the threshold temporarily) → confirm it reaches the right channel and the runbook link works
## Common Rationalizations
| Rationalization | Reality |
|---|---|
| "I'll add logging after it works" | "After" becomes "after the first incident", which is the most expensive moment to discover you're blind. Instrument as you build. |
| "More logs = more observability" | Unstructured noise makes incidents slower, not faster. Three queryable events beat three hundred prose lines. |
| "console.log is fine for now" | Unstructured output can't be filtered, correlated, or alerted on. The structured logger costs five extra minutes once. |
| "We can just look at the dashboards when something breaks" | Dashboards built without defined questions show you everything except the answer. Start from on-call questions. |
| "Alert on everything important, we'll tune later" | A noisy pager trains people to ignore it. The tuning never happens; the missed real page does. |
| "User ID as a metric label makes debugging easier" | It also makes your metrics backend fall over. High-cardinality lookups belong in logs and traces. |
| "Tracing is overkill for our two services" | Two services already means cross-service latency questions logs can't answer. Auto-instrumentation makes the cost trivial. |
## Red Flags
- A feature PR with retries, queues, or external calls and zero new telemetry
- Log lines built by string interpolation instead of structured fields
- No correlation/request ID — each log line is an orphan
- One log stream fed by a scheduler, a webhook, and manual runs, with no field naming which one produced the line
- Metrics labeled with user IDs, raw URLs, or error message text (cardinality bomb)
- Latency tracked as an average with no percentiles
- Alerts that fire daily and get acknowledged without action
- Alerts on causes (CPU, memory) paging humans while user-facing error rate is unmonitored
- Secrets, tokens, or full request bodies appearing in logs
- "It works on my machine" as the only evidence a production feature is healthy
## Verification
After instrumenting a feature, confirm:
- [ ] The on-call questions for this feature are written down, and each signal maps to one
- [ ] All log output is structured (JSON), with stable event names and a correlation ID on every line
- [ ] Every log sink written by more than one entry point carries an entry-point field, set where the run starts and propagated with the correlation ID rather than inferred downstream
- [ ] No secrets, tokens, or unredacted PII in any log line (spot-check actual output)
- [ ] RED metrics exist for every new endpoint and every external dependency, with bounded label sets
- [ ] Latency is a histogram; p95/p99 are queryable
- [ ] A single request can be followed end-to-end in the tracing UI without broken spans
- [ ] Every new alert is symptom-based, has a runbook link, and was test-fired once
- [ ] An induced failure in staging was located via telemetry alone, without reading the source
For the at-a-glance version of this list, including the pre-launch instrumentation gate, see `../../references/observability-checklist.md`.
Plan logs, metrics, traces and actionable runbooks for an existing service; use API Contracts first when the missing piece is interface behavior rather than runtime evidence.
A bounded telemetry design, correlation strategy, sampling tradeoffs, alert/runbook definitions and staging verification steps.
Inspect the actual system and constraints, identify invariants and failure cases, draft the design, then define observable acceptance checks.
Service boundaries, on-call questions, SLOs, current telemetry stack, data classification and an authorized staging/test environment.
This delivers an instrumentation plan and the complete checklist, not an installed monitoring stack. Validate incoming request/correlation IDs as bounded safe strings or generate new IDs; do not blindly echo arrays or arbitrary header values, and never trust trace or entry-point labels for authorization. Set entry-point provenance at trusted boundaries. Propagate only approved context to intended services, not every third-party URL. Redact secrets and personal information in both logs and spans, restrict access and retention, and bound cardinality, volume and export cost. Correct the head/tail sampling claim: tail sampling cannot recover spans already dropped by a head sampler. Capturing all error traces requires an appropriately provisioned recording/export path to the tail sampler, and even then capacity/drop limits must be measured; do not promise 100 percent error retention from a low-rate head sample. See https://opentelemetry.io/docs/specs/otel/trace/sdk/ and https://opentelemetry.io/docs/languages/js/sampling/. Keep counts/rates and useful means alongside histogram percentiles where relevant; never average independently computed percentiles. Choose histogram buckets and sampling based on workload/SLO evidence. Capacity/cause alerts can be actionable before users fail; two severities and universal thresholds are examples, not rules for all organizations. Configure and verify an approved exporter/destination and SDK version; the NodeSDK snippet alone proves neither collection nor delivery. Fault injection, test paging and exporter activation need a separately authorized staging plan, owners and rollback. Other linked skills are optional pointers, not installed capabilities. No production failures, messages or telemetry exports are triggered by loading this guidance.
Prepare a observability design for [SERVICE]. Inspect the actual stack, consumers, trust boundaries and failure modes first. Separate verified behavior from assumptions; return concrete contracts or instrumentation changes and a staging validation plan. Preserve existing behavior unless a change is authorized. Do not deploy, charge accounts, export telemetry, induce failures or send alerts.
Automatic production changes, unreviewed dependency installation, runtime/security certification or executing the source examples without validation.
German task routing, explicit scope and failure-mode corrections. This delivers an instrumentation plan and the complete checklist, not an installed monitoring stack. Validate incoming request/correlation IDs as bounded safe strings or generate new IDs; do not blindly echo arrays or arbitrary header values, and never trust trace or entry-point labels for authorization. Set entry-point provenance at trusted boundaries. Propagate only approved context to intended services, not every third-party URL. Redact secrets and personal information in both logs and spans, restrict access and retention, and bound cardinality, volume and export cost. Correct the head/tail sampling claim: tail sampling cannot recover spans already dropped by a head sampler. Capturing all error traces requires an appropriately provisioned recording/export path to the tail sampler, and even then capacity/drop limits must be measured; do not promise 100 percent error retention from a low-rate head sample. See https://opentelemetry.io/docs/specs/otel/trace/sdk/ and https://opentelemetry.io/docs/languages/js/sampling/. Keep counts/rates and useful means alongside histogram percentiles where relevant; never average independently computed percentiles. Choose histogram buckets and sampling based on workload/SLO evidence. Capacity/cause alerts can be actionable before users fail; two severities and universal thresholds are examples, not rules for all organizations. Configure and verify an approved exporter/destination and SDK version; the NodeSDK snippet alone proves neither collection nor delivery. Fault injection, test paging and exporter activation need a separately authorized staging plan, owners and rollback. Other linked skills are optional pointers, not installed capabilities. No production failures, messages or telemetry exports are triggered by loading this guidance.
Addy Osmani · MIT · SHA-256 25fcd0e854596a5ab6f11b002050d245bf78f5ae0fcbd1df699b89451ae0e1e8
# Observability Checklist Quick reference for instrumenting production code. Use alongside the `observability-and-instrumentation` skill. ## Table of Contents - [On-Call Questions (Start Here)](#on-call-questions-start-here) - [Structured Logging](#structured-logging) - [Metrics](#metrics) - [Distributed Tracing](#distributed-tracing) - [Alerting](#alerting) - [Dashboards](#dashboards) - [Verify the Telemetry](#verify-the-telemetry) - [Pre-Launch Gate](#pre-launch-gate) ## On-Call Questions (Start Here) Telemetry without a question is noise. Before instrumenting anything: - [ ] 2–4 questions an on-call engineer will ask about this feature are written down - [ ] Every signal below maps to one of those questions - [ ] Each question is matched to the right signal type: metrics say **that** something is wrong, traces say **where**, logs say **why** ## Structured Logging - [ ] Logs are structured (JSON) with stable event names — not free-form strings - [ ] Every log line carries a correlation/request ID, generated or accepted at the system boundary - [ ] Correlation ID is propagated on every outbound call and async boundary (HTTP headers, queue metadata) - [ ] Any log stream written by more than one entry point (scheduler, replay endpoint, manual run) carries an entry-point field, set where the run starts and propagated alongside the correlation ID - [ ] Log levels are consistent: `error` = invariant broken, someone may act; `warn` = degraded but handled; `info` = significant business event; `debug` = off in production - [ ] No secrets, tokens, passwords, or unredacted PII in any log line (hard rule from `security-and-hardening`) - [ ] Fields are allowlisted — no whole request/response bodies, no auth headers - [ ] External service calls logged with metadata only: endpoint, status, latency, attempt count, sanitized identifiers - [ ] Actual log output spot-checked: structured fields, not `[object Object]` ## Metrics - [ ] **RED** instrumented for every endpoint and every external dependency: Rate, Errors, Duration - [ ] **USE** instrumented for every resource (queues, pools, hosts): Utilization, Saturation, Errors - [ ] Latency is a histogram; p50/p95/p99 queryable — never an average - [ ] All labels come from small, fixed sets (route template, status class, provider name) - [ ] No unbounded label values: no user IDs, tenant IDs, emails, raw URLs, request IDs, or error message text - [ ] Status codes grouped by class (`5xx`, not `503`) - [ ] Queue depth and processing duration tracked for every worker/queue ## Distributed Tracing - [ ] OpenTelemetry (or equivalent) initialized at service startup, before other imports - [ ] Auto-instrumentation enabled for HTTP, gRPC, and DB clients - [ ] Trace context propagated on every outbound call (W3C `traceparent`/`tracestate`) and extracted from every inbound request - [ ] Context survives async boundaries — queue messages carry trace metadata - [ ] Manual spans only around meaningful internal units of work, with the attributes on-call will filter by - [ ] No secrets or PII as span attributes - [ ] Head-based sampling at a low default rate; 100% of errors kept if tail sampling is available ## Alerting - [ ] Every alert is symptom-based (error rate, p99 latency, queue age) — causes (CPU, disk, restarts) go to dashboards, not pagers - [ ] Every alert is actionable; "ignore it, it self-heals" alerts are deleted - [ ] Every alert links to a runbook — minimum three lines: what it means, first query to run, escalation path - [ ] Thresholds and durations justified by an SLO or historical data, not guesses - [ ] Two severities only: **page** (user-facing, act now) and **ticket** (degradation, act this week) - [ ] Each new alert test-fired once: it reached the right channel and the runbook link works - [ ] No alerts that fire daily and get acknowledged without action ## Dashboards - [ ] Service health dashboard exists: error rate, latency p99, traffic, saturation - [ ] Dependency health panel shows per-service error rates and latency - [ ] Dashboard answers the on-call questions from the top of this checklist — not "everything except the answer" - [ ] Default time range is sensible (1h–6h, not 30d) ## Verify the Telemetry Instrumentation is code; it can be wrong: - [ ] Forced an error in staging → found it in the logs by correlation ID - [ ] Sent test traffic → metric series appear with expected labels and sane values - [ ] Followed one request end-to-end in the tracing UI → no broken spans - [ ] An induced failure was diagnosed from telemetry alone, without reading the source ## Pre-Launch Gate Before a feature ships to production, all of the following are true: - [ ] Structured logs flowing to the log aggregator - [ ] RED metrics visible in dashboards for every new endpoint and dependency - [ ] At least one symptom-based alert configured, with runbook, test-fired - [ ] A request can be traced across every service it touches - [ ] On-call knows where the runbooks are For launch-day monitoring sequence and rollback triggers, see the `shipping-and-launch` skill.
MIT License Copyright (c) 2025 Addy Osmani Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
MIT License Copyright (c) 2025 Addy Osmani Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Copy the text below, then paste it into your chat.