From 6a9f2eb2d6f318e89faaff03a23ed84d8862c8e0 Mon Sep 17 00:00:00 2001 From: notbucki <103531753+notbucki@users.noreply.github.com> Date: Fri, 4 Sep 2026 09:51:20 +0200 Subject: [PATCH] docs(observability): name the entry point when several write to one log MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A correlation ID identifies a run, not the code path that started it. When a scheduler, a replay endpoint and a manual CLI run all reach the same job and write into one sink, the lines are interchangeable — attributing one falls back to elimination against external records (scheduler history, process table, deploy log) that only happen to still exist. Adds a paragraph plus a short example to "3. Structured logging" — stamp the entry point where the run starts, next to the correlation ID, and propagate both across queue metadata and outbound calls — and one line each to Red Flags, Verification, and references/observability-checklist.md. The field is `entryPoint` rather than `source`: Elastic Common Schema reserves `source.*` for network fields (source.ip, source.port, …), so a scalar `source` collides with that mapping and can get documents rejected. No new heading, no new directory, frontmatter unchanged. Co-Authored-By: Claude Opus 5 (1M context) --- references/observability-checklist.md | 1 + .../observability-and-instrumentation/SKILL.md | 17 +++++++++++++++++ 2 files changed, 18 insertions(+) diff --git a/references/observability-checklist.md b/references/observability-checklist.md index fff9f584..ad40d7aa 100644 --- a/references/observability-checklist.md +++ b/references/observability-checklist.md @@ -26,6 +26,7 @@ Telemetry without a question is noise. Before instrumenting anything: - [ ] Logs are structured (JSON) with stable event names — not free-form strings - [ ] Every log line carries a correlation/request ID, generated or accepted at the system boundary - [ ] Correlation ID is propagated on every outbound call and async boundary (HTTP headers, queue metadata) +- [ ] Any log stream written by more than one entry point (scheduler, replay endpoint, manual run) carries an entry-point field, set where the run starts and propagated alongside the correlation ID - [ ] Log levels are consistent: `error` = invariant broken, someone may act; `warn` = degraded but handled; `info` = significant business event; `debug` = off in production - [ ] No secrets, tokens, passwords, or unredacted PII in any log line (hard rule from `security-and-hardening`) - [ ] Fields are allowlisted — no whole request/response bodies, no auth headers diff --git a/skills/observability-and-instrumentation/SKILL.md b/skills/observability-and-instrumentation/SKILL.md index 7d56593e..87a848a8 100644 --- a/skills/observability-and-instrumentation/SKILL.md +++ b/skills/observability-and-instrumentation/SKILL.md @@ -88,6 +88,21 @@ app.use((req, res, next) => { }); ``` +**When several entry points write to one log, name the entry point.** A correlation ID identifies a run; it does not say which code path started it. The same job reached by a scheduler, by a replay endpoint, and by a manual CLI run produces interchangeable lines in one sink, so attributing a line falls back to elimination — cross-reading the scheduler's history, the process table, a deploy log — and that argument holds only as long as those external records happen to still exist. Stamp the entry point where the run starts, next to the correlation ID, and propagate both the same way: + +```typescript +// One helper for every entry point: the run's own logger carries both fields. +// `entryPoint`, not `source` — ECS reserves `source.*` for network fields. +export const runLog = (entryPoint: 'scheduler' | 'replay_endpoint' | 'cli', runId: string) => + logger.child({ entryPoint, requestId: runId }); + +// scheduler tick -> runLog('scheduler', crypto.randomUUID()) +// POST /jobs/:id/replay -> runLog('replay_endpoint', req.id) +// CLI invocation -> runLog('cli', process.env.RUN_ID ?? crypto.randomUUID()) +``` + +Both fields have to cross the same boundaries as the correlation ID — queue metadata, HTTP headers — or a worker re-derives the entry point and guesses. A field that merely correlates with an entry point is a hint, not an attribution: anything that can invoke the job can reproduce it. + **Never log secrets, tokens, passwords, or full PII.** This is a hard rule from the `security-and-hardening` skill — telemetry pipelines are a classic data-leak path. Allowlist fields; don't log whole request bodies. ### 4. Metrics @@ -180,6 +195,7 @@ Instrumentation is code; it can be wrong. Before calling the work done, trigger - A feature PR with retries, queues, or external calls and zero new telemetry - Log lines built by string interpolation instead of structured fields - No correlation/request ID — each log line is an orphan +- One log stream fed by a scheduler, a webhook, and manual runs, with no field naming which one produced the line - Metrics labeled with user IDs, raw URLs, or error message text (cardinality bomb) - Latency tracked as an average with no percentiles - Alerts that fire daily and get acknowledged without action @@ -193,6 +209,7 @@ After instrumenting a feature, confirm: - [ ] The on-call questions for this feature are written down, and each signal maps to one - [ ] All log output is structured (JSON), with stable event names and a correlation ID on every line +- [ ] Every log sink written by more than one entry point carries an entry-point field, set where the run starts and propagated with the correlation ID rather than inferred downstream - [ ] No secrets, tokens, or unredacted PII in any log line (spot-check actual output) - [ ] RED metrics exist for every new endpoint and every external dependency, with bounded label sets - [ ] Latency is a histogram; p95/p99 are queryable