Harness garbage collection: drift checks (Phase 5)
Phase 5 of the Harness Engineering series: small continuous payments against entropy. Scheduled drift checks — dead code, outdated dependencies, doc/code desync — that don't just alert but PRODUCE an artifact: a PR, an issue, an updated quality.md. A semi-automated quality.md (a script gathers the numbers, a human writes the interpretation ~15 min/week), a doc-freshness gate, scheduling in CI, and honesty about metrics (Goodhart). Stack-agnostic, with copyable prompts.
AdvancedAI DevOps20 minCI, quality.md, knip, dependency-cruiser
1
Why GC: drift accumulates silently
One principle of harness engineering is 'small continuous payments': you remove entropy with scheduled background checks, not a 'tech-debt sprint once a quarter'. Garbage collection (GC) here isn't memory cleanup — it's the background sweeping of drift: the gap between what's written down and what's actually in the code.
Drift is dangerous because it accumulates silently. You delete a module, yet docs/architecture.md still describes it. A dependency falls three minor versions behind — nobody notices until a CVE lands. A function becomes dead code after a refactor — the linter says nothing because nothing checks it. Each gap is trivial alone; together they turn the 'source of truth' into fiction, and the agent reads that fiction as fact and hallucinates out of nowhere.
The key difference between phase 5 and ordinary linting: the check doesn't just fail with a red X. It PRODUCES an artifact a human reads or the next agentic pass picks up — a PR with a fix, an issue with a list, an updated markdown in the repo.
🟥 A check that only alerts
- Fails with a red X — and that's it
- Log noise you learn to ignore
- Nobody picks up the result
🟩 A check that PRODUCES an artifact
- A PR with a fix / an issue / an updated MD
- A human or agent picks up the artifact
- A harness that fixes itself
A usefulness test for a GC task: what happens to its result in a week? If 'it lands in a PR/issue someone closes' — good. If 'it adds a red mark to a dashboard everyone ignores' — that's not GC, it's noise.
2
What to measure → which artifact (a real class of problems, not a dashboard)
Each GC task is described as a ROLE — 'what must be checked' — and you pick the concrete tool for your stack and record it in an ADR. Where it says 'dead-code search', that's knip / ts-prune for JS/TS, vulture for Python, deadcode for Go. Where it says 'dependency hygiene', that's npm audit + npm outdated, pip-audit, bundler-audit, cargo audit.
The key selection rule: every check catches a REAL class of problems, not a metric painted on a dashboard. Dead code is future confusion for the agent (it'll start 'fixing' things nobody calls). Dependency drift is vulnerabilities and incompatibilities. Doc-freshness is source-of-truth desync. Cycles in the module graph erode the boundaries the whole architectural discipline rests on. Each item maps to a concrete output artifact.
Avoid vanity metrics: lines of code, an abstract 'tech-debt score', a number for its own sake — they're easily gamed and say nothing about whether real bugs are caught.
| Check role | Tools across stacks (examples) | Output artifact |
|---|---|---|
| Dead code | knip / ts-prune · vulture · deadcode | PR removing it / issue with a list |
| Dependency drift | npm audit/outdated · pip-audit · cargo audit | minor/patch → PR · major → issue |
| Cycles & module boundaries | madge · dependency-cruiser · import-linter | issue with the cycles found |
| Documentation freshness | git log last-modified · path-cross-check script | issue listing stale docs |
| Quality snapshot (quality.md) | aggregator of all checks above + diff | PR with an updated quality.md |
Coverage is deliberately NOT a target row in this table. If you record it, do so only as a descriptive signal inside quality.md, paired with a test-integrity barrier: the agent may not weaken or delete existing tests for a green number (checked in CI against the diff).
3
Semi-automated quality.md: the script gathers, the human interprets
The central artifact of phase 5 is a living quality.md reflecting the CURRENT state of quality across domains. It's built semi-automatically, and that's a deliberate choice, not a temporary crutch.
The script handles the mechanics: it runs coverage, the boundary linter, dead-code search, npm outdated, checks last-modified in docs/ — gathers the numbers and computes the diff against the previous report (what improved / regressed). The human handles what the script can't: interpretation. Why did domain X drop to C — deliberate debt under a deadline (there's an ADR) or unnoticed decay? That's ~15 minutes a week reading a ready-made draft.
Why not full LLM automation right now: without deep project context an LLM writes generic filler — 'coverage could be improved', 'there are outdated dependencies'. The numbers are gathered by a script, deterministically and cheaply; the judgement about what those numbers MEAN for this project stays with the human for now. Semi-automation here is the sweet spot: ~80% of the benefit for ~5% of the effort.
Script: run all checks
Numbers + diff vs last week
Draft quality.md
why so?
Human: interpretation ~15 min/week
PR with updated quality.md
# Quality Report (last updated: YYYY-MM-DD)
## Domains
| Domain | Test coverage | Architectural compliance | Documentation freshness | Overall grade |
|--------|---------------|--------------------------|-------------------------|---------------|
| ... | (signal, %) | (cycles / boundary viol.)| (stale docs count) | A / B / C / D |
Grades: A (clean), B (minor), C (significant gaps), D (broken).
Coverage is a descriptive SIGNAL, not a target. No global % mandate.
## Known violations
- <issue/PR link> — what it is, what's planned (ref docs/decisions/ or docs/plans/)
## Trends
What improved / regressed since the last report (auto-diff by the script).Grades A/B/C/D are assigned by the human, not the script. The script gives raw material (coverage 62%, 3 cycles, 5 outdated deps); translating '62% — is that an A or a C for this domain?' needs knowing which code is critical. Automating signal-to-grade is premature.
4
Doc-freshness gate: docs must not lag the code
A documentation staleness check catches the sneakiest class of drift — desync between docs/ and the code. The agent reads docs/architecture.md, AGENTS.md, docs/decisions/ as the source of truth; if it describes a deleted module or a renamed path, the agent confidently builds on a non-existent foundation.
Two implementations, from soft to hard. Soft (a background task): a script compares a doc's last-modified time with the change time of the relevant code; if the code changed but the linked doc didn't, it opens an issue listing potentially stale documents. Hard (a CI gate): a separate script parses docs/ and verifies that all paths / modules it mentions actually exist in the codebase. Delete or rename a path without updating the docs — CI goes red on that PR. This turns 'the doc should be updated' from a convention (on discipline) into a barrier (mechanically enforced), and in a harness you can only rely long-term on what has a barrier.
Put the hard gate where it's genuinely automatable: checking path/symbol existence is deterministic. Semantic freshness ('the text still describes what the code does') can't be closed with a barrier — it stays in the soft issue task and on the human.
Doc-freshness gate: from soft to hard
Soft: docs older than linked code → issue with a list (background task)
Hard: paths/modules mentioned in docs really exist → else CI goes red
Path/symbol existence is deterministic — fit for a barrier
Semantic freshness can't be a barrier — stays an issue + the human
A hard gate on text semantics ('sounds current') — false confidence
'Machine over text': the rule 'update the doc when you change a module' in AGENTS.md lasts exactly until the first deadline. The same invariant as a CI gate on path existence is always honored — because without it, nothing merges.
5
Scheduling in CI + honesty (Goodhart)
GC tasks live in the background scheduler chosen at adoption time: GitHub Actions, GitLab CI, Jenkins, n8n, server cron. The role is one — 'regularly run the script and drop an artifact somewhere the team can reach'. A reasonable default schedule is weekly: often enough that drift can't pile up, rare enough that PRs and issues don't become noise. If the project has no scheduler, phase 5 requires first standing up a minimal one (enough to run a script on a schedule) and recording an ADR.
Honesty at the end. This rollout does NOT validate functional correctness — that the code does what the user needs is still checked by a human and product tests. GC ensures architectural integrity and maintainability, not 'feature correctness'.
And the key point about metrics — Goodhart: once a metric becomes a target, it stops being a good metric. Don't introduce a global '100% coverage everywhere' mandate: the agent will immediately write junk tests for the number. Want a gate — only on new/changed code, and not as a mandate but as a reasonable default of your choosing. What truly matters isn't the number but the test-integrity policy: the agent doesn't edit or weaken existing tests (checked in CI against the diff). Every GC check should catch a real class of problems, not please a dashboard.
# Weekly GC schedule (role: background scheduler — GH Actions / GitLab CI / cron)
# Each job PRODUCES an artifact (PR / issue / updated MD), never just alerts.
on:
schedule:
- cron: '0 6 * * 1' # weekly — Monday 06:00; a default, not a law
jobs:
quality-refresh: # → PR updating docs/quality.md (numbers + diff)
dead-code: # knip / vulture / deadcode → PR/issue
dep-hygiene: # audit + outdated → minor/patch PR, major issue
doc-freshness: # stale docs + path-existence → issue (+ CI gate on PRs)
# Honesty:
# - GC ensures architectural integrity, NOT functional correctness.
# - No global coverage mandate (Goodhart). Coverage = signal, not target.
# - Test-integrity barrier: agent must not weaken/delete existing tests.If nobody closes a GC task's artifacts for weeks — that's not a reason to crank up alerts but a sign the task catches a non-problem or the schedule is too frequent. A dead GC check itself becomes drift.
6
What's next: the series map
Garbage collection is phase 5 of the rollout plan: background drift checks that turn entropy from a 'tech-debt sprint once a quarter' into small continuous payments. You assembled four scheduled tasks (quality refresh, dead code, dependency hygiene, doc-freshness), each PRODUCING an artifact, with quality.md kept semi-automatic.
The next phase in the series logic is 5.5, the lessons-ledger: corrections as code. It rides the scheduler already chosen here (the lessons maintain cycle is a background task of the same class as phase 5) and closes what GC can't catch: recurring NON-deterministic agent mistakes (taste, contextual choice). Its dedicated recipe is Phase 5.5, linked in the series map below. For the full map of all rollout phases and how they connect, head to the series hub.
| Phase | What | Recipe |
|---|---|---|
| Hub | Harness Engineering series map | harness-engineering — done ✓ |
| 5 | Garbage collection: drift → artifacts | this recipe ✓ |
| 5.5 | Lessons-ledger: corrections as code | lessons-ledger — done ✓ |
GC and the lessons-ledger are background tasks of the same class on the same scheduler, but catch different things: GC catches deterministic drift (dead code, deps, docs), the ledger catches non-deterministic taste mistakes. Don't conflate them: a computational lesson belongs in a barrier (phase 3), not piled up as text.
Result
Phase 5 is assembled: drift is removed through small continuous payments, not a quarterly tech-debt sprint. You understand how to describe GC tasks as roles with stack-agnostic tools (knip/vulture, npm audit/pip-audit, madge), why each check must PRODUCE an artifact (PR/issue/quality.md) rather than just alert, how to keep quality.md semi-automatic (the script gathers numbers, the human writes interpretation ~15 min/week), and how to set up a doc-freshness gate from a soft issue to a hard CI barrier on path existence. And above all — honesty: GC ensures architectural integrity, not functional correctness, and test-integrity (the agent doesn't weaken tests) matters more than any coverage percentage. Next: phase 5.5, the lessons-ledger.