sgeboers/motief
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

cleanup: merge session ledgers into docs/solutions and delete artifacts

Browse Source 
- Remove stale thoughts/ledgers/ and thoughts/shared/ artifacts
- Fix .gitignore duplicate .worktrees entry
- Move pyright to [dependency-groups] dev
- Replace hardcoded blog correlation with reproducible metric reference
- Add docs: verify-session-artifacts, fusion-vector-dimensions,
  working-tree-hygiene
- Update blog-numbers-from-pipeline-outputs with correlation example
This commit is contained in:
sgeboers
2026-04-30 23:24:43 +02:00
parent 5f9e8965cd
commit 375955dbc4
61 changed files with 480 additions and 9120 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/solutions/best-practices/blog-numbers-from-pipeline-outputs-2026-04-16.md
+22 -2
View File
@@ -1,6 +1,7 @@
---
title: Always Derive Blog Numbers from Pipeline Outputs, Not Memory
date: 2026-04-16
last_updated: 2026-04-24
category: docs/solutions/best-practices
module: documentation
problem_type: best_practice
@@ -8,9 +9,9 @@ component: documentation
severity: medium
applies_when:
- Writing or updating a data-driven blog post
- Adding EVR percentages, vote counts, or any quantitative claims
- Adding EVR percentages, vote counts, correlation coefficients, or any quantitative claims
- Referencing pipeline components (embeddings, fusion, similarity) in public-facing docs
tags: [blog, pipeline, evr, svd, canonical-outputs, data-driven-docs]
tags: [blog, pipeline, evr, svd, canonical-outputs, data-driven-docs, reproducibility, correlation]
---
# Always Derive Blog Numbers from Pipeline Outputs, Not Memory
@@ -29,6 +30,7 @@ The political compass blog post was written with hardcoded numbers (EVR ~32%/~21
| Vote/motion counts | `SELECT COUNT(*) FROM motions / mp_votes` via `data/motions.db` |
| Window count | `analysis.political_axis` — count of aligned windows |
| Party agreement | `analysis.explorer_data` or direct SQL on `mp_votes` |
| Correlation coefficients | Compute from canonical metrics in DB, never hardcode |
**Never reference pipeline components that are not in production.** If `fused_embeddings` rows exist in the DB but the fusion pipeline is not yet in active use, do not describe it as part of the current workflow in blog copy.
@@ -87,6 +89,24 @@ sql = """
"""
```
**Correlation between voting extremity and policy extremity:**
- ❌ **Before (hardcoded, unverifiable):**
```html
<p>De correlatie tussen beide maten is r = -0.011 — ze meten totaal verschillende dingen.</p>
```
Problem: No script, query, or function reproduces this number. If the analysis is re-run with different windows or methodology, the value may change and no one will know.
- ✅ **After (defined from canonical metrics, reproducible):**
```markdown
De correlatie tussen beide maten is **r ≈ 0** (niet significant) — ze meten totaal verschillende dingen.
*Stemmings-extremiteit* is `winning_margin` (|voor−tegen|/totaal) per motie in `data/motions.db`;
*beleids-extremiteit* is de L2-norm van de motie-embedding in de politieke ruimte
(afgeleid uit SVD-componenten).
```
The metrics are defined canonically. Anyone can recompute the correlation from the database.
## Related
- `docs/solutions/best-practices/svd-labels-voting-patterns-not-semantics.md` — companion guidance on keeping SVD axis *labels* aligned with voting data rather than semantic assumptions
docs/solutions/best-practices/fusion-vector-dimension-consistency-2026-03-23.md
+95
View File
@@ -0,0 +1,95 @@
---
title: "Fusion pipeline: vector dimension inconsistency causes padding"
date: 2026-03-23
module: pipeline
problem_type: best_practice
component: fusion-pipeline
severity: low
tags:
- fusion
- embeddings
- vector-dimensions
- pipeline
- data-quality
---
# Fusion Pipeline: Vector Dimension Inconsistency Causes Padding
## Context
During a fusion + similarity pipeline run (2026-03-23), several windows had inconsistent vector dimensions. The pipeline padded vectors to a common dimension to allow fusion and similarity processing, logging warnings per affected window.
## Pipeline Run Summary
| Metric | Value |
|--------|-------|
| Start | 2026-03-23T15:30:00Z |
| End | 2026-03-23T16:47:04Z |
| Duration | 1h 17m 4s |
| Embeddings processed | 28,172 |
| Fused embeddings | 40,524 |
| Similarity rows | 405,216 |
## Per-Window Warnings
| Window | Inserted | Warnings | Issue |
|--------|----------|----------|-------|
| win-002 | 2,048 | 1 | Padded vectors due to dim mismatch |
| win-003 | 4,096 | 2 | Padded vectors due to dim mismatch |
| win-005 | 15,344 | 3 | Padded vectors due to dim mismatch |
**Note:** win-001 and win-004 had no warnings (consistent dimensions).
## Why This Happens
Vector dimensions can become inconsistent across windows when:
1. **Embedding model changes** between window processing runs
2. **Text truncation** produces different effective lengths
3. **Pipeline restarts** after partial failures create mixed batches
4. **Different window sizes** (annual vs quarterly) aggregate different numbers of motions
## Impact
- **Fused embeddings are padded**, not truncated — data is preserved but with zero-padding
- **Similarity scores** may be slightly affected for padded dimensions
- **No data loss**, but quality degradation in affected windows
## Prevention
1. **Validate dimensions before fusion**
```python
# Before calling fusion, assert all vectors have the same dimension
dims = {len(v) for v in window_vectors}
assert len(dims) == 1, f"Dimension mismatch: {dims}"
```
2. **Re-embed with consistent model/settings** if dimensions differ
- Don't mix embeddings from different model versions
- Re-run the full embedding pipeline if the model changes
3. **Window-level dimension checks** in the pipeline:
```python
# In pipeline/fusion.py or equivalent
for window_id, vectors in window_vectors.items():
dim = len(vectors[0])
if not all(len(v) == dim for v in vectors):
raise ValueError(f"Window {window_id}: inconsistent vector dimensions")
```
4. **QA sampling after fusion**
- Perform sample similarity lookups across N=20-50 items
- Validate fused vectors against source embeddings
- Check for anomalies in similarity scores for affected windows
## When to Apply
- Before running the fusion pipeline
- After re-running the embedding pipeline with new model/settings
- When adding new windows to an existing fused embedding set
- During QA of similarity cache results
## Related
- `docs/solutions/best-practices/blog-numbers-from-pipeline-outputs-2026-04-16.md` — Canonical pipeline output sources
- `pipeline/fusion.py` — Fusion pipeline implementation
- `data/motions.db` — `fused_embeddings` and `similarity_cache` tables
docs/solutions/best-practices/verify-transient-artifacts-against-canonical-sources.md
+113
View File
@@ -0,0 +1,113 @@
---
title: Verify Transient Session Artifacts Against Canonical Sources Before Compounding
date: "2026-04-24"
category: docs/solutions/best-practices
module: documentation
problem_type: best_practice
component: documentation
severity: medium
applies_when:
- Merging session ledgers or other transient artifacts into durable documentation
- Creating or updating docs/solutions/ entries from agent session outputs
- Extracting code constants, labels, or configurations from non-canonical files
- Compounding knowledge from temporary workspace artifacts
tags:
- compound-documentation
- canonical-sources
- session-ledgers
- svd-labels
- verification
- transient-artifacts
---
# Verify Transient Session Artifacts Against Canonical Sources Before Compounding
## Context
The `ce-compound` workflow involves merging session ledgers from `thoughts/ledgers/` into durable documentation under `docs/solutions/`. During one such session, an agent was instructed to create a compounding doc based on a ledger file. The agent extracted SVD component labels directly from the ledger and wrote them into a new `docs/solutions/` file.
The problem: the labels in the ledger were outdated. They had since been updated in the canonical source (`analysis/config.py` `SVD_THEMES`). The agent did not cross-check the ledger content against the canonical codebase before creating the durable doc. The user had to manually catch the discrepancy, instruct the agent to verify against canonical sources, and the inaccurate doc was deleted.
## Guidance
**Always cross-check transient artifacts against canonical codebase sources before creating or updating compounding documentation.**
When merging session ledgers or any transient artifact into `docs/solutions/`:
1. **Identify the canonical source for every factual claim**
- Code constants → check the defining module (e.g., `analysis/config.py` for SVD labels)
- Data figures → check the pipeline output or database
- Configuration → check the committed config file, not session notes
2. **Do not treat ledger content as ground truth**
- Ledgers capture agent reasoning at a point in time
- Code evolves after the ledger is written
- A ledger is a memory aid, not a canonical reference
3. **Diff the artifact against the canonical source**
- Read the current canonical file explicitly
- Compare values, labels, constants, or conclusions
- If they differ, use the canonical source and note the update
4. **Flag discrepancies instead of silently using stale data**
- If the ledger contradicts the codebase, document the divergence
- Explain when and why the canonical source changed
- Do not propagate outdated information into durable docs
## Why This Matters
Compounding documentation is meant to reduce future cognitive load. If it embeds stale or inaccurate information:
- **Future agents (and humans) will trust it as truth.** `docs/solutions/` is explicitly referenced in `AGENTS.md` as a source of guidance. An inaccurate doc becomes a source of repeated errors.
- **Outdated labels or constants propagate downstream.** In this case, outdated SVD labels would have misled every future agent working on SVD analysis, visualization, or blog updates.
- **Correcting a published doc costs more than verifying before writing.** Deleting and rewriting a doc is cheap; discovering and fixing a stale doc months later requires archaeology.
## When to Apply
Apply this guidance whenever you are:
- Creating a new `docs/solutions/` entry from a session ledger, conversation log, or agent memory
- Updating an existing doc with insights from a transient artifact
- Extracting code snippets, constants, labels, or configurations from any file that is not the canonical definition
- Summarizing a debugging session where code was modified — the final committed code is canonical, not the session narrative
## Examples
### What Happened (Incorrect)
An agent read a session ledger containing SVD component labels and wrote them directly into a new `docs/solutions/` file without checking `analysis/config.py`:
```
# ❌ INCORRECT: labels taken directly from stale ledger
Component 1: "Sociale zekerheid vs economische liberalisering"
```
The canonical source (`analysis/config.py` `SVD_THEMES`) had since been updated to reflect voting-pattern-based labels. The doc was inaccurate and had to be deleted.
### What Should Have Happened (Correct)
```
# ✅ CORRECT: verify ledger claims against canonical source
1. Read analysis/config.py SVD_THEMES
2. Compare ledger labels with current SVD_THEMES values
3. Use the canonical labels from config.py
4. If the ledger contained useful context (e.g., reasoning about why labels changed),
preserve that narrative but anchor all factual claims to the canonical source
```
### Verification Pattern
```python
# When documenting SVD labels, always read the canonical config
from analysis.config import SVD_THEMES
for comp_num, theme in SVD_THEMES.items():
print(f"Component {comp_num}: {theme['label']}")
# Use these values in the doc, not ledger-cached values
```
## Related
- `docs/solutions/best-practices/svd-labels-voting-patterns-not-semantics.md` — how SVD labels should be derived from voting patterns
- `docs/solutions/best-practices/blog-numbers-from-pipeline-outputs-2026-04-16.md` — deriving quantitative claims from canonical pipeline outputs
- `analysis/config.py` — canonical source for SVD themes and other constants
docs/solutions/best-practices/working-tree-hygiene-dependency-groups-and-gitignore-2026-04-24.md
+118
View File
@@ -0,0 +1,118 @@
---
title: Working Tree Hygiene — Dependency Groups and Gitignore
date: 2026-04-24
category: docs/solutions/best-practices
module: development_workflow
problem_type: best_practice
component: development_workflow
severity: low
applies_when:
- Reviewing uncommitted changes before committing
- Adding new dependencies to pyproject.toml
- Updating .gitignore with new ignore patterns
tags: [dependencies, pyproject, gitignore, hygiene, code-review, dev-tools]
---
# Working Tree Hygiene — Dependency Groups and Gitignore
## Context
A code review of uncommitted changes on `main` caught three preventable hygiene issues:
1. `pyright` (a static type checker) was added to `[project] dependencies` in `pyproject.toml` instead of `[dependency-groups] dev`
2. `.gitignore` contained a duplicate `.worktrees` entry
3. A blog post included a hardcoded correlation coefficient with no reproducible source (documented separately in `blog-numbers-from-pipeline-outputs`)
All three were caught before commit, but they illustrate a pattern: small working tree cleanups accumulate friction when not reviewed systematically.
## Guidance
### Dependency classification
When adding a package to `pyproject.toml`, ask: **does this run in production?**
| If... | Put it in... |
|-------|-------------|
| The app imports it at runtime | `[project] dependencies` |
| It is a type checker, test runner, linter, or dev server | `[dependency-groups] dev` |
| It is only used in build scripts or CI | `[dependency-groups] dev` |
**Concrete check:** search the codebase for `import <package>` or `from <package>`. If it only appears in `tests/`, `scripts/`, or type stubs, it belongs in `dev`.
### Gitignore hygiene
Before committing a `.gitignore` change, run:
```bash
sort .gitignore | uniq -d
```
If anything prints, you have duplicates. Remove them.
Also check that your new entry does not overlap with an existing pattern:
- `.worktrees/` and `.worktrees` are redundant — keep the slash form for directories
- `data/*.json` already covers `data/motions.json` — do not add the specific file
### Pre-commit audit checklist
For every set of uncommitted changes:
1. **Dependencies**: Any new packages in the right group?
2. **Gitignore**: Any duplicates or redundant patterns?
3. **Blog/docs**: Any hardcoded numbers without canonical sources? (see `blog-numbers-from-pipeline-outputs`)
4. **Config**: Any secrets or local paths committed by accident?
## Why This Matters
These issues are individually trivial, but together they create a "broken windows" effect. A `pyproject.toml` with dev tools in runtime dependencies signals that the project does not distinguish between production and development concerns. Duplicate `.gitignore` entries suggest the file is append-only and never reviewed. Small hygiene lapses compound into larger maintainability debt.
The fix is cheap: a 30-second scan of the diff before committing prevents all of them.
## When to Apply
- Before every commit that touches `pyproject.toml`, `.gitignore`, or `uv.lock`
- When onboarding a new dependency
- During code review of any PR that adds build tools, test frameworks, or local config
## Examples
**Dependency misclassification:**
```toml
# ❌ Before
[project]
dependencies = [
"duckdb>=1.3.2",
"pyright>=1.1.408", # dev tool in runtime deps
]
# ✅ After
[project]
dependencies = [
"duckdb>=1.3.2",
]
[dependency-groups]
dev = [
"pytest>=9.0.2",
"pyright>=1.1.408",
]
```
**Gitignore duplicate:**
```diff
# Worktrees
.worktrees/
# Generated analysis files
thoughts/explorer/*.json
-
- # Stray temp files
- .worktrees # ← duplicate, remove
```
## Related
- `docs/solutions/best-practices/blog-numbers-from-pipeline-outputs-2026-04-16.md` — companion guidance on keeping quantitative claims reproducible
- `docs/solutions/workflow-issues/verify-session-artifacts-against-canonical-sources-2026-04-24.md` — same verification principle applied to session artifacts