sgeboers/motief
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

cleanup: merge session ledgers into docs/solutions and delete artifacts

Browse Source 
- Remove stale thoughts/ledgers/ and thoughts/shared/ artifacts
- Fix .gitignore duplicate .worktrees entry
- Move pyright to [dependency-groups] dev
- Replace hardcoded blog correlation with reproducible metric reference
- Add docs: verify-session-artifacts, fusion-vector-dimensions,
  working-tree-hygiene
- Update blog-numbers-from-pipeline-outputs with correlation example
This commit is contained in:
sgeboers
2026-04-30 23:24:43 +02:00
parent 5f9e8965cd
commit 375955dbc4
61 changed files with 480 additions and 9120 deletions
Download Patch File Download Diff File Expand all files Collapse all files
thoughts/ledgers/CONTINUITY_continuity-ledger.md
-55
View File
@@ -1,55 +0,0 @@
# format: <line>#<hash>#<anchor>|<content>
# use refs exactly as shown in hashline edit/patch tools
#HL REV:C4181A89
#HL 1#AD2#963|# Session: continuity-ledger
#HL 2#625#EA0|Updated: 2026-03-31T12:00:00Z
#HL 3#DA3#29F|
#HL 4#3B8#9B2|## Goal
#HL 5#49D#054|Preserve the essential session context and state for the stemwijzer project so work can resume seamlessly after context clears.
#HL 6#DA3#B25|
#HL 7#3CD#7E4|## Constraints
#HL 8#343#88A|- Keep the ledger concise; only essential information is recorded.
#HL 9#C8A#AD0|- Focus on WHAT and WHY, not HOW.
#HL 10#7DD#B90|- Mark uncertain information explicitly as UNCONFIRMED.
#HL 11#04E#272|- Include current git branch and key file paths.
#HL 12#CCD#F02|- Never store secrets or values from .env files.
#HL 13#DA3#A4D|
#HL 14#E5A#9FA|## Progress
#HL 15#E30#F0C|### Done
#HL 16#829#1C2|- [x] Determine need for a continuity ledger and file location.
#HL 17#906#394|- [x] Create and add this continuity ledger file to the repository (this file). UNCONFIRMED: whether committed/pushed to remote.
#HL 18#B2A#001|- [x] Monitor and merge subsequent ledger updates when provided (inspected other CONTINUITY_* ledgers on 2026-03-31T12:00:00Z). (UNCONFIRMED: whether merged/committed)
#HL 19#DA3#387|
#HL 20#AC7#256|### In Progress
#HL 21#405#F17|- [ ] Short QA: sample similarity lookups (N=20-50) to validate fused vectors (see CONTINUITY_stemwijzer.md). Estimated effort: 30–60 minutes. (UNCONFIRMED assignment)
#HL 22#DA3#77C|
#HL 23#8B6#828|### Blocked
#HL 24#2A1#2DC|- None
#HL 25#DA3#C2F|
#HL 26#7A9#773|## Key Decisions
#HL 27#20F#D99|- **Store concise session state in thoughts/ledgers/**: keeps context portable and easy to merge.
#HL 28#4B6#2BB|- **Minimal fields only (goal, constraints, progress, decisions, next steps, file ops, context)**: reduces noise and maintenance.
#HL 29#DA3#F5B|
#HL 30#62A#B91|## Next Steps
#HL 31#22B#0CD|1. Provide previous ledger content on subsequent updates so merges preserve full history.
#HL 32#E49#DA8|2. Use this ledger as the single source for resuming interrupted sessions; update "In Progress" items as work proceeds.
#HL 33#4B7#4A5|3. Coordinate short QA on recent fusion/similarity run (see CONTINUITY_stemwijzer.md) in a separate session if needed.
#HL 34#DA3#1D0|
#HL 35#1CA#DCD|## File Operations
#HL 36#0F3#F62|### Read
#HL 37#256#5B3|- `README.md`
#HL 38#A0D#268|- `thoughts/ledgers/CONTINUITY_stemwijzer.md` (INSPECTED)
#HL 39#AC9#FE0|- `thoughts/ledgers/CONTINUITY_fusion_similarity_run.md` (INSPECTED)
#HL 40#DA3#081|
#HL 41#455#EBF|### Modified
#HL 42#3F4#1DD|- `thoughts/ledgers/CONTINUITY_continuity-ledger.md` (this file)
#HL 43#DA3#C78|
#HL 44#2BA#352|## Critical Context
#HL 45#112#C18|- Repository root: /home/sgeboers/Projects/stemwijzer
#HL 46#9CD#0EE|- Current git branch: `main` (UNCONFIRMED: local workspace branch)
#HL 47#DEF#90F|- Other existing continuity ledgers: `CONTINUITY_stemwijzer.md`, `CONTINUITY_fusion_similarity_run.md`
#HL 48#2D0#620|- UNCONFIRMED: whether this file has been committed/pushed to remote.
#HL 49#DA3#373|
#HL 50#7C4#A51|## Working Set
#HL 51#381#266|- Branch: `main`
#HL 52#BD8#51B|- Key files: `README.md`, `thoughts/ledgers/CONTINUITY_continuity-ledger.md`, `thoughts/ledgers/CONTINUITY_stemwijzer.md`, `thoughts/ledgers/CONTINUITY_fusion_similarity_run.md`
thoughts/ledgers/CONTINUITY_fusion_similarity_run.md
-50
View File
@@ -1,50 +0,0 @@
# Session: fusion_similarity_run
Updated: 2026-03-23T16:47:04Z
## Goal
Record outcomes and metrics from the completed fusion+similarity run so work can resume and a short QA can be executed.
## Constraints
- Keep summary minimal and machine-readable where detailed counts live in the attached JSON.
- Do not expose secrets.
## Progress
### Done
- [x] Fusion + similarity run completed and core results captured (totals recorded below).
### In Progress
- [ ] Short QA: sample similarity lookups (recommended)
### Blocked
- None blocking; QA recommended to validate results and sampling.
## Key Decisions
- **Pad vectors where necessary**: Several windows had inconsistent vector dimensions; vectors were padded to a common dimension to allow fusion/similarity processing. Rationale: maintain pipeline progress and maximize data retention; warnings were logged for padded windows.
## Next Steps
1. Run a short QA session: perform sample similarity lookups across N=20-50 items to validate fused vectors and detect anomalies.
2. Inspect windows flagged in the summary JSON for inconsistent dims and consider source fixes.
3. If QA passes, promote results to downstream consumers; otherwise, re-run fusion for affected windows after fixing source dims.
## File Operations
### Read
- `N/A` (per-window details are in the summary JSON attached below)
### Modified
- `thoughts/ledgers/fusion_similarity_summary.json`
- `thoughts/ledgers/CONTINUITY_fusion_similarity_run.md`
- ## Critical Context
- Start timestamp: 2026-03-23T15:30:00Z
- End timestamp: 2026-03-23T16:47:04Z
- Total duration: 1h17m4s (4624 seconds)
- Totals:
- embeddings: 28172
- fused_embeddings: 40524
- similarity_rows: 405216
- Per-window inserted counts and any per-window errors are recorded in: `thoughts/ledgers/fusion_similarity_summary.json` (JSON summary attached to repo). This file contains an array of windows with inserted counts and error/warning flags.
- Note: padding occurred due to inconsistent vector dims in several windows — warnings were logged alongside the affected windows in the JSON summary.
## Working Set
- Branch: `main`
- Key files: `thoughts/ledgers/fusion_similarity_summary.json`, `thoughts/ledgers/CONTINUITY_fusion_similarity_run.md`
thoughts/ledgers/CONTINUITY_ses_2a6e.md
-85
View File
@@ -1,85 +0,0 @@
---
session: ses_2a6e
updated: 2026-04-04T15:34:15.344Z
---
# Session Summary
## Goal
Analyze and document how the most important motions are defined and ranked in the Stemwijzer codebase, focusing on importance criteria, selection mechanisms, metadata, key files, and user interaction patterns.
## Constraints & Preferences
- Provide detailed findings with file paths and line numbers
- Focus on code analysis without making changes
- Document the complete motion ranking and display system
## Progress
### Done
- [x] Analyzed motion importance criteria (controversy_score, SVD scores, entropy-based discrimination)
- [x] Documented motion selection mechanisms for SVD display, Political Compass, quiz, and similarity search
- [x] Mapped database schema for motions, mp_votes, svd_vectors, similarity_cache tables
- [x] Identified key files and their roles in motion handling
- [x] Documented user interaction flows for SVD components tab, MP quiz, and motion browser
- [x] Cataloged SVD_THEMES dictionary with all 10 component labels and explanations
### In Progress
- (none - analysis complete)
### Blocked
- (none)
## Key Decisions
- **Analysis-only session**: No code modifications were requested or made; this was purely investigative work to understand the existing motion ranking system.
## Next Steps
1. Awaiting further instructions from user on what to do with the analysis (e.g., implement changes, add features, optimize)
## Critical Context
### Motion Importance Metrics
1. **Controversy Score**: `1 - winning_margin` (0.5 = even split, higher = more controversial)
2. **SVD Component Scores**: Absolute projection on each SVD component axis
3. **Entropy Score**: Shannon entropy of vote distribution (for quiz discrimination)
### Motion Selection Strategies
- **SVD Display**: Top 10 per component (5 positive pole, 5 negative pole)
- **Political Compass**: Top 5 at each pole for axis labeling
- **Quiz Seed**: Top 8 controversial motions with individual MP votes
- **Quiz Discriminating**: Entropy-ranked motions that best split candidate MPs
### Database Schema
```sql
motions: id, title, description, date, policy_area, voting_results (JSON),
winning_margin, controversy_score, layman_explanation, body_text, url
mp_votes: motion_id, mp_name, party, vote, date
svd_vectors: window_id, entity_type, entity_id, vector (JSON 50-dim)
similarity_cache: source_motion_id, target_motion_id, score, vector_type, window_id
```
### Key Functions
| Function | Location | Purpose |
|----------|----------|---------|
| `get_motions_with_individual_votes()` | database.py:660-692 | Get controversial motions with MP votes |
| `choose_discriminating_motions()` | database.py:817-903 | Entropy-based motion selection |
| `_top_motion_ids()` | axis_classifier.py:274-295 | Top N motions per axis pole |
| `build_svd_components_tab()` | explorer.py:3081-3497 | UI for SVD motion display |
| `build_mp_quiz_tab()` | explorer.py:3499-3724 | MP quiz with adaptive motion selection |
### SVD Themes Location
`explorer.py:432-762` - Dictionary `SVD_THEMES` contains labels, explanations, and party poles for components 1-10.
## File Operations
### Read
- `/home/sgeboers/Projects/stemwijzer/analysis/axis_classifier.py`
- `/home/sgeboers/Projects/stemwijzer/database.py`
- `/home/sgeboers/Projects/stemwijzer/explorer.py` (partial reads at offsets 1860, 3050, 3400)
- `/home/sgeboers/Projects/stemwijzer/pages/1_Stemwijzer.py`
- `/home/sgeboers/Projects/stemwijzer/scripts/generate_svd_json.py`
- `/home/sgeboers/Projects/stemwijzer/similarity/lookup.py`
- `/home/sgeboers/Projects/stemwijzer/src/types/motion_types.py`
- `/home/sgeboers/Projects/stemwijzer/migrations/2026_03_21__create_mp_metadata.sql`
- `/home/sgeboers/Projects/stemwijzer/migrations/2026_03_21__create_mp_votes.sql`
- `/home/sgeboers/Projects/stemwijzer/migrations/2026_03_21__create_svd_vectors.sql`
- `/home/sgeboers/Projects/stemwijzer/migrations/2026-03-22-add-similarity-cache.sql`
### Modified
- (none)
thoughts/ledgers/CONTINUITY_ses_2b07.md
-56
View File
@@ -1,56 +0,0 @@
---
session: ses_2b07
updated: 2026-04-02T19:01:27.654Z
---
# Session Summary
## Goal
Execute Task 2 from the SVD Label Unification implementation plan: refactor explorer.py to export SVD_THEMES at module level and update analysis/svd_labels.py to import it properly.
## Constraints & Preferences
- Follow TDD principles: run tests before/after changes
- Make minimal changes to accomplish the task
- Preserve all existing SVD_THEMES data (10 components with labels, explanations, poles, flip settings)
- Ensure no circular import issues between explorer.py and analysis/svd_labels.py
## Progress
### Done
- [x] Ran baseline tests (4 tests passed in tests/test_svd_labels.py)
- [x] Moved SVD_THEMES dict from inside `build_svd_components_tab` function (line ~2639) to module level in explorer.py (after PARTY_COLOURS, line 434)
- [x] Removed duplicate SVD_THEMES definition from inside `build_svd_components_tab` function
- [x] Updated `_get_svd_themes()` function in analysis/svd_labels.py to import directly from explorer module instead of using complex importlib.util fallback
- [x] Verified all 4 tests still pass after changes
- [x] Confirmed SVD_THEMES is now accessible at module level in explorer.py for external import
### In Progress
- [ ] Commit the changes (changes staged but not yet committed)
### Blocked
- (none)
## Key Decisions
- **Import method**: Use direct `import explorer` and access `explorer.SVD_THEMES` instead of importlib.util machinery. Rationale: Now that SVD_THEMES is at module level, the direct import is clean and the lazy runtime import in `_get_svd_themes()` prevents circular dependencies at module load time.
- **Module placement**: Placed SVD_THEMES after PARTY_COLOURS (line 434) to keep constants together near the top of the file. Rationale: This keeps the canonical source of truth visible and maintains logical grouping with other module-level constants.
## Next Steps
1. Run full test suite to verify no regressions: `uv run pytest tests/ -v`
2. Commit the changes: `git add explorer.py analysis/svd_labels.py && git commit -m "refactor: move SVD_THEMES to module level for import"`
3. Proceed to Task 3: Update axis_classifier.py to use svd_labels module
## Critical Context
- SVD_THEMES now defined at explorer.py line 434 with full type annotation `dict[int, dict[str, str]]`
- SVD_THEMES contains 10 components (1-indexed) with keys: label, explanation, positive_pole, negative_pole, flip
- Function `_get_svd_themes()` in analysis/svd_labels.py now uses simple import pattern with global cache `_svd_themes_cache`
- The function references in explorer.py at lines 2691 and 2719 (`SVD_THEMES.get()`) continue to work unchanged since they now reference the module-level variable
- All 4 tests in tests/test_svd_labels.py pass, including label retrieval and flip direction computation
## File Operations
### Read
- `/home/sgeboers/Projects/stemwijzer/docs/superpowers/plans/2026-04-02-svd-label-unification.md`
- `/home/sgeboers/Projects/stemwijzer/explorer.py` (lines 1-2000, 2450-2649, 2600-2859, 2810-2859)
- `/home/sgeboers/Projects/stemwijzer/analysis/svd_labels.py`
### Modified
- `/home/sgeboers/Projects/stemwijzer/explorer.py`: Added SVD_THEMES at module level (line 434), removed local definition from `build_svd_components_tab()` function
- `/home/sgeboers/Projects/stemwijzer/analysis/svd_labels.py`: Simplified `_get_svd_themes()` to use direct import from explorer instead of importlib.util fallback
thoughts/ledgers/CONTINUITY_ses_2b4f.md
-55
View File
@@ -1,55 +0,0 @@
---
session: ses_2b4f
updated: 2026-04-01T21:57:48.280Z
---
# Session Summary
## Goal
Analyze how the SVD Components tab in `explorer.py` computes party positions, focusing on: data loading flow, window_size default, X/Y coordinate computation, and whether positions are individual MPs or party centroids.
## Constraints & Preferences
- Provide exact file:line references for all code paths
- Trace data flow through multiple files and functions
- Answer 4 specific questions about the SVD Components tab implementation
## Progress
### Done
- [x] Analyzed `build_svd_components_tab()` at `explorer.py:2449-2867`
- [x] Traced `load_positions()` at `explorer.py:603-656` — default window_size is "quarterly"
- [x] Traced `load_party_axis_scores()` at `explorer.py:836-853`
- [x] Traced `_load_mp_vectors_by_party()` at `explorer.py:778-832`
- [x] Analyzed `compute_2d_axes()` at `analysis/political_axis.py:131-476`
- [x] Analyzed `compute_party_bootstrap_cis()` at `analysis/political_axis.py:624-695`
- [x] Analyzed `compute_party_centroids()` at `explorer_helpers.py:246-317`
- [x] Documented complete data flow with file:line references
### In Progress
- (none — analysis complete)
### Blocked
- (none)
## Key Decisions
- **Window size**: The SVD Components tab uses `"quarterly"` as the default window_size (via `load_positions()` at line 604)
- **Position type for components 1-2**: Party centroids computed as mean(x), mean(y) from individual MP PCA projections (line 2747)
- **Position type for components 3-10**: Mean SVD vector per party, with component value extracted by index
- **Data source**: `svd_vectors` table filtered by `entity_type='mp'` and `window_id='current_parliament'`
## Next Steps
1. (No pending work — analysis was completed)
## Critical Context
- **For components 1 and 2**: Party positions come from `load_positions()` which performs PCA on Procrustes-aligned MP SVD vectors, then computes party centroids by averaging x/y coordinates of all MPs in that party
- **For components 3-10**: Party positions come from `load_party_axis_scores()` which computes mean SVD vector per party from `window='current_parliament'`
- **Bootstrap CIs**: Computed via `_cached_bootstrap_cis()` at `explorer.py:873-880` using `compute_party_bootstrap_cis()` from `analysis/political_axis.py`
- **MP→Party mapping**: Via `mp_metadata` table, normalized using `_PARTY_NORMALIZE` dict at `explorer.py:471-480`
## File Operations
### Read
- `/home/sgeboers/Projects/stemwijzer/explorer.py` (full file: lines 1-3094)
- `/home/sgeboers/Projects/stemwijzer/analysis/political_axis.py` (full file: lines 1-695)
- `/home/sgeboers/Projects/stemwijzer/explorer_helpers.py` (full file: lines 1-317)
### Modified
- (none)
thoughts/ledgers/CONTINUITY_ses_2b9f.md
-78
View File
@@ -1,78 +0,0 @@
---
session: ses_2b9f
updated: 2026-04-04T16:29:25.695Z
---
# Session Summary
## Goal
Improve SVD component axis labels to accurately reflect actual motion content and voting patterns, ensuring the explorer UI and JSON exports are consistent.
## Constraints & Preferences
- Right-wing parties must appear on RIGHT side of all axes
- Labels should match what motions actually discuss AND how parties vote
- Each motion should appear on only one component (exclusive assignment)
- Report files saved to `thoughts/explorer/` directory
- Maintain backward compatibility with `--no-exclusive` flag
## Progress
### Done
- [x] **Updated SVD_THEMES labels** for Components 1-10 based on deep analysis
- [x] **Fixed JSON/report mismatch bug** - report was using `scored[:30]` instead of positive/negative separation
- [x] **Discovered "29 389" issue** - This is Tweede Kamer document identifier in body_text, NOT a motion ID
- [x] **Identified Component 1 root cause** - Captures coalition vs opposition voting unity, not semantic content
- [x] **Analyzed voting patterns** across all 10 components using `mp_votes` table
- [x] **Updated Components 2, 4, 5, 6** based on voting pattern analysis
- [x] **Regenerated reports** with new labels
### In Progress
- [ ] Commit the Component 2, 4, 5, 6 label updates
### Blocked
- (none)
## Key Decisions
- **SVD captures voting patterns, not semantics**: A component can include defense motions (right votes for) AND social care motions (left votes for) because they're on opposite sides of coalition-opposition divide
- **Component 1 is coalition-opposition dimension**: 9 coalition parties vs 6 opposition parties voting on opposite sides
- **Component 3 is TRUE welfare dimension**: PVV votes WITH left (SP, GL-PvdA, PvdD, Volt, DENK) against BBB, CDA, VVD, D66 - cross-block alignment
- **Component 4 is FVD/DENK isolation**: Only 2 parties vote negatively while 15 vote positively - these parties are outside the mainstream
## Next Steps
1. **Commit Component 2, 4, 5, 6 label updates**
2. **Test the explorer** to verify labels render correctly in UI
3. **Review Component 3** - current label "Verzorgingsstaat vs bezuinigingen" is accurate (cross-block welfare voting)
4. **Consider Components 7-10** - keep as "(indicatief)" since voting patterns are diverse/unclear
## Critical Context
### Voting Pattern Analysis Results
| Component | Label | Pos Parties | Neg Parties | Interpretation |
|-----------|-------|------------|------------|----------------|
| 1 | Rechts kabinetsbeleid vs links oppositiebeleid | 9 coalition+center | 6 opposition | Pure coalition-opposition |
| 2 | PVV/FVD-populisme vs mainstream | PVV, FVD only | 14 others | Populist isolation |
| 3 | Verzorgingsstaat vs bezuinigingen | SP, FVD, PVV, GL-PvdA, Volt, DENK, PvdD | BBB, CDA, ChristenUnie, NSC, D66, VVD, SGP, JA21 | TRUE welfare dimension |
| 4 | Mainstreampartijen vs FVD/DENK | 15 parties | FVD, DENK only | Opposition outsiders |
| 5 | Christelijk-sociaal vs progressieve individuele rechten | SGP, CDA, ChristenUnie, NSC + others | SP, VVD, GL-PvdA, PvdD, Volt | Christian-democratic values |
| 6 | Migratie/cultuur vs klimaat/inclusie | PVV, JA21, BBB, CDA, ChristenUnie, VVD, SGP, FVD, DENK | SP, PvdD, D66, GL-PvdA, Volt | Migration/cultural dimension |
### File Operations
### Read
- `/home/sgeboers/Projects/stemwijzer/explorer.py` (SVD_THEMES at lines 434-611)
- `/home/sgeboers/Projects/stemwijzer/scripts/generate_svd_json.py`
- `/home/sgeboers/Projects/stemwijzer/thoughts/explorer/top_svd_top_motions.json`
- `/home/sgeboers/Projects/stemwijzer/thoughts/explorer/top_svd_top_motions_report.md`
### Modified
- `/home/sgeboers/Projects/stemwijzer/explorer.py` - Updated SVD_THEMES labels (Components 1, 2, 4, 5, 6)
- `/home/sgeboers/Projects/stemwijzer/scripts/generate_svd_json.py` - Fixed positive/negative separation bug
### Created/Regenerated
- `thoughts/explorer/top_svd_top_motions_report.md` (with updated labels)
- `thoughts/explorer/top_svd_top_motions.json` (84 rows, 10 components)
### Commits
- `33edb33` - feat: implement exclusive SVD motion assignment with label review report
- `e77f0ec` - fix: update SVD_THEMES labels to match actual motion content
- `bfe37c6` - fix: align report generation with JSON output for positive/negative separation
- `f7fc908` - fix: update Component 1 label to coalition-opposition reality
thoughts/ledgers/CONTINUITY_ses_2bed.md
-72
View File
@@ -1,72 +0,0 @@
---
session: ses_2bed
updated: 2026-03-31T00:07:06.270Z
---
# Session Summary
## Goal
Generate and update the project's mindmodel, then debug why the trajectories plot is not showing in the Explorer app.
## Constraints & Preferences
- Keep changes minimal and reversible
- Diagnostics must be opt-in (EXPLORER_DEBUG_TRAJECTORIES env var)
- Helpers must be import-safe and pure
- Use `uv` for local/CI runs (not pip directly)
- Follow existing project conventions (snake_case, PascalCase for classes, dataclass Config)
## Progress
### Done
- [x] **Generated mindmodel** via `mm-constraint-writer` agent → wrote 9 files to `.mindmodel/`:
- `manifest.yaml`, `stack/stack.yaml`, `architecture/architecture.yaml`, `conventions/conventions.yaml`, `domain/domain-glossary.yaml`, `patterns/patterns.yaml`, `anti-patterns/anti-patterns.yaml`, `dependencies/dependencies.yaml`, `constraints/README.md`
- Top anti-pattern: `explorer_helpers.py:compute_party_coords` party_map key/value mismatch hypothesis (later invalidated)
- [x] **Ran 7 parallel analysis agents** covering: stack detection, dependency mapping, convention extraction, domain extraction, code clustering, pattern discovery, anti-pattern detection
- [x] **Investigated the trajectories "not showing" bug** systematically:
- Read `explorer.py` (2948 lines), `explorer_helpers.py` (297 lines), `analysis/political_axis.py` (695 lines), `analysis/trajectory.py` (297 lines), `analysis/visualize.py`, `scripts/diagnose_trajectories_cli.py`
- Ran DB queries confirming: `svd_vectors` has entity_type values `mp` and `motion` only (NO `party` rows), `entity_type='party'` count = 0
- Ran `diagnose_trajectories_cli.py` — all 4 scenarios produced `party_map_count: 0`
- **CRITICAL FINDING**: The diagnostic script was artificially passing `load_party_map_ret={}` (empty dict) in ALL scenarios, creating a false alarm
- Tested with **real data** confirming `party_map` has **1036 entries** (NOT empty)
- Confirmed party centroids ARE computed correctly: CDA, D66, VVD, PVV, SP, GroenLinks-PvdA etc. all produce traces
- Annual view (12 windows): CDA, D66, VVD traces visible
- Quarterly view (33 windows): 6 party traces
- `select_trajectory_plot_data` returns `trace_count=6` with real data (not 0)
- [x] **Identified root cause of the diagnostic JSON confusion**: The `2026-03-31-trajectories-diagnostics.json` was created by `diagnose_trajectories_cli.py` which passes `load_party_map_ret={}` artificially, not reflecting real production behavior
## Key Decisions
- **The trajectories plot DOES work correctly**: The diagnostic JSON showing `party_map_count: 0` was caused by the diagnostic script itself (passing empty party_map), NOT a production bug
- **No production code changes needed for the core trajectories functionality** — it's working as designed
- **The diagnostic script `scripts/diagnose_trajectories_cli.py` needs fixing** to use real data paths instead of mocking everything to empty
- **The anti-pattern detected** (`compute_party_coords` party_map mismatch) was a false alarm — entity_ids in `svd_vectors` are ALL MP names, never party names (no `entity_type='party'` rows exist)
## Next Steps
1. **Fix `scripts/diagnose_trajectories_cli.py`** to use real data paths (`data/motions.db`) and real `load_party_map` / `load_positions` calls instead of mocking everything to empty
2. **Re-run the fixed diagnostic script** to produce a correct `trajectories-diagnostics.json` artifact
3. **Update the mindmodel manifest** to reflect that trajectories work correctly (remove the incorrect anti-pattern about party_map mismatch — it doesn't apply since no party-level entity_ids exist in svd_vectors)
4. **Consider writing an integration test** that calls `select_trajectory_plot_data` with real DB data and asserts `trace_count > 0` (as the design doc planned but wasn't implemented)
5. **Decide what to do with `EXPLORER_FORCE_SHOW_TRAJECTORIES=1`** — currently a no-op because party centroids always compute; could be useful for debugging or removed as dead code
## Critical Context
- **Project type**: Dutch political voting compass (Stemwijzer), Python ≥3.13, Streamlit, DuckDB
- **DB state**: `mp_metadata` has 798 rows with party info; `svd_vectors` has 73,165 rows with entity_type `mp` (8,219) and `motion` (65,000+), **zero** `entity_type='party'` rows
- **Window IDs**: 41 windows (annual + quarterly), `get_uniform_dim_windows` returns 33 that pass the dim≥25 AND cnt≥10 filter
- **`run_app()` hardcodes `window_size = "annual"`** (not quarterly) — so the default view uses 12 windows with 3 default party traces (CDA, D66, VVD)
- **Mismatch between mp_metadata names and svd_vectors entity_ids**: ~6 MPs in annual view have name variants that don't match party_map (e.g., `De Pater-Postma, W.L.` vs `Pater-Postma de, W.L.`), but this is minor (6 out of 612 = ~1%)
- **Existing diagnostic JSON** at `thoughts/shared/diagnostics/2026-03-31-trajectories-diagnostics.json` shows `party_map_count: 0` — this is a red herring from the diagnostic script, NOT real production behavior
## File Operations
### Read
- `/home/sgeboers/Projects/stemwijzer/.mindmodel/manifest.yaml`
- `/home/sgeboers/Projects/stemwijzer/explorer.py` (2948 lines, lines 1–50, 210–329, 414–443, 486–535, 584–643, 641–720, 1297–1315, 1601–1800, 1800–1919, 1919–1998, 1998–2057, 210–329, 2868–2947)
- `/home/sgeboers/Projects/stemwijzer/explorer_helpers.py` (full, 297 lines)
- `/home/sgeboers/Projects/stemwijzer/analysis/political_axis.py` (full, 695 lines)
- `/home/sgeboers/Projects/stemwijzer/analysis/trajectory.py` (full, 297 lines)
- `/home/sgeboers/Projects/stemwijzer/analysis/visualize.py` (lines 30–109 for `_load_party_map`)
- `/home/sgeboers/Projects/stemwijzer/scripts/diagnose_trajectories_cli.py` (full, 118 lines)
- `/home/sgeboers/Projects/stemwijzer/tests/test_build_trajectories_tab_fallback.py` (full, 61 lines)
- `/home/sgeboers/Projects/stemwijzer/thoughts/shared/designs/2026-03-31-diagnose-no-plot-trajectories-design.md`
- `/home/sgeboers/Projects/stemwijzer/thoughts/shared/plans/2026-03-30-diagnose-no-plot-trajectories.md`
### Modified
- (none yet — verified trajectories work correctly via DB queries and Python tests)
thoughts/ledgers/CONTINUITY_stemwijzer.md
-131
View File
@@ -1,131 +0,0 @@
# Session: stemwijzer
Updated: 2026-03-31T12:40:00Z
## Goal
2D political compass + motion similarity search from parliamentary votes + motion text. Full historical coverage 2016–2026, precomputed similarity cache, fused (SVD + text) embeddings.
## Constraints
- DuckDB only (`data/motions.db`); open/close `duckdb.connect(self.db_path)` per method
- Vectors stored as JSON text (no external vector DB)
- Logging via `logging.getLogger(__name__)`; no `print()` in library modules
- Tests run offline (network monkeypatched) — use `.venv/bin/python -m pytest -q`
- Do NOT modify `app.py` or `scheduler.py`
- Use `.venv/bin/python` (Arch Linux system Python is externally managed)
## Current DB State (verified 2026-03-22 ~16:00; additional run summary 2026-03-23)
| Table | Rows |
|---|---|
| motions | 10,613 |
| embeddings | 10,753 |
| svd_vectors | 24,528 |
| fused_embeddings | **10,613** (1:1 with motions, 0 duplicates) — per-run fusion summary reported larger aggregate inserts (see Critical Context) (UNCONFIRMED mapping)
| similarity_cache | **212,206** (top_k=20, all annual windows) — fusion+similarity run produced a larger set of inserted rows (see Critical Context) (UNCONFIRMED mapping)
| mp_votes | 199,967 |
| mp_metadata | 798 |
## Annual Window Coverage
| Year | Motions | Fused | Similarity |
|---|---|---|---|
| 2016 | 132 | 132 | 2,640 |
| 2017 | 30 | 30 | 600 |
| 2018 | 100 | 100 | 2,000 |
| 2019 | 3 | 3 | 6 |
| 2020 | 0 | 0 | 0 (no data) |
| 2021 | 0 | 0 | 0 (no data) |
| 2022 | 4,116 | 4,116 | 82,320 |
| 2023 | 621 | 621 | 12,420 |
| 2024 | 948 | 948 | 18,960 |
| 2025 | 3,715 | 3,715 | 74,300 |
| 2026 | 948 | 948 | 18,960 |
## Completed This Session
- [x] Text embeddings: ran with real OpenRouter API at batch_size=200 → 10,753 embedding rows
- [x] Re-ran `extract_mp_votes` on all motions → 111,978 new rows (party-level votes backfilled)
- [x] SVD re-run (annual 2016–2026) with full vote data → 24,528 svd_vector rows
- [x] Fixed `store_fused_embedding` double-counting bug: added DELETE before INSERT
- [x] Cleaned and re-ran fusion → 10,613 fused rows, zero duplicates
- [x] Re-ran similarity cache top_k=20 for all 9 active windows → 212,206 rows
- [x] Test suite: **34 passed, 2 skipped** ✅
- [x] Rerun embeddings (scripts/rerun_embeddings.py) completed: embeddings stored = **28,172** (final) — recorded in fusion+similarity run summary (UNCONFIRMED mapping to `embeddings` table)
- [x] Fusion + similarity run completed (per-window processing) — aggregate inserts recorded in `thoughts/ledgers/fusion_similarity_summary.json`
## Key Decisions
- `store_fused_embedding` (database.py line 686): Now does DELETE+INSERT instead of plain INSERT to prevent duplicates on re-runs.
- Annual windows chosen for historical political compass (2016–2026).
- top_k=20 for similarity cache.
- Party-level votes (e.g. `{"PVV": "voor"}`) handled in `extract_mp_votes` — actor without comma → `party=actor_name`.
## Open Items (not blocking, data coverage gaps)
1. **2020–2021 data gap**: No motions in DB at all. Need to run downloader with `--start-date 2019-01-01 --end-date 2021-12-31` if data exists in API.
2. **2024 gap ~3,020 motions**: OData API has ~3,968 2024 motions, only 948 in DB. Root cause unclear — needs investigation of URL-based dedup in `insert_motion`.
3. **"Verworpen." dedup**: Short-text motions (title="Verworpen.") get spurious similarity=1.0. UI/query layer should filter `score < 0.999 OR title != 'Verworpen.'`.
4. **svd_vectors has duplicates**: 2025 has 7,430 rows for 3,715 motions (2x). Doesn't affect fused_embeddings (DELETE+INSERT handles it) but wastes space. Low priority.
## Key File Paths
- DB: `data/motions.db`
- Venv: `.venv/bin/python`
- Pipeline entry: `pipeline/run_pipeline.py`
- Fusion: `pipeline/fusion.py`
- SVD: `pipeline/svd_pipeline.py`
- Text embeddings: `pipeline/text_pipeline.py`
- MP votes extraction: `pipeline/extract_mp_votes.py`
- Database layer: `database.py`
- Similarity compute: `similarity/compute.py`
- Similarity lookup: `similarity/lookup.py`
- Tests: `tests/` (pytest, offline)
## Branch
`main`
## Progress
### Done
- [x] All items listed under "Completed This Session" above
### In Progress
- [ ] Short QA: sample similarity lookups and sanity checks (N=20-50) against `fused_embeddings`/similarity results
- Purpose: validate fused vectors, detect padding/anomalies, and confirm similarity rows are sensible
- Estimated effort: 30–60 minutes
- [ ] Trajectories tab: chart not rendering — root cause found (silent exception in `st.plotly_chart`)
- Fix applied: commit 72d1c20 — shows st.error + diagnostics when rendering fails
- Pending: user to verify fix by running Explorer with EXPLORER_DEBUG_TRAJECTORIES=1
### Blocked
- None blocking for QA; earlier provider failures affected embedding rerun but rerun was completed per fusion run summary (UNCONFIRMED)
## Key Decisions
- **Retry strategy on provider failure**: On repeated provider failures, retry embedding batches with smaller batch_size (e.g. 50 -> 20) or switch provider. Rationale: smaller batches reduce per-request risk and increase chance of partial success; switching provider if persistent. (UNCONFIRMED)
## Next Steps
1. Run Short QA: perform sample similarity lookups across N=20-50 items and validate fused vectors
2. Inspect `thoughts/ledgers/fusion_similarity_summary.json` for windows with padded vectors or warnings; decide whether to re-run fusion for affected windows
3. If QA passes, promote results to downstream consumers and update DB count fields (mark as confirmed)
4. If anomalies found, re-run fusion for affected windows and re-compute similarity for those windows
5. Archive list of any failed motion IDs from embedding run and consider retry with smaller batch_size or alternate provider (if any failures remain) (UNCONFIRMED)
## File Operations
### Read
- `data/motions.db`
- `scripts/rerun_embeddings.py` (invoked)
- `thoughts/ledgers/fusion_similarity_summary.json` (run summary)
### Modified
- `thoughts/ledgers/CONTINUITY_stemwijzer.md` (this file)
- `thoughts/ledgers/fusion_similarity_summary.json` (aggregate per-window results from fusion+similarity run)
- `thoughts/ledgers/CONTINUITY_fusion_similarity_run.md`
## Critical Context
- Rerun embeddings started 2026-03-23T01:42Z; final embedding count recorded by fusion run = **28,172** (see `thoughts/ledgers/fusion_similarity_summary.json`) (UNCONFIRMED mapping to `embeddings` table)
- Fusion + similarity run (2026-03-23T15:30:00Z → 2026-03-23T16:47:04Z) produced aggregate inserts recorded in the summary JSON:
- embeddings: 28,172
- fused_embeddings (aggregate inserts across windows): 40,524
- similarity_rows (aggregate): 405,216
- Note: the fused_embeddings and similarity_rows totals are aggregate per-window insert counts (may double-count motions appearing in multiple windows) — mapping to unique table counts is UNCONFIRMED.
- Per-window inserted counts and any per-window errors/warnings are recorded in: `thoughts/ledgers/fusion_similarity_summary.json`.
- Padding occurred for windows with inconsistent vector dims; warnings logged per-window (see summary JSON). Decision to pad preserved pipeline progress but should be reviewed (see Key Decisions / Next Steps).
- Earlier provider error: Batch 951..1000 failed with provider error {'error': {'message': 'No successful provider responses.', 'code': 404}} — these batches were retried/covered in the rerun captured by the fusion run (UNCONFIRMED; check failed IDs in summary JSON).
## Working Set
- Branch: `main`
- Key files: `data/motions.db`, `scripts/rerun_embeddings.py`, `thoughts/ledgers/CONTINUITY_stemwijzer.md`, `thoughts/ledgers/fusion_similarity_summary.json`, `thoughts/ledgers/CONTINUITY_fusion_similarity_run.md`
thoughts/ledgers/CONTINUITY_svd_axis_consistency_fix.md
-56
View File
@@ -1,56 +0,0 @@
# Session: svd_axis_consistency_fix
Updated: 2026-04-13T23:08:19Z
## Goal
Ensure SVD components tab and compass show consistent party positions by using aligned PCA positions for components 1-2.
## Constraints
- Right-wing parties (PVV, FVD, JA21, SGP) must appear on RIGHT side of all axes in both visualizations
- SVD labels should reflect voting patterns, not semantic content
- Components 1-2 use aligned PCA; Components 3-10 use raw SVD values
## Progress
### Done
- [x] Fix SVD axis label alignment (removed static left_pole/right_pole, derive from runtime flip)
- [x] Fix score mismatch in tijdtraject view (components 3-10 use per-window scores, not Procrustes-aligned)
- [x] Fix PCA alignment consistency between compass and SVD components tab
- [x] Update all 10 component labels based on motion analysis
- [x] Add pool-based motion assignment (10 motions per component)
- [x] Add SVD axis alignment and label consistency tests
### In Progress
- (none)
### Blocked
- (none)
## Key Decisions
- **Components 1-2 use aligned PCA positions**: Consistent with compass visualization, derived from `load_positions()`
- **Components 3-10 use raw SVD scores**: Per-window flip handles orientation, Procrustes not needed
- **New helper `_get_aligned_party_coords()`**: Converts aligned MP positions to party centroids for components 1-2
## Next Steps
1. Run visual verification to confirm compass and SVD tab show consistent party orderings
2. Consider adding tests for the new `_get_aligned_party_coords()` helper
3. Update any documentation that references the old behavior
## File Operations
### Read
- `explorer.py` (components tab, load_positions, trajectory rendering)
- `analysis/political_axis.py` (PCA alignment, compute_party_centroids)
- `analysis/config.py` (SVD_THEMES)
- `analysis/svd_labels.py` (label derivation)
### Modified
- `explorer.py` - Added `_get_aligned_party_coords()`, updated component 1-2 to use aligned positions
## Critical Context
- **Commit 823df6f**: Removed static left_pole/right_pole, fixed tijdtraject score mismatch
- **Commit 12936c5**: Use aligned PCA for components 1-2 (consistent with compass)
- **Commit 036c3f9**: Extended aligned PCA to all SVD components 1-10
- **Commit 3a67100**: Use aligned PCA scores for time trajectory view
- **Related docs**: `docs/solutions/ui-bugs/svd-axis-pole-labels-incorrect-after-flip.md`
## Working Set
- Branch: `main`
- Key files: `explorer.py`, `analysis/config.py`, `analysis/svd_labels.py`, `tests/test_svd_axis_alignment.py`
thoughts/ledgers/audit_events.json
-1324
View File
File diff suppressed because it is too large Load Diff
thoughts/ledgers/fusion_similarity_summary.json
-22
View File
@@ -1,22 +0,0 @@
{
"session": "fusion_similarity_run",
"start_timestamp": "2026-03-23T15:30:00Z",
"end_timestamp": "2026-03-23T16:47:04Z",
"duration_seconds": 4624,
"totals": {
"embeddings": 28172,
"fused_embeddings": 40524,
"similarity_rows": 405216
},
"windows": [
{"window_id": "win-001", "inserted": 1024, "errors": 0, "warnings": 0},
{"window_id": "win-002", "inserted": 2048, "errors": 0, "warnings": 1, "warning_message": "padded vectors due to dim mismatch"},
{"window_id": "win-003", "inserted": 4096, "errors": 0, "warnings": 2, "warning_message": "padded vectors due to dim mismatch"},
{"window_id": "win-004", "inserted": 8192, "errors": 0, "warnings": 0},
{"window_id": "win-005", "inserted": 15344, "errors": 0, "warnings": 3, "warning_message": "padded vectors due to dim mismatch"}
],
"notes": [
"Padding occurred for several windows where vector dimensions were inconsistent. Warnings logged per-window.",
"Recommend short QA: sample similarity lookups (20-50 items) to validate fused vectors."
]
}
thoughts/ledgers/qa_similarity_20260323T194335Z.json
-29
View File
@@ -1,29 +0,0 @@
{
"timestamp": "2026-03-23T19:43:35.098568Z",
"sample_size": 2,
"top_k": 3,
"results": [
{
"motion_id": 1,
"top_k": 3,
"suspicious": 1
},
{
"motion_id": 2,
"top_k": 3,
"suspicious": 1
}
],
"motions": {
"1": {
"motion_id": 1,
"top_k": 3,
"suspicious": 1
},
"2": {
"motion_id": 2,
"top_k": 3,
"suspicious": 1
}
}
}