Sven Geboers
49a1f2f67d
fix: correct axis 4 and 5 pole labels and flip orientation in SVD_THEMES
...
Axes 4 and 5 had inverted sign conventions relative to actual party votes.
Diagnostic confirmed SP/PvdD scored negative on axis 4 (free trade motions)
and FVD scored negative on axis 5 (secular motions), opposite to their
voting behaviour. Fix: swap positive_pole/negative_pole for both axes and
set correct flip direction so progressive parties appear on the left.
1 month ago
Sven Geboers
361cf9fd35
feat(explorer): SVD tab redesign — pole-split motions, party axis chart, inline expanders with voting
...
- Add ChristenUnie colour alias and CURRENT_PARLIAMENT_PARTIES frozenset (15 parties)
- Add load_party_axis_scores() — queries party SVD vectors from window=2025, cached
- Add _render_party_axis_chart() — 1D Plotly scatter of party positions per axis
- Restructure build_svd_components_tab: replace session-state button/detail-pane with
inline st.expander per motion, split into pos/neg pole columns, batch DB query for
all 10 motions including voting_results, rendered via _render_voting_results
Smoke-tested: 15 parties loaded, all 10 axis-1 motions returned with voting data.
1 month ago
Sven Geboers
9caaa8baca
feat(explorer): finalise SVD tab helper robustness and constants
...
Include plan: docs/superpowers/plans/2026-03-24-svd-tab-redesign.md
1 month ago
Sven Geboers
32fe3aed18
feat(explorer): harden _render_party_axis_chart axis bounds and range
...
Plan: docs/superpowers/plans/2026-03-24-svd-tab-redesign.md
1 month ago
Sven Geboers
5e061b0e40
feat(explorer): include CU alias in CURRENT_PARLIAMENT_PARTIES
...
Include plan path docs/superpowers/plans/2026-03-24-svd-tab-redesign.md
1 month ago
Sven Geboers
9f3ae15a16
feat(explorer): harden load_party_axis_scores (close DB, deterministic params)
...
Plan: docs/superpowers/plans/2026-03-24-svd-tab-redesign.md
1 month ago
Sven Geboers
fc1884ecd8
feat(explorer): harden SVD tab batch-fetch motion details
...
Include plan: docs/superpowers/plans/2026-03-24-svd-tab-redesign.md
1 month ago
Sven Geboers
6b8ec93fe0
feat(explorer): restructure SVD tab — pole-split motions, party axis chart, inline expanders with voting
1 month ago
Sven Geboers
1515661929
feat(explorer): add _render_party_axis_chart helper
1 month ago
Sven Geboers
35dbc8118a
feat(explorer): add load_party_axis_scores helper
...
docs/superpowers/plans/2026-03-24-svd-tab-redesign.md
1 month ago
Sven Geboers
9f538a8784
feat(explorer): add ChristenUnie colour alias and CURRENT_PARLIAMENT_PARTIES constant
...
docs/superpowers/plans/2026-03-24-svd-tab-redesign.md
1 month ago
Sven Geboers
521385c832
docs: add SVD tab redesign implementation plan
1 month ago
Sven Geboers
8c79aaf917
docs: add SVD tab redesign spec
1 month ago
Sven Geboers
c96c681641
feat(explorer): apply rigorous subagent-produced SVD axis themes
...
Replace draft SVD_THEMES with themes produced by per-axis analysis of all
10 unique top motions (zero cross-axis overlap, window=2025). Each axis now
has a detailed Dutch-language explanation, positive_pole and negative_pole
labels, displayed as colour-coded columns in the UI.
1 month ago
Sven Geboers
b85a9cebd7
fix(data): remove 18 duplicate motions and redo SVD top-motions analysis
...
Deduplication:
- Identified 18 motion pairs with identical body_text and externe_identifier
- Kept the lower ID (first inserted) from each pair
- Cascaded deletes: 18 motions, 18 embeddings, 28 svd_vectors, 23 fused_embeddings
- motions table: 28172 → 28154, zero body_text duplicate groups remaining
SVD analysis:
- Regenerated top_svd_top_motions.json for window=2025 with clean data
(7424 vectors, down from 7430)
- 100 unique motions across 10 axes, no title or ID duplicates
- De Vos huiseigenaren motie no longer appears twice in axis 3
1 month ago
Sven Geboers
8b73ab8bce
feat(explorer): add SVD axis themes and unique-motion deduplication
...
- Regenerated top_svd_top_motions.json for window=2025 with strict
cross-axis deduplication: 100 unique motions across 10 axes (10 per
axis, zero overlap), sorted by absolute SVD score
- Added SVD_THEMES dict to build_svd_components_tab with Dutch-language
theme label and political-polarisation explanation for each of the 10
axes (e.g. 'Confessioneel-conservatief vs. seculier-progressief')
- Selectbox now shows 'As N — <theme>' instead of bare component number
- Each selected axis shows an info banner with the full explanation
- Motion list buttons show ▲/▼ to indicate positive/negative SVD loading
- Translated UI strings to Dutch for consistency
1 month ago
Sven Geboers
e29d8a8055
ci(mindmodel): add scheduled validation workflow
1 month ago
Sven Geboers
2755dc373a
chore(format): add pre-commit and formatter configs
1 month ago
Sven Geboers
c24ab48704
chore(deps): move pytest to dev-dependencies
1 month ago
Sven Geboers
540099f2b7
ci(mindmodel): add report-only mindmodel validation workflow
1 month ago
Sven Geboers
d1faf2b3e4
feat(mindmodel): add CLI wrapper, edge-case tests, and manifest schema tests
1 month ago
Sven Geboers
ed289ff582
chore(mindmodel): add sanitized read-only manifest and reviewer README
1 month ago
Sven Geboers
f091846dc8
feat(mindmodel): add report-only validator skeleton, types, and tests
1 month ago
Sven Geboers
dbd8cc801a
docs(mindmodel): add README and tooling note; chore(tooling): add minimal pre-commit
1 month ago
Sven Geboers
5caebe1331
chore(ci): add mindmodel validator workflow
1 month ago
Sven Geboers
f77875ed54
feat(mindmodel): add CLI wrapper and tests
1 month ago
Sven Geboers
a74e6006f5
feat(mindmodel): add validator and tests
1 month ago
Sven Geboers
7bd7d0d18c
feat(mindmodel): add checks utilities and tests
1 month ago
Sven Geboers
2efd7ba3a0
feat(mindmodel): add manifest loader and tests
1 month ago
Sven Geboers
9c82962d47
chore: add .mindmodel/ project constraints and conventions
...
Generated mindmodel covering stack, architecture, domain glossary,
coding conventions, DuckDB/requests/embeddings/error-handling patterns,
anti-patterns, and explicit constraints for naming, imports, DB access,
error handling, and testing.
1 month ago
Sven Geboers
504400faf2
fix: quiz seed from motions with actual individual MP votes
...
Root causes:
- Seed selection sorted by controversy_score across all 28k motions, but
only 282 have individual MP vote records. Top controversial motions only
have party-level votes, so match_mps_for_votes always returned empty.
- global_db singleton was used for match/discriminate instead of the db_path
passed to the tab builder.
Fixes:
- Add MotionDatabase.get_motions_with_individual_votes(k) which queries
motions with comma-formatted mp_name votes, ordered by controversy_score
- Replace broken seed logic in build_mp_quiz_tab with this new method
- Replace global_db usages with a local MotionDatabase(db_path) instance
- Guard against motion IDs present in votes but absent from motions DataFrame
1 month ago
Sven Geboers
238d9e9ec2
test: add full quiz tab test suite and fix Geen stem normalization
...
- Fix match_mps_for_votes: 'Geen stem'/'no vote' now normalize to None (skipped),
not 'afwezig' — so unanswered questions don't inflate overlap count
- Add 5 additional tests to test_match_mps.py: zero-overlap exclusion,
empty input validation, Geen stem overlap skip, excluded motions respected
- Add tests/test_explorer_quiz.py: builder import smoke test plus 3 real-DB
end-to-end scenarios (unique match, indistinguishable MPs, discriminating
question reduces candidate set)
- Full suite: 73 passed, 2 skipped
1 month ago
Sven Geboers
eb73275f32
feat(mp-quiz): add MP quiz tab and DB helpers; add design and plan docs
1 month ago
Sven Geboers
b09e580f65
feat: motion content enrichment pipeline hardening
...
- ai_provider_wrapper: retry/fallback with exponential backoff, None sentinel for failed items
- text_pipeline: use wrapper, return 5-tuple (stored, skipped_existing, skipped_no_text, errors, failed_ids)
- similarity/compute: filter trivial 1.0 matches on identical short titles (<12 chars)
- rerun_embeddings: --retry-missing mode, calls ensure_text_embeddings_for_ids on failed ids
- sync_motion_content: per-ext_id retries, HTTPAdapter pool, --max-body-workers CLI flag, audit on failure
- qa_similarity script: samples motions, writes JSON ledger to thoughts/ledgers/
- All tests green: 61 passed, 2 skipped
1 month ago
Sven Geboers
aef7c45074
Refactor tests: replace sys.modules hacks with real DI + in-memory DB
...
- Add db=None, embedder=None params to ai_provider_wrapper, text_pipeline, compute_similarities
- New conftest.py: FakeEmbedder, mem_db (in-memory DuckDB), fake_embedder fixtures
- Rewrite test_ai_provider_wrapper (4 tests), test_rerun_embeddings_retry (2 tests), test_similarity_compute_filter (1 test) with real implementations
- Fix rerun_embeddings tests hanging on _get_all_windows by patching it alongside _clear_embeddings
- All 53 tests pass (2 skipped), 0 sys.modules hacks in refactored files
1 month ago
Sven Geboers
b7350d8f87
test: rewrite test_database_audit using mem_db fixture, no disk writes required
1 month ago
Sven Geboers
e4f2c7ff59
fix: update integration test to unpack 5-tuple from ensure_text_embeddings
1 month ago
Sven Geboers
ebb663aa8f
docs: add test refactor implementation plan
1 month ago
Sven Geboers
07a89a207c
docs: add test refactor design - replace monkeypatching with DI + in-memory DuckDB
1 month ago
Sven Geboers
ce27dc6ac5
chore(ledgers): record fusion+similarity run summary and JSON details
1 month ago
Sven Geboers
22f53840b8
fix(explorer): normalise party name 'Nieuw Sociaal Contract' -> 'NSC'
1 month ago
Sven Geboers
cbbdc6e738
fix(explorer): cleaner trajectories, NSC support, controversy filter, voting display, URL links
1 month ago
Sven Geboers
cd7bb3b1e0
fix(explorer): filter to uniform-dim windows before PCA to prevent np.vstack shape mismatch
1 month ago
Sven Geboers
2891e9ee70
feat: add StemAtlas Streamlit app, explorer, Docker deployment, blog charts
1 month ago
Sven Geboers
daa22c5e2b
feat: complete parliamentary embedding pipeline with full historical coverage
...
- Add fused (SVD + text) embedding pipeline for annual windows 2016-2026
- Fix store_fused_embedding duplicate bug: DELETE before INSERT (idempotent)
- Add --text-batch-size CLI flag to run_pipeline.py (default 200)
- Add explicit --start-date/--end-date to download_past_year.py
- Backfill mp_votes for all motions (party-level votes, 111k new rows)
- Add similarity cache recompute: 212k rows across 9 annual windows
- Improve ai_provider retry logic, text_pipeline batching
- Improve analysis/political_axis PCA handling and visualizations
- Add diagnostic/utility scripts: compare_svd, generate_compass, inspect_axis, etc.
- Untrack data/motions.db (3.6GB binary), add to .gitignore with outputs/
- Update continuity ledger with full session state
1 month ago
Sven Geboers
a78bee9b0a
feat(similarity): add precomputed similarity cache, fix fusion N+1, add 429 retry
...
- Add similarity/ package (compute.py, lookup.py) with numpy-based
pairwise cosine similarity and cached lookup
- database.py: create embeddings + similarity_cache tables in _init_database(),
add store_similarity_batch/get_cached_similarities/clear_similarity_cache helpers
- pipeline/fusion.py: replace N+1 per-motion embedding SELECT with single
bulk JOIN using DuckDB QUALIFY window function
- ai_provider.py: retry HTTP 429 with Retry-After header support
- migrations/2026-03-22-add-similarity-cache.sql: make executable
- Add tests for similarity compute, db helpers, and 429 retry (34 pass, 2 skip)
1 month ago
Sven Geboers
a248807e03
Add design: embedding-based motion similarity cache
...
Precomputed top-K similarity cache replacing the naive Python-scan
search_similar(). Also covers fixes for: embeddings table missing from
_init_database, fusion N+1 query, and ai_provider 429 retry.
1 month ago
Sven Geboers
bf68e48460
fix(analysis): improve PCA handling when PC1 dominates, add pca_residual option and plot autoscaling/variance annotation
1 month ago
Sven Geboers
23a1234314
feat(analysis): add 2D political compass (PCA/anchor) and 2D trajectories + visualizations
...
- compute_2d_axes (pca, anchor) with optional L2-normalisation pre-projection
- compute_2d_trajectories: per-MP coords, step vectors, magnitudes, totals
- plot_political_compass and plot_2d_trajectories (Plotly HTML)
- tests/test_political_compass.py (synthetic unit test)
1 month ago
Sven Geboers
3551a82f83
feat(analysis): add 2D political compass and 2D trajectories
...
- compute_2d_axes (PCA + anchor)
- compute_2d_trajectories
- plot_political_compass, plot_2d_trajectories
- unit test: tests/test_political_compass.py
1 month ago
