Commit Graph
286 Commits
Author SHA1 Message Date
sgeboers 414c16ae9e refactor: extract data loading and trajectory logic from explorer.py
- Move trajectory analysis to analysis/trajectory.py (+136 lines)
- Move projection helpers to analysis/projections.py (+128 lines)
- Extract tab-specific data loaders to analysis/tabs/ (8 modules, +133 lines)
- Remove 702 lines from explorer.py (data loading extracted to
  analysis/explorer_data.py and new modules)
- Add axis label fallback tests (tests/test_axis_label_fallback.py)
- Add session docs: brainstorms, ideation, plans, and test-failures
2026-04-05 00:51:30 +02:00
sgeboers 154762a4c8 docs: refresh 2 stale references in solutions docs
- refactoring-streamlit-data-loading.md: update test count
  164/164 → 173/173 (7 new axis validation tests added)
- svd-component-labels-mismatch.md: SVD_THEMES moved from
  explorer.py:434-611 → analysis/config.py:67+ per the
  refactoring that extracted constants to analysis/config.py
2026-04-05 00:46:57 +02:00
sgeboers 5afbad11ad feat: add right-wing party axis validation
- Add CANONICAL_RIGHT (PVV, FVD, JA21, SGP) and CANONICAL_LEFT frozensets
  to analysis/config.py as the canonical source of truth
- Update analysis/svd_labels.py to import from config; re-export as
  RIGHT_PARTIES/LEFT_PARTIES for backward compatibility
- Add build_window_party_scores helper to analysis/explorer_data.py
- Add 7 integration tests in tests/test_axis_political_orientation.py
  validating that canonical right parties appear on the right side of SVD
  axes (x=component 1, y=component 2) using real DuckDB data
2026-04-05 00:42:46 +02:00
sgeboers 5ddf2cd85a chore: confirm deletion of stale files 2026-04-04 18:47:27 +02:00
sgeboers eb71328967 chore: commit remaining modified files from refactoring 2026-04-04 18:47:12 +02:00
sgeboers 805cc7e284 docs: add session ledger files and logic-errors docs 2026-04-04 18:47:05 +02:00
sgeboers 1dbf8da3a2 docs: move active plan to docs/plans/ 2026-04-04 18:47:01 +02:00
sgeboers f376300804 refactor: delete stale files and consolidate .mindmodel structure
Deleted stale root-level Python files:
- main.py (unused 'Hello world' script)
- verify.py (unused table info script)
- scraper.py (unused MotionScraper class)
- scheduler.py (unused DataUpdateScheduler class)

Deleted duplicate .mindmodel root YAML files (subdirectory versions are more comprehensive):
- anti-patterns.yaml, architecture.yaml, conventions.yaml
- dependencies.yaml, domain.yaml, domain-glossary.yaml
- stack.yaml, tech-stack.yaml, workflows.yaml

Added comprehensive .mindmodel subdirectories:
- constraints/ (naming, db-schema, error-handling, types, etc.)
- patterns/ (api, architecture, database, python, streamlit, etc.)
- examples/ (code examples for each pattern)
- anti-patterns/, architecture/, conventions/, dependencies/, domain/, stack/

Updated ARCHITECTURE.md to reflect current codebase:
- Removed references to non-existent files
- Added missing files (explorer.py, explorer_helpers.py, pipeline/)
- Added directory structure documentation
- Updated tech stack to include scipy, sklearn, umap

Updated .gitignore:
- Added patterns for generated analysis files
- Added .worktrees/ pattern (was already in gitignore but dir was deleted)

Removed empty .worktrees/ directory
2026-04-04 18:46:53 +02:00
sgeboers 0308d20f12 docs: add AGENTS.md with docs/solutions reference and SVD label best practice
- Add AGENTS.md with documented solutions reference
- Include SVD label convention (right-wing parties on right side)
- Document SVD insight: labels reflect voting patterns, not semantics
- Fix SQL verification example to use Python approach
2026-04-04 18:36:05 +02:00
sgeboers 92c3c0ee01 fix: update Components 2, 4, 5, 6 SVD labels based on voting pattern analysis 2026-04-04 18:29:48 +02:00
sgeboers f7fc908b58 fix: update Component 1 label to coalition-opposition reality
The component captures voting unity of the right-wing coalition vs left
opposition, NOT semantic content like 'defense' or 'EU integration'.
Motions about elderly care (Dobbe) appear because the left votes for them
while the right coalition votes against - this is coalition-opposition
polarization, not policy domain.
2026-04-04 18:25:19 +02:00
sgeboers bfe37c6806 fix: align report generation with JSON output for positive/negative separation
Bug: report_per_component used scored[:args.report_top_n] which took
top N by score (all positive for components with only positive scores).
JSON correctly separated positive and negative poles.

Fix: Use same positive/negative separation logic for report as JSON.
2026-04-04 18:19:31 +02:00
sgeboers e77f0ec9e3 fix: update SVD_THEMES labels to match actual motion content
- Component 1: EU-integratie → Defensie-uitgaven en NAVO-commitment
- Component 2: Flip False (was True), clarify populistisch vs institutioneel
- Component 4: Pragmatisch centrisme → Gematigde middenpartijen
- Component 5: Christelijk-sociaal → Gemeenschapszin en sociale zekerheid
- Component 6: Add migratie-integratie to label
- Component 8: Complete rewrite - was COMPLETELY WRONG (vaccinatie, onderwijs)
- Component 9: Decentraal bestuur → Pragmatische probleemoplossing
- Component 10: Institutioneel toezicht → Kritisch op overheidsbemoeienis
2026-04-04 18:09:46 +02:00
sgeboers 33edb334c4 feat: implement exclusive SVD motion assignment with label review report
- Each motion now assigned to exactly one component (highest absolute score)
- Added --exclusive flag (default: True) for backward compatibility
- Added markdown report generation with motion details for label review
- Added --report-top-n for report size (default: 20 per component)
- Updated JSON output with 'exclusive' flag for transparency
2026-04-04 17:55:28 +02:00
sgeboers ee8ffea6e2 fix: add health check wait to ansible deploy
Wait for Streamlit to be ready before finishing deployment to prevent 502 errors
2026-04-02 22:46:01 +02:00
sgeboers d8bee43c15 feat: add voting discipline analysis paragraph under political compass
- Add Dutch paragraph explaining Rice index and party discipline patterns
- Analysis covers high discipline parties (PVV, SGP) vs lower discipline parties
- Explains what discipline reveals about party dynamics
2026-04-02 22:03:57 +02:00
sgeboers f5f0c8d6b1 feat: add year selector for SVD components 3-10
- Add _load_mp_vectors_by_party_for_window() to load SVD vectors for specific windows
- Add load_party_axis_scores_for_window() cached function
- Add year selector UI for components 3-10 similar to components 1-2
- Uses get_uniform_dim_windows() to get available windows
2026-04-02 21:49:12 +02:00
sgeboers 5f7126f53f docs: add voting discipline analysis
Explain Rice index methodology and what it reveals about Dutch political parties
2026-04-02 21:47:08 +02:00
sgeboers abd3281044 refactor: remove Stemgedrag cohesie section and fallback axis message
- Remove voting discipline (cohesie) section from Political Compass tab
- Remove 'empirisch stempatroon zonder duidelijke ideologische richting' fallback message from axis classifier
- Clean up unused fallback template from _INTERPRETATION_TEMPLATES
2026-04-02 21:46:24 +02:00
sgeboers a5e95c33d7 refactor: use scatter plot format for SVD components 3-10
- Changed _render_party_axis_chart_1d from horizontal bar chart to scatter plot
- Same format as components 1-2: markers on horizontal line with axis arrows- Axis labels now show correct direction with arrows (← left | right →)
- Ensures consistent visualization across all SVD components
2026-04-02 21:39:09 +02:00
sgeboers fa019d8a9c test: add test for auto-flip computation for all components 2026-04-02 21:21:58 +02:00
sgeboers ed2b4c1fae test: add tests for 1D party position chart 2026-04-02 21:19:23 +02:00
sgeboers 95183fec5b test: update tests for unified SVD label system (Task 7) 2026-04-02 21:08:50 +02:00
sgeboers ba24ad4fe6 feat: auto-compute flip directions for all SVD components (Task 6) 2026-04-02 21:07:46 +02:00
sgeboers bda803089a feat: add 1D party position charts for SVD components 3-10 (Task 5) 2026-04-02 21:07:11 +02:00
sgeboers 5b3cf23d36 refactor: use svd_labels for fallback labels in explorer and axis_classifier (Task 4) 2026-04-02 21:05:30 +02:00
sgeboers 36b58ad50d refactor: use svd_labels module for fallback labels in axis_classifier (Task 3) 2026-04-02 21:02:53 +02:00
sgeboers 5b1be26050 refactor: move SVD_THEMES to module level for import (Task 2) 2026-04-02 21:02:03 +02:00
sgeboers a1c3e92fab docs: add SVD label unification implementation plan 2026-04-02 20:45:44 +02:00
sgeboers bed776f295 docs: add SVD label unification design spec 2026-04-02 20:40:56 +02:00
sgeboers c9c59dd166 feat(diagnostics): enhance trajectory diagnostic script with real data mode 2026-04-01 01:59:41 +02:00
sgeboers 7e202e15be test(trajectory): fix test quality issues 2026-04-01 01:57:59 +02:00
sgeboers 8bc43b67fd test(trajectory): add tests for plot rendering with edge cases 2026-04-01 01:56:04 +02:00
sgeboers 31e1dd4371 fix(trajectory): correct import for diagnose_trajectories 2026-04-01 01:51:14 +02:00
sgeboers 5cd031777c fix(trajectory): improve fallback handling and diagnostics when trace_count is 0 2026-04-01 01:49:14 +02:00
sgeboers 8e67b89a1d fix(trajectory): fix division by zero and None handling in name normalization 2026-04-01 01:46:13 +02:00
sgeboers 0b79709847 fix(trajectory): normalize MP names to improve party_map matching 2026-04-01 01:43:39 +02:00
sgeboers 26bdb4c61c refactor(trajectory): fix code quality issues in centroid diagnostics 2026-04-01 01:40:40 +02:00
sgeboers 7d93753530 fix(trajectory): add diagnostics to compute_party_centroids for NaN detection 2026-04-01 01:35:57 +02:00
sgeboers 385a25853c diagnose(trajectory): add diagnostics to identify why trace_count is 0 2026-04-01 01:31:49 +02:00
sgeboers 24796f97d3 test: add trajectory pipeline integration test 2026-03-31 15:08:50 +02:00
sgeboers 69208e0bf6 fix: skip second trace loop when helper succeeds to avoid duplicate traces 2026-03-31 15:04:45 +02:00
sgeboers 5d1328f824 chore: add TRAJ DEBUG print checkpoints to build_trajectories_tab 2026-03-31 14:55:08 +02:00
sgeboers 1a83f0f319 docs: add trajectory plots debugging plan 2026-03-31 02:12:49 +02:00
sgeboers 9f98dbae60 Add debug st.info before st.plotly_chart to diagnose invisible chart 2026-03-31 01:49:38 +02:00
sgeboers 72d1c20340 Show error and diagnostics when st.plotly_chart fails instead of silent pass
Previously the st.plotly_chart call was wrapped in 'except Exception: pass'
which silently swallowed all rendering errors. The user would see no chart
and no error message.

Now:
- Exception message is shown via st.error()
- Diagnostics JSON is shown when debug is enabled (EXPLORER_DEBUG_TRAJECTORIES=1
  or UI checkbox), even when trace_count > 0

This reveals the actual root cause when the chart fails to render.
2026-03-31 01:38:30 +02:00
sgeboers baee50f3a5 feat(explorer): extend diagnostic inspector to surface mp samples/counts
chore(explorer): add get_debug_trajectories_enabled helper

feat(explorer): instrument trajectories with debug diagnostics and un-silence helper exceptions
2026-03-31 00:28:14 +02:00
sgeboers 0f2db0a9be chore(explorer): add get_debug_trajectories_enabled helper 2026-03-31 00:22:26 +02:00
sgeboers 525cd157c0 docs: add diagnose-no-plot-trajectories design (2026-03-30) 2026-03-30 23:57:18 +02:00
sgeboers ce1fc86bcb docs(design): add fix-missing-trajectories design 2026-03-30 22:24:05 +02:00