feat: add StemAtlas Streamlit app, explorer, Docker deployment, blog charts
This commit is contained in:
@@ -0,0 +1,165 @@
|
||||
---
|
||||
date: 2026-03-22
|
||||
topic: "Dynamic motion explorer + analysis refresh"
|
||||
status: validated
|
||||
---
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The parliamentary embedding pipeline now covers 2019–2026 with ~25,000 motions, quarterly SVD windows, fused embeddings, and a 200k+ similarity cache. None of this is visible to anyone in an interactive form. The only outputs today are static HTML files written by `generate_compass.py` (if it's been run), and a blog post with placeholder numbers.
|
||||
|
||||
We need to:
|
||||
1. Regenerate all analyses and output graphs with the full dataset
|
||||
2. Build an interactive Streamlit explorer that surfaces the political compass, party trajectories, and motion similarity search
|
||||
3. Update the blog post with real numbers and findings
|
||||
|
||||
## Constraints
|
||||
|
||||
- Do NOT modify `app.py` or `scheduler.py` — these are the production quiz app
|
||||
- All DB access in the explorer must be **read-only** (no writes) — pipeline may be running
|
||||
- Explorer must work with existing `analysis.*` modules; no new analysis logic
|
||||
- Use `@st.cache_data` aggressively — `compute_2d_axes` runs PCA across all windows and is expensive (seconds, not milliseconds)
|
||||
- No new external dependencies beyond what's already installed (streamlit, plotly, umap-learn, scikit-learn are all present)
|
||||
- Follow existing code style: functional Python, `logging.getLogger(__name__)`, no print statements in library code
|
||||
|
||||
## Approach
|
||||
|
||||
**Single-file `explorer.py`** at the project root alongside `app.py`.
|
||||
|
||||
Four Streamlit tabs:
|
||||
1. **Politiek Kompas** — 2D MP/party scatter with a window slider
|
||||
2. **Partij Trajectories** — Line traces of party positions over time on the compass
|
||||
3. **Motie Zoeken** — Free-text + filter search, returns ranked similar motions
|
||||
4. **Motie Browser** — Filterable table of all motions, click to expand detail + similar motions
|
||||
|
||||
Run with: `streamlit run explorer.py`
|
||||
|
||||
This approach is chosen because:
|
||||
- Reuses all existing `analysis.*` modules without changes
|
||||
- Single file means no new package structure to maintain
|
||||
- Streamlit tabs map naturally to the four distinct views a researcher would want
|
||||
- Read-only DB access means it can run concurrently with the pipeline
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
explorer.py
|
||||
├── Tab 1: Politiek Kompas
|
||||
│ └── analysis.political_axis.compute_2d_axes (cached)
|
||||
│ └── analysis.visualize.plot_political_compass → Plotly figure
|
||||
│
|
||||
├── Tab 2: Partij Trajectories
|
||||
│ └── analysis.trajectory.compute_2d_trajectories (cached)
|
||||
│ └── analysis.visualize.plot_2d_trajectories → Plotly figure
|
||||
│
|
||||
├── Tab 3: Motie Zoeken
|
||||
│ └── database.get_all_motions (cached, read-only)
|
||||
│ └── database.search_similar (similarity_cache lookup)
|
||||
│ └── Custom search: filter title/description + show voting_results
|
||||
│
|
||||
└── Tab 4: Motie Browser
|
||||
└── database.get_filtered_motions (cached, read-only)
|
||||
└── On click: database.search_similar for related motions
|
||||
```
|
||||
|
||||
## Key Components & Responsibilities
|
||||
|
||||
**`explorer.py`**
|
||||
- Page config: `st.set_page_config(layout="wide", page_title="Parlement Explorer")`
|
||||
- Sidebar: DB path input (default `data/motions.db`), window-size toggle (annual/quarterly)
|
||||
- `@st.cache_data` wrappers for all expensive DB reads and computations
|
||||
- Four tabs via `st.tabs([...])`
|
||||
|
||||
**Tab 1 — Politiek Kompas**
|
||||
- Calls `compute_2d_axes(db_path, method='pca', pca_residual=True)` — cached
|
||||
- Window selector slider showing available windows
|
||||
- Renders the Plotly scatter for the selected window using `_render_compass_for_window(positions_by_window, window_id, party_map, axis_def)` — a thin Plotly figure builder (not writing to file)
|
||||
- Hover: MP name, party, (x, y) coordinates
|
||||
- Color by party using `_load_party_map(db_path)` — cached
|
||||
|
||||
**Tab 2 — Partij Trajectories**
|
||||
- Same `positions_by_window` data from Tab 1 (shared cache hit)
|
||||
- Multi-select party filter (default: all major parties)
|
||||
- Plotly figure: one trace per party, x/y positions connected by lines, labeled by window_id
|
||||
- Toggle between showing MPs or just party centroids (computed as mean of MP positions per party per window)
|
||||
|
||||
**Tab 3 — Motie Zoeken**
|
||||
- Search input (Dutch text, free-form)
|
||||
- Filters: year range (slider), policy area (multi-select), controversy score (slider)
|
||||
- On search: filter `motions` table in-memory against title + layman_explanation text (case-insensitive substring; no embedding search needed at this level)
|
||||
- Results list: each result shows title, date, policy area, controversy, layman_explanation
|
||||
- Expandable section per result: full description/body_text + "Vergelijkbare moties" from `similarity_cache`
|
||||
- Voting breakdown: parse `voting_results` JSON to show Voor/Tegen/Onthouden per party
|
||||
|
||||
**Tab 4 — Motie Browser**
|
||||
- `st.dataframe` with all motions (title, date, policy_area, controversy_score, winning_margin)
|
||||
- Column filters at top: year, policy area
|
||||
- Sort by: date DESC, controversy DESC, winning_margin ASC (most contested first)
|
||||
- Click row → `st.session_state` stores selected motion_id → detail panel below table
|
||||
- Detail panel: full motion text + top-10 similar motions from similarity_cache
|
||||
|
||||
## Data Flow
|
||||
|
||||
1. On startup: `compute_2d_axes` runs PCA, results cached in Streamlit's in-memory cache
|
||||
2. Tab 1/2: pure reads from `svd_vectors` + `mp_metadata` — all cached after first load
|
||||
3. Tab 3: on each search, filter pre-loaded motions DataFrame in-memory (no DB query per keypress)
|
||||
4. Tab 4: full motions table loaded once and cached; similarity lookups hit `similarity_cache` table via existing `database.get_cached_similarities`
|
||||
|
||||
All DuckDB connections are opened with `read_only=True` to allow concurrent pipeline access.
|
||||
|
||||
## Error Handling
|
||||
|
||||
- If `compute_2d_axes` fails (insufficient data for a window), skip that window and log warning — don't crash the app
|
||||
- If `similarity_cache` has no entries for a motion (e.g., new motion not yet processed), show "Nog geen vergelijkbare moties beschikbaar" placeholder
|
||||
- If DB file doesn't exist at startup, show an error banner with the path and instructions
|
||||
- All `duckdb.connect` calls wrapped in try/finally to guarantee close
|
||||
|
||||
## Analysis Refresh Plan
|
||||
|
||||
Before building the explorer, regenerate all outputs:
|
||||
|
||||
```bash
|
||||
# 1. Generate political compass HTML for latest window (annual)
|
||||
.venv/bin/python scripts/generate_compass.py \
|
||||
--db data/motions.db --out outputs \
|
||||
--method pca --pca-residual
|
||||
|
||||
# 2. Generate similarity cache for new windows (2019–2021, 2024 quarters)
|
||||
# (run_pipeline with --skip-metadata --skip-extract --skip-svd --skip-text)
|
||||
.venv/bin/python -m pipeline.run_pipeline \
|
||||
--db-path data/motions.db \
|
||||
--start-date 2019-01-01 --end-date 2025-01-01 \
|
||||
--window-size quarterly \
|
||||
--skip-metadata --skip-extract --skip-svd --skip-text
|
||||
|
||||
# 3. Recompute similarity cache for all windows
|
||||
.venv/bin/python -c "
|
||||
from similarity.compute import recompute_all_windows
|
||||
recompute_all_windows('data/motions.db', window_size='quarterly', top_k=20)
|
||||
"
|
||||
```
|
||||
|
||||
## Blog Post Updates
|
||||
|
||||
Target: `thoughts/blog-post-political-compass.md`
|
||||
|
||||
- Replace placeholder motion counts table with real numbers from DB query
|
||||
- Add actual findings from quarterly analysis (not visible in annual windows):
|
||||
- 2020-Q2 COVID vote clustering — parties converge on emergency measures
|
||||
- 2022-Q4 nitrogen crisis — sharpest left-right split in dataset
|
||||
- 2023-Q1 → 2024-Q1 gap (data missing for Q2-Q4 2023)
|
||||
- Add "Explorer" section describing `explorer.py` and how to run it
|
||||
- Update similarity cache row count (was 212k, now higher with new windows)
|
||||
- Fix the "fused = [10] + [2560] = 2570" claim — verify actual dimensions
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- Explorer has no tests (it's a UI script) — verify manually by running `streamlit run explorer.py` after pipeline completes
|
||||
- Existing 34 tests stay green — no changes to library modules
|
||||
- Run tests after completing implementation: `.venv/bin/python -m pytest -q`
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Should the explorer ship as a separate port from `app.py`? (Recommendation: yes, `app.py` stays on its port, `explorer.py` runs on a different port for internal/research use)
|
||||
- Should `Verworpen.` motions be filtered from search results by default? (Recommendation: yes, add a "Toon verworpen" toggle defaulting to off)
|
||||
- Annual or quarterly windows as the default for the compass? (Recommendation: annual — less noise, cleaner trajectories; quarterly available via sidebar toggle)
|
||||
@@ -0,0 +1,229 @@
|
||||
---
|
||||
date: 2026-03-22
|
||||
topic: "StemAtlas — Public Deployment on sgeboers.nl"
|
||||
status: validated
|
||||
---
|
||||
|
||||
# StemAtlas Deployment Design
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The stemwijzer project has three user-facing products ready to publish:
|
||||
1. **A blog post** explaining the political compass methodology and findings
|
||||
2. **An interactive explorer** (political compass, party trajectories, motion search)
|
||||
3. **The stemwijzer quiz** (vote on motions, see which parties match you)
|
||||
|
||||
These need to be deployed publicly on sgeboers.nl using the existing VPS + Gitea + Drone + Docker stack.
|
||||
|
||||
---
|
||||
|
||||
## The Name: StemAtlas
|
||||
|
||||
**`stematlas.sgeboers.nl`**
|
||||
|
||||
Dutch wordplay: **stem** = *vote* AND *voice* (as in "the voice of parliament") + **atlas** = a comprehensive map of the world. Together: *an atlas of voices* — a map of how Dutch democracy sounds from the inside.
|
||||
|
||||
It's broader than "stemwijzer" (which implies a voting guide) — it positions the site as a data exploration and journalism tool.
|
||||
|
||||
---
|
||||
|
||||
## Constraints
|
||||
|
||||
- Existing VPS running Nginx, Gitea, Drone
|
||||
- Deployment pipeline: Docker build → push to registry → SSH `docker-compose up -d`
|
||||
- sgeboers.nl is a **raw HTML/CSS site** (not Hugo) hosted as a repo on git.sgeboers.nl
|
||||
- DuckDB file lives on the VPS — single writer (scheduler), multiple readers (Streamlit)
|
||||
- No new cloud services or hosting costs
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Internet
|
||||
│
|
||||
├── sgeboers.nl (raw HTML/CSS site, existing repo on git.sgeboers.nl)
|
||||
│ └── blog/stematlas.html ← blog post with inline charts + link to subdomain
|
||||
│
|
||||
└── stematlas.sgeboers.nl
|
||||
└── Nginx (reverse proxy)
|
||||
└── Streamlit multi-page app (port 8501)
|
||||
├── Page 1: Stemwijzer Quiz (app.py)
|
||||
└── Page 2: Explorer (explorer.py)
|
||||
|
||||
VPS filesystem:
|
||||
/srv/stematlas/
|
||||
├── data/motions.db ← DuckDB (shared, read-write by scheduler)
|
||||
└── docker-compose.yml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Components
|
||||
|
||||
### 1. Streamlit Multi-Page App
|
||||
|
||||
Restructure entry point from `app.py` → `Home.py` with a `pages/` directory:
|
||||
|
||||
```
|
||||
Home.py ← landing page / about
|
||||
pages/
|
||||
1_Stemwijzer.py ← quiz (app.py content)
|
||||
2_Explorer.py ← explorer.py content
|
||||
```
|
||||
|
||||
Streamlit's built-in multi-page routing handles navigation. One Docker container, one port (8501).
|
||||
|
||||
**Why not two separate containers?**
|
||||
Single shared DuckDB file on VPS filesystem. Both pages open read-only connections (quiz opens read-write for session data, but that's the existing behaviour). One container = one volume mount = no coordination overhead.
|
||||
|
||||
### 2. Docker Compose
|
||||
|
||||
The existing `.drone.yml` already calls `docker-compose up -d` on the VPS. We add/update `docker-compose.yml`:
|
||||
|
||||
```
|
||||
Services:
|
||||
stematlas:
|
||||
image: registry/stematlas:latest
|
||||
ports: 8501 (internal only)
|
||||
volumes:
|
||||
- /srv/stematlas/data:/app/data ← persistent DB
|
||||
restart: unless-stopped
|
||||
|
||||
scheduler:
|
||||
image: registry/stematlas:latest
|
||||
command: python scheduler.py
|
||||
volumes:
|
||||
- /srv/stematlas/data:/app/data ← same DB, write access
|
||||
restart: unless-stopped
|
||||
```
|
||||
|
||||
**Scheduler as a sidecar**: runs in the same image but different container, keeps DB updated nightly. Streamlit container never writes to DB (except user sessions in the quiz).
|
||||
|
||||
### 3. Nginx Vhost
|
||||
|
||||
New server block on the VPS:
|
||||
|
||||
```
|
||||
stematlas.sgeboers.nl → proxy_pass http://127.0.0.1:8501
|
||||
```
|
||||
|
||||
Standard Streamlit proxy requirements: `proxy_http_version 1.1`, WebSocket upgrade headers for `/_stcore/stream`. Let's Encrypt cert via Certbot (standard pattern).
|
||||
|
||||
### 4. Drone CI Pipeline Update
|
||||
|
||||
Existing `.drone.yml` steps remain identical — build, push, SSH deploy. The only change: `docker-compose.yml` in the repo now references both the `stematlas` and `scheduler` services, so `docker-compose up -d` picks them both up.
|
||||
|
||||
No new Drone secrets needed if `DOCKER_REGISTRY`, `DEPLOY_HOST` etc. are already set.
|
||||
|
||||
### 5. Blog Post (Raw HTML page on sgeboers.nl)
|
||||
|
||||
The blog post is a new `blog/stematlas.html` file added to the sgeboers.nl repo on git.sgeboers.nl. The Drone pipeline for that repo deploys it like any other static file — push to git, Drone copies to webroot, Nginx serves it.
|
||||
|
||||
**Chart embedding strategy — inline Plotly divs:**
|
||||
|
||||
Rather than iframes, we extract just the chart `<div>` + `<script>` from `generate_compass.py`'s output (using `fig.to_html(include_plotlyjs='cdn', full_html=False)`) and paste them directly into the blog post HTML. This is cleaner than iframes — no border, no scroll issues, full-width, loads with the page.
|
||||
|
||||
Plotly CDN script included once in the `<head>`. Each chart is just a `<div id="chart-N">` + a `<script>` block below it.
|
||||
|
||||
**Linking to the subdomain:**
|
||||
|
||||
The blog post is the *article* — it tells the story with static charts. The subdomain is the *playground*. The post links to `stematlas.sgeboers.nl` at two natural moments:
|
||||
- After the political compass chart: *"Explore every window interactively →"*
|
||||
- At the end: *"Take the quiz yourself →"*
|
||||
|
||||
This is the right split: blog post brings readers in via search/sharing, subdomain gives them something to do.
|
||||
|
||||
**Chart generation workflow:**
|
||||
|
||||
```
|
||||
scripts/generate_compass.py → outputs/
|
||||
├── compass_2025.html ← main compass (latest window)
|
||||
├── trajectories_2019_2025.html ← party drift over time
|
||||
└── compass_2024-Q4.html ← quarterly detail
|
||||
```
|
||||
|
||||
Run `fig.to_html(include_plotlyjs='cdn', full_html=False)` to extract embeddable snippets, paste into `blog/stematlas.html` in the sgeboers.nl repo.
|
||||
|
||||
---
|
||||
|
||||
## Blog Post Charts — What to Include
|
||||
|
||||
The blog post narrates three acts. Each gets a supporting chart:
|
||||
|
||||
### Act 1: The Method
|
||||
**No chart needed** — the SVD explanation is conceptual. Use a simple HTML table for the vote matrix illustration.
|
||||
|
||||
### Act 2: The Political Compass
|
||||
**Chart: `compass_latest_annual.html`**
|
||||
|
||||
- 2D scatter of all parties for the most recent full annual window (2024 or 2025)
|
||||
- Axes: PC1 (left-right) × PC2 (residual, typically progressive-traditionalist)
|
||||
- Points coloured and labelled by party
|
||||
- Interactive: hover shows party name + coordinates
|
||||
- Caption: "Each party's position computed purely from voting patterns — no labels applied by us"
|
||||
|
||||
**Chart: `trajectories_all_parties.html`**
|
||||
|
||||
- Line chart of party positions across all annual windows (2016–2025)
|
||||
- One line per party, coloured consistently
|
||||
- Key narrative moments annotated: BBB arrival (2022), coalition formation (2022), Rutte → Schoof (2024)
|
||||
- Interactive: toggle parties on/off via legend
|
||||
|
||||
### Act 3: Motion Similarity
|
||||
**Chart: `compass_motions_sample.html`** (optional, depends on data quality)
|
||||
|
||||
- 2D UMAP scatter of ~500 sampled motions, coloured by policy area
|
||||
- Shows clustering: climate motions cluster together, budget motions cluster together, etc.
|
||||
- If UMAP results aren't clean enough to tell a clear story, skip this one
|
||||
|
||||
**Static table: Motion counts by year**
|
||||
Just a markdown table in the blog post — no chart needed.
|
||||
|
||||
---
|
||||
|
||||
## Data Flow
|
||||
|
||||
```
|
||||
scheduler.py (nightly)
|
||||
└── api_client → downloads new motions → DuckDB
|
||||
|
||||
On demand (manual or cron):
|
||||
└── run_pipeline.py → SVD + embeddings + fusion + similarity cache → DuckDB
|
||||
└── generate_compass.py → static HTML charts → sgeboers.nl repo (blog/stematlas.html)
|
||||
|
||||
Streamlit (reads only):
|
||||
└── duckdb.connect(read_only=True) → all analysis queries
|
||||
```
|
||||
|
||||
The DB is the source of truth. Charts are regenerated and re-copied to Hugo whenever the pipeline produces new data — probably monthly.
|
||||
|
||||
---
|
||||
|
||||
## Error Handling Strategy
|
||||
|
||||
- **Streamlit crash**: Docker `restart: unless-stopped` brings it back automatically
|
||||
- **Scheduler crash**: Same restart policy; DuckDB's WAL handles partial writes
|
||||
- **DB file corruption**: Not handled beyond OS-level backup. Mitigate by adding a weekly `cp data/motions.db data/motions.db.bak` to the scheduler or as a cron job on the VPS
|
||||
- **Blog charts stale**: Acceptable — charts are labelled with their window date; stale by 30 days is fine for a blog post
|
||||
- **Streamlit + scheduler write conflict**: Scheduler is the only writer. Streamlit and quiz sessions both use separate connections; DuckDB handles concurrent reads fine. The quiz writes `user_sessions` rows — low frequency, no conflict risk with scheduler
|
||||
|
||||
---
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- Import smoke test for `explorer.py` already exists (`tests/test_explorer_import.py`)
|
||||
- `Home.py` and `pages/` restructure needs a corresponding smoke test
|
||||
- Drone build will catch import errors before deploy
|
||||
- Manual verification: `docker-compose up` locally against a copy of `data/motions.db`, check all four Streamlit tabs render without error
|
||||
- Blog post charts: visual review after `generate_compass.py` run — no automated test needed
|
||||
|
||||
---
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Multi-page restructure scope**: Does the quiz (`app.py`) need any changes beyond being wrapped in a `pages/` file, or can it be imported as-is? The `if __name__ == "__main__"` guard in `app.py` needs reviewing.
|
||||
2. **Streamlit base path**: Subdomain approach (`stematlas.sgeboers.nl`) means no subpath complexity — Streamlit runs at `/`. Clean.
|
||||
3. **Chart update cadence**: Manual (run `generate_compass.py`, extract snippets, paste into blog post HTML, push to sgeboers.nl repo). Fine initially — charts are labelled with window date.
|
||||
4. **sgeboers.nl nav structure**: No blog directory exists yet. Need to add `blog/` dir, a `blog/stematlas.html` file, and a nav link on the main site. Structure TBD after inspecting the existing HTML/CSS site.
|
||||
5. **Nginx already running**: Need to confirm Certbot/Let's Encrypt workflow matches what's already set up on the VPS for other subdomains.
|
||||
Reference in New Issue
Block a user