feat: add StemAtlas Streamlit app, explorer, Docker deployment, blog charts
This commit is contained in:
@@ -0,0 +1,530 @@
|
||||
# Motion Explorer Implementation Plan
|
||||
|
||||
**Goal:** Regenerate analyses (compass + similarity cache), add an interactive Streamlit explorer (explorer.py) exposing political compass, party trajectories, motion search and browser, and update the blog post with real counts and vector-dimension facts.
|
||||
|
||||
**Design doc:** thoughts/shared/designs/2026-03-22-motion-explorer-design.md
|
||||
|
||||
---
|
||||
|
||||
## Summary / Architecture
|
||||
|
||||
We'll perform three high-level workstreams in dependency order:
|
||||
1. Analysis rerun: after the running pipeline releases the DB lock, run the minimal pipeline steps to (re)compute fused vectors and then recompute the similarity cache for all quarterly windows 2019-Q1 → 2024-Q4. Also run the static compass generator for verification.
|
||||
2. explorer.py: single-file Streamlit app placed at project root. It will use the existing analysis.* modules for heavy computations (cached via @st.cache_data) and duckdb read-only connections for all DB reads. Figures are produced with plotly and rendered inline in Streamlit.
|
||||
3. Blog post update: update thoughts/blog-post-political-compass.md with real DB numbers, updated similarity cache counts and correct fused vector dimensions.
|
||||
|
||||
Key implementation decisions (gap-filling):
|
||||
- Explorer is a single import-safe module: top-level definitions only, no expensive work on import. Running the UI triggers computations.
|
||||
- Use @st.cache_data for expensive functions: load_positions (compute_2d_axes), load_party_map, load_motions_df.
|
||||
- All DuckDB access in explorer.py will use duckdb.connect(database=..., read_only=True).
|
||||
- For similarity lookups we'll query similarity_cache directly via read-only DuckDB rather than calling MotionDatabase (which opens non-read-only connections), to respect the "DB may be running" constraint.
|
||||
- The UI will filter out motions with title exactly "Verworpen." by default; a sidebar toggle allows showing them.
|
||||
- Tests: explorer is a UI script so no behavioural TDD possible. We'll add a minimal import/sanity test ensuring the module is import-safe and key functions exist. Blog-post updates are manual but the plan includes a small helper script to compute exact counts to paste into the markdown.
|
||||
|
||||
---
|
||||
|
||||
## Dependency Graph
|
||||
|
||||
```
|
||||
Batch 1 (parallel): 1.1 [analysis-rerun - single operator task] (depends: none)
|
||||
Batch 2 (parallel): 2.1, 2.2 [explorer implementation + test] (depends: 1.1 for verification, but code can be implemented earlier)
|
||||
Batch 3 (serial): 3.1 [blog post update] (depends: 1.1)
|
||||
```
|
||||
|
||||
NOTE: The actual critical dependency is that the DB lock must be released before running the analysis rerun (Batch 1). The explorer code (Batch 2) can be implemented while the pipeline is running — it will only attempt DB reads at runtime and uses read-only connections.
|
||||
|
||||
---
|
||||
|
||||
## Batch 1: Analysis rerun (operator tasks — no repo files changed)
|
||||
|
||||
These are operational steps to run after the pipeline finishes and the DB lock is released. Run from the repository root.
|
||||
|
||||
Task 1.1: Regenerate compass outputs and fused vectors
|
||||
**What:** Run generate_compass.py and run the pipeline to (re)fuse vectors for quarterly windows covering 2019-Q1 → 2024-Q4. We will not re-run expensive fetch/extract/SVD/text steps if they are already up-to-date; only fusion (phase 5) must run so fused_embeddings exists for all windows.
|
||||
**Commands (run after pipeline finishes and DB unlocked):**
|
||||
|
||||
- Verify DB file exists:
|
||||
.venv/bin/python -c "import os,sys; p='data/motions.db'; print('exists' if os.path.exists(p) else 'MISSING'); sys.exit(0)"
|
||||
|
||||
- Run static compass for quick visual check (produces HTML output):
|
||||
.venv/bin/python scripts/generate_compass.py --db data/motions.db --out outputs --method pca --pca-residual
|
||||
|
||||
- Run the pipeline orchestrator so Phase 5 (fusion) runs for quarterly windows 2019-01-01 → 2025-01-01.
|
||||
We explicitly skip metadata/extract/svd/text since those may already be present; this minimizes rework and avoids mixing read/write connections in the current process.
|
||||
|
||||
.venv/bin/python -m pipeline.run_pipeline \
|
||||
--db-path data/motions.db \
|
||||
--start-date 2019-01-01 --end-date 2025-01-01 \
|
||||
--window-size quarterly \
|
||||
--skip-metadata --skip-extract --skip-svd --skip-text
|
||||
|
||||
**Notes:** run_pipeline.py includes a --skip-fusion flag; we MUST NOT pass --skip-fusion here because we want fusion to execute. The script supports exactly the flags shown.
|
||||
|
||||
**Verify:**
|
||||
- After run_pipeline completes, verify fused_embeddings rows exist for expected windows:
|
||||
.venv/bin/python - <<'PY'
|
||||
import duckdb
|
||||
conn = duckdb.connect(database='data/motions.db', read_only=True)
|
||||
print(conn.execute("SELECT window_id, COUNT(*) FROM fused_embeddings GROUP BY window_id ORDER BY window_id DESC").fetchall())
|
||||
conn.close()
|
||||
PY
|
||||
|
||||
Task 1.2: Recompute similarity cache for all quarterly windows 2019-Q1 → 2024-Q4
|
||||
**What:** Compute top-20 similarities per motion per window for the fused vectors and insert rows into similarity_cache. We will run similarity.compute.compute_similarities per window. The repository's similarity/compute.py exposes compute_similarities(vector_type='fused', window_id=..., top_k=20).
|
||||
|
||||
**Command (one-liner loop):**
|
||||
.venv/bin/python - <<'PY'
|
||||
from similarity.compute import compute_similarities
|
||||
windows = []
|
||||
years = range(2019, 2025) # 2019..2024
|
||||
for y in years:
|
||||
for q in (1,2,3,4):
|
||||
windows.append(f"{y}-Q{q}")
|
||||
total = 0
|
||||
for wid in windows:
|
||||
inserted = compute_similarities(vector_type='fused', window_id=wid, top_k=20, db_path='data/motions.db')
|
||||
print(f"window={wid} inserted={inserted}")
|
||||
total += inserted
|
||||
print('DONE total_inserted=', total)
|
||||
PY
|
||||
|
||||
**Notes & decisions:**
|
||||
- The compute_similarities function already clears existing rows for (vector_type, window_id) before inserting new ones, so this is safe to re-run.
|
||||
- If compute_similarities raises memory pressure for large windows, run on subsets (split windows further) — but try the simple loop first.
|
||||
|
||||
**Verify:**
|
||||
- Basic counts per window:
|
||||
.venv/bin/python - <<'PY'
|
||||
import duckdb
|
||||
conn = duckdb.connect(database='data/motions.db', read_only=True)
|
||||
print(conn.execute("SELECT window_id, COUNT(*) FROM similarity_cache WHERE vector_type = 'fused' GROUP BY window_id ORDER BY window_id").fetchall())
|
||||
print('total', conn.execute("SELECT COUNT(*) FROM similarity_cache WHERE vector_type = 'fused'").fetchone())
|
||||
conn.close()
|
||||
PY
|
||||
|
||||
- Spot-check top neighbors for a known motion id (replace 123 with a real id observed from motions table):
|
||||
.venv/bin/python - <<'PY'
|
||||
import duckdb
|
||||
conn = duckdb.connect(database='data/motions.db', read_only=True)
|
||||
print(conn.execute("SELECT id FROM motions ORDER BY id LIMIT 1").fetchall())
|
||||
src = conn.execute("SELECT id FROM motions ORDER BY id LIMIT 1").fetchone()[0]
|
||||
print('example source id=', src)
|
||||
print(conn.execute("SELECT target_motion_id, score FROM similarity_cache WHERE source_motion_id = ? AND vector_type = 'fused' ORDER BY score DESC LIMIT 10", (src,)).fetchall())
|
||||
conn.close()
|
||||
PY
|
||||
|
||||
---
|
||||
|
||||
## Batch 2: Explorer implementation (code + test) — parallel implementers
|
||||
|
||||
All tasks in this batch are independent and can be worked on in parallel. The single file to add is explorer.py at the project root. A small unit test ensures import-safety.
|
||||
|
||||
Decision: explorer.py will be placed at project root (same level as app.py) as requested by design. It will avoid performing DB work at import time so tests and other scripts can import it safely.
|
||||
|
||||
### Task 2.1: explorer.py
|
||||
**File:** explorer.py
|
||||
**Test:** tests/test_explorer_import.py
|
||||
**Depends:** none (safe to implement while pipeline runs)
|
||||
|
||||
Implementation (copy-paste-ready). This is a minimal, well-documented, and import-safe Streamlit app that follows the design requirements. It uses @st.cache_data on heavy functions, opens DuckDB with read_only=True for all reads, and uses existing analysis modules for computing 2D axes.
|
||||
|
||||
```python
|
||||
# explorer.py
|
||||
"""Streamlit motion explorer.
|
||||
|
||||
Import-safe: heavy computations are behind functions guarded by @st.cache_data
|
||||
and only run when the user opens the app (streamlit run explorer.py).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Dict, List, Optional, Tuple
|
||||
|
||||
import duckdb
|
||||
import pandas as pd
|
||||
import plotly.express as px
|
||||
import streamlit as st
|
||||
|
||||
# keep a module-level logger
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ---------- Cached data loaders ----------
|
||||
|
||||
|
||||
@st.cache_data
|
||||
def load_positions(db_path: str = "data/motions.db", window_size: str = "annual") -> Tuple[Dict[str, Dict[str, Tuple[float, float]]], Optional[Dict]]:
|
||||
"""Load positions_by_window and axis_def using existing analysis.political_axis.compute_2d_axes.
|
||||
|
||||
This delegates heavy computation to the analysis module and caches the result in Streamlit.
|
||||
The function intentionally accepts db_path so callers (tests) can pass a different path.
|
||||
"""
|
||||
try:
|
||||
from analysis.political_axis import compute_2d_axes
|
||||
except Exception as e:
|
||||
logger.exception("analysis.political_axis not available: %s", e)
|
||||
return {}, None
|
||||
|
||||
# compute_2d_axes may be expensive; we let the analysis module handle internals
|
||||
positions_by_window, axis_def = compute_2d_axes(
|
||||
db_path, method="pca", pca_residual=True, normalize_vectors=True
|
||||
)
|
||||
return positions_by_window, axis_def
|
||||
|
||||
|
||||
@st.cache_data
|
||||
def load_party_map(db_path: str = "data/motions.db") -> Dict[str, str]:
|
||||
"""Return mp_name -> party mapping.
|
||||
|
||||
Uses the helper in analysis.visualize which already knows heuristics.
|
||||
"""
|
||||
try:
|
||||
from analysis.visualize import _load_party_map
|
||||
|
||||
return _load_party_map(db_path)
|
||||
except Exception:
|
||||
logger.exception("Failed to load party map")
|
||||
return {}
|
||||
|
||||
|
||||
@st.cache_data
|
||||
def load_motions_df(db_path: str = "data/motions.db") -> pd.DataFrame:
|
||||
"""Load motions table into a cached pandas DataFrame (read-only connection).
|
||||
|
||||
Columns returned: id, title, description, date, policy_area, voting_results, layman_explanation, winning_margin, controversy_score
|
||||
"""
|
||||
conn = None
|
||||
try:
|
||||
conn = duckdb.connect(database=db_path, read_only=True)
|
||||
df = conn.execute(
|
||||
"SELECT id, title, description, date, policy_area, voting_results, layman_explanation, winning_margin, controversy_score FROM motions"
|
||||
).fetchdf()
|
||||
return df
|
||||
finally:
|
||||
if conn is not None:
|
||||
try:
|
||||
conn.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def query_similar_from_cache(db_path: str, source_motion_id: int, vector_type: str = "fused", window_id: Optional[str] = None, top_k: int = 10) -> List[Dict]:
|
||||
"""Query similarity_cache table using a read-only connection.
|
||||
|
||||
Returns list of dicts with keys target_motion_id, score, id.
|
||||
"""
|
||||
conn = None
|
||||
try:
|
||||
conn = duckdb.connect(database=db_path, read_only=True)
|
||||
params = [source_motion_id, vector_type]
|
||||
query = "SELECT target_motion_id, score, id, window_id FROM similarity_cache WHERE source_motion_id = ? AND vector_type = ?"
|
||||
if window_id is not None:
|
||||
query += " AND window_id = ?"
|
||||
params.append(window_id)
|
||||
query += " ORDER BY score DESC LIMIT ?"
|
||||
params.append(top_k)
|
||||
rows = conn.execute(query, params).fetchall()
|
||||
cols = [c[0] for c in conn.description]
|
||||
return [dict(zip(cols, r)) for r in rows]
|
||||
finally:
|
||||
if conn is not None:
|
||||
try:
|
||||
conn.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
# ---------- UI builders ----------
|
||||
|
||||
|
||||
def build_compass_tab(db_path: str, window_size: str, show_rejected: bool):
|
||||
positions_by_window, axis_def = load_positions(db_path, window_size)
|
||||
party_map = load_party_map(db_path)
|
||||
|
||||
if not positions_by_window:
|
||||
st.error("No position data available. Run the pipeline or check data/motions.db")
|
||||
return
|
||||
|
||||
windows = sorted(positions_by_window.keys())
|
||||
# default: latest window
|
||||
default_index = max(0, len(windows) - 1)
|
||||
idx = st.slider("Window", 0, len(windows) - 1, default_index)
|
||||
window_id = windows[idx]
|
||||
|
||||
pos = positions_by_window.get(window_id, {})
|
||||
names = list(pos.keys())
|
||||
xs = [p[0] for p in pos.values()]
|
||||
ys = [p[1] for p in pos.values()]
|
||||
parties = [party_map.get(n, "Unknown") for n in names]
|
||||
|
||||
fig = px.scatter(x=xs, y=ys, color=parties, hover_name=names, title=f"Political Compass ({window_id})")
|
||||
st.plotly_chart(fig, use_container_width=True)
|
||||
|
||||
|
||||
def build_trajectories_tab(db_path: str, window_size: str):
|
||||
positions_by_window, _ = load_positions(db_path, window_size)
|
||||
if not positions_by_window:
|
||||
st.error("No trajectories available")
|
||||
return
|
||||
|
||||
window_ids = sorted(positions_by_window.keys())
|
||||
# Build per-party centroids per window
|
||||
import numpy as _np
|
||||
|
||||
party_map = load_party_map(db_path)
|
||||
# user control
|
||||
show_mps = st.checkbox("Show MPs (individual trajectories)", value=False)
|
||||
selected_parties = st.multiselect("Parties (select to restrict)", options=sorted(set(party_map.values())), default=None)
|
||||
|
||||
fig = None
|
||||
if show_mps:
|
||||
# plot a small subset by default to avoid clutter
|
||||
mp_limit = 200
|
||||
traces = []
|
||||
# build mp_coords
|
||||
mp_coords = {}
|
||||
for wid in window_ids:
|
||||
for mp, coord in positions_by_window.get(wid, {}).items():
|
||||
mp_coords.setdefault(mp, []).append((wid, coord))
|
||||
|
||||
# optionally filter by party map
|
||||
mps = [m for m in mp_coords.keys() if (not selected_parties) or (party_map.get(m) in selected_parties)]
|
||||
mps = sorted(mps)[:mp_limit]
|
||||
|
||||
fig = px.line()
|
||||
for mp in mps:
|
||||
items = sorted(mp_coords[mp], key=lambda it: window_ids.index(it[0]))
|
||||
xs = [c[1][0] for c in items]
|
||||
ys = [c[1][1] for c in items]
|
||||
fig.add_scatter(x=xs, y=ys, mode='lines+markers', name=mp)
|
||||
else:
|
||||
# party centroids
|
||||
party_centroids = {}
|
||||
for wid in window_ids:
|
||||
coords_by_party = {}
|
||||
for mp, coord in positions_by_window.get(wid, {}).items():
|
||||
party = party_map.get(mp)
|
||||
if party is None:
|
||||
continue
|
||||
|
||||
|
||||
coords_by_party.setdefault(party, []).append(coord)
|
||||
for party, coords in coords_by_party.items():
|
||||
xs = [c[0] for c in coords]
|
||||
ys = [c[1] for c in coords]
|
||||
centroid = (_np.mean(xs), _np.mean(ys))
|
||||
party_centroids.setdefault(party, {'windows': [], 'coords': []})
|
||||
party_centroids[party]['windows'].append(wid)
|
||||
party_centroids[party]['coords'].append(centroid)
|
||||
|
||||
fig = px.line()
|
||||
for party, data in party_centroids.items():
|
||||
if selected_parties and party not in selected_parties:
|
||||
continue
|
||||
|
||||
xs = [c[0] for c in data['coords']]
|
||||
ys = [c[1] for c in data['coords']]
|
||||
fig.add_scatter(x=xs, y=ys, mode='lines+markers', name=party)
|
||||
|
||||
if fig is not None:
|
||||
st.plotly_chart(fig, use_container_width=True)
|
||||
|
||||
|
||||
def build_search_tab(db_path: str, show_rejected: bool):
|
||||
df = load_motions_df(db_path)
|
||||
if df is None or df.empty:
|
||||
st.info("No motions table available")
|
||||
return
|
||||
|
||||
# filters
|
||||
years = sorted(pd.to_datetime(df['date']).dt.year.dropna().unique().tolist())
|
||||
if years:
|
||||
start_year, end_year = min(years), max(years)
|
||||
else:
|
||||
start_year, end_year = 2019, 2024
|
||||
|
||||
year_range = st.slider("Year range", int(start_year), int(end_year), (int(start_year), int(end_year)))
|
||||
policy_areas = sorted(df['policy_area'].dropna().unique().tolist())
|
||||
policy_filter = st.multiselect("Policy areas", options=policy_areas, default=None)
|
||||
query = st.text_input("Search text (title / layman_explanation)")
|
||||
|
||||
# in-memory filter
|
||||
working = df.copy()
|
||||
# filter rejected default
|
||||
if not show_rejected:
|
||||
working = working[working['title'].str.strip() != 'Verworpen.']
|
||||
|
||||
working['y'] = pd.to_datetime(working['date']).dt.year
|
||||
working = working[(working['y'] >= year_range[0]) & (working['y'] <= year_range[1])]
|
||||
if policy_filter:
|
||||
working = working[working['policy_area'].isin(policy_filter)]
|
||||
if query:
|
||||
q = query.lower()
|
||||
mask = working['title'].fillna('').str.lower().str.contains(q) | working['layman_explanation'].fillna('').str.lower().str.contains(q)
|
||||
working = working[mask]
|
||||
|
||||
st.write(f"{len(working)} results")
|
||||
for _, row in working.sort_values(by='controversy_score', ascending=False).head(50).iterrows():
|
||||
with st.expander(f"{row['title']} — {row['date']}"):
|
||||
st.write(row.get('layman_explanation') or row.get('description') or '')
|
||||
st.write('Policy area:', row.get('policy_area'))
|
||||
st.write('Controversy score:', row.get('controversy_score'))
|
||||
# similar
|
||||
similar = query_similar_from_cache(db_path, int(row['id']), vector_type='fused', top_k=10)
|
||||
if similar:
|
||||
st.write('Vergelijkbare moties:')
|
||||
for s in similar:
|
||||
st.write(f"- id={s['target_motion_id']} score={s['score']:.3f} window={s.get('window_id')}")
|
||||
else:
|
||||
st.info('Nog geen vergelijkbare moties beschikbaar')
|
||||
|
||||
|
||||
def build_browser_tab(db_path: str, show_rejected: bool):
|
||||
df = load_motions_df(db_path)
|
||||
if df is None or df.empty:
|
||||
st.info("No motions table available")
|
||||
return
|
||||
|
||||
if not show_rejected:
|
||||
df = df[df['title'].str.strip() != 'Verworpen.']
|
||||
|
||||
df_display = df[['id', 'title', 'date', 'policy_area', 'controversy_score', 'winning_margin']].copy()
|
||||
df_display = df_display.sort_values(by=['date'], ascending=False)
|
||||
|
||||
sel = st.experimental_data_editor(df_display, num_rows='dynamic')
|
||||
# store selected id via session_state: user clicks a row and then presses a button
|
||||
st.write('Select a row and click "Show details"')
|
||||
sel_row_idx = st.number_input('Select row index (0-based)', min_value=0, max_value=max(0, len(df_display)-1), value=0)
|
||||
if st.button('Show details'):
|
||||
row = df_display.iloc[int(sel_row_idx)]
|
||||
st.subheader(row['title'])
|
||||
st.write(df.loc[df['id'] == row['id']].iloc[0].get('description') or '')
|
||||
similar = query_similar_from_cache(db_path, int(row['id']), vector_type='fused', top_k=10)
|
||||
if similar:
|
||||
st.write('Top similar:')
|
||||
for s in similar:
|
||||
st.write(f"- id={s['target_motion_id']} score={s['score']:.3f} window={s.get('window_id')}")
|
||||
else:
|
||||
st.info('Nog geen vergelijkbare moties beschikbaar')
|
||||
|
||||
|
||||
def run_app():
|
||||
st.set_page_config(layout='wide', page_title='Parlement Explorer')
|
||||
|
||||
st.sidebar.title('Explorer settings')
|
||||
db_path = st.sidebar.text_input('DuckDB path', value='data/motions.db')
|
||||
window_granularity = st.sidebar.selectbox('Window granularity', ['annual', 'quarterly'], index=0)
|
||||
show_rejected = st.sidebar.checkbox('Toon verworpen', value=False)
|
||||
|
||||
tabs = st.tabs(['Politiek Kompas', 'Partij Trajectories', 'Motie Zoeken', 'Motie Browser'])
|
||||
with tabs[0]:
|
||||
build_compass_tab(db_path, window_granularity, show_rejected)
|
||||
with tabs[1]:
|
||||
build_trajectories_tab(db_path, window_granularity)
|
||||
with tabs[2]:
|
||||
build_search_tab(db_path, show_rejected)
|
||||
with tabs[3]:
|
||||
build_browser_tab(db_path, show_rejected)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run_app()
|
||||
```
|
||||
|
||||
**Verify (local/dev):**
|
||||
- Run the app once the DB is available: streamlit run explorer.py
|
||||
- Verify that Tab 1 loads and you can slide windows, plot renders inline
|
||||
- Verify Tab 3 search returns results and shows similar motions
|
||||
- Verify all long-running operations are cached (first call slow, subsequent fast)
|
||||
|
||||
### Task 2.2: Test for explorer import-safety
|
||||
**File:** tests/test_explorer_import.py
|
||||
**Depends:** none
|
||||
|
||||
Minimal pytest to ensure the module can be imported without triggering heavy work and that run_app and key functions exist.
|
||||
|
||||
```python
|
||||
# tests/test_explorer_import.py
|
||||
import importlib
|
||||
|
||||
|
||||
def test_explorer_importable():
|
||||
mod = importlib.import_module('explorer')
|
||||
assert hasattr(mod, 'run_app')
|
||||
assert callable(mod.run_app)
|
||||
# key helpers
|
||||
assert hasattr(mod, 'load_positions')
|
||||
assert hasattr(mod, 'load_motions_df')
|
||||
```
|
||||
|
||||
**Verify:**
|
||||
- Run tests (no DB required for import test):
|
||||
.venv/bin/python -m pytest tests/test_explorer_import.py -q
|
||||
|
||||
---
|
||||
|
||||
## Batch 3: Blog post update (manual / single-file edit)
|
||||
|
||||
The blog post at thoughts/blog-post-political-compass.md contains placeholder numbers for motion counts, similarity cache totals and fused vector dimension claim. After analysis rerun completes, update the markdown with exact numbers.
|
||||
|
||||
### Task 3.1: Update blog post with real numbers
|
||||
**File to modify:** thoughts/blog-post-political-compass.md
|
||||
**Depends:** 1.1, 1.2 (analysis rerun and similarity cache recompute must finish first)
|
||||
|
||||
Steps to compute authoritative numbers (run after Batch 1 completes):
|
||||
1. Motion counts per year (SQL):
|
||||
.venv/bin/python - <<'PY'
|
||||
import duckdb
|
||||
conn = duckdb.connect(database='data/motions.db', read_only=True)
|
||||
rows = conn.execute("SELECT EXTRACT(year FROM date) AS y, COUNT(*) FROM motions GROUP BY y ORDER BY y").fetchall()
|
||||
print(rows)
|
||||
conn.close()
|
||||
PY
|
||||
|
||||
2. Similarity cache total count (fused vectors):
|
||||
.venv/bin/python - <<'PY'
|
||||
import duckdb
|
||||
conn = duckdb.connect(database='data/motions.db', read_only=True)
|
||||
total = conn.execute("SELECT COUNT(*) FROM similarity_cache WHERE vector_type = 'fused'").fetchone()[0]
|
||||
print('similarity_cache_fused_total=', total)
|
||||
conn.close()
|
||||
PY
|
||||
|
||||
3. Verify fused vector dimensions claim (inspect fused_embeddings.vector JSON lengths) — the fused field is stored as JSON array; compute distinct lengths:
|
||||
.venv/bin/python - <<'PY'
|
||||
import duckdb, json
|
||||
conn = duckdb.connect(database='data/motions.db', read_only=True)
|
||||
lens = conn.execute("SELECT DISTINCT CARDINALITY(vector) FROM fused_embeddings ORDER BY 1 DESC").fetchall()
|
||||
print('distinct_fused_lengths=', lens)
|
||||
conn.close()
|
||||
PY
|
||||
|
||||
Replace the placeholder table and counts in thoughts/blog-post-political-compass.md with the outputs above. Also correct the fused dimensions claim (line that currently reads "fused = [svd_dims (10)] + [text_dims (2560)] = 2570") by pasting the real dimensions found.
|
||||
|
||||
Verification: After editing, spell-check and run a quick search to ensure the old placeholder numbers are gone:
|
||||
grep -n "212,206\|2570\|~450 (newly backfilled)" -n thoughts/blog-post-political-compass.md || echo "No placeholders remain"
|
||||
|
||||
Commit message suggestions (to use when committing these changes):
|
||||
- feat(explorer): add initial Streamlit explorer (explorer.py) + import test
|
||||
- chore(analysis): recompute fused embeddings + similarity cache for 2019-Q1..2024-Q4 (instructions)
|
||||
- docs(blog): update political compass blog post with real counts and vector dims
|
||||
|
||||
---
|
||||
|
||||
## Rollout / verification checklist (final acceptance)
|
||||
- [ ] Analysis rerun finished without errors; fused_embeddings rows present for 2019-Q1..2024-Q4
|
||||
- [ ] similarity_cache contains top-k neighbors for each window (spot-check 3 windows)
|
||||
- [ ] explorer.py runs: streamlit run explorer.py renders tabs and figures inline
|
||||
- [ ] explorer uses read-only DuckDB connections (manual code review + spot-check)
|
||||
- [ ] thoughts/blog-post-political-compass.md updated with real numbers and vector dims
|
||||
- [ ] All tests still pass: .venv/bin/python -m pytest -q
|
||||
|
||||
---
|
||||
|
||||
## Appendix: reasoning & decisions
|
||||
- Design requires read-only DB access: MotionDatabase methods often open connections without read_only flag. To guarantee read-only behaviour while the pipeline runs, explorer.py queries DuckDB directly with read_only=True for all SELECTs. This avoids accidentally holding write locks.
|
||||
- The design required using existing analysis.* modules. compute_2d_axes is used as-is and wrapped by @st.cache_data; we rely on it to perform heavy PCA/SVD logic.
|
||||
- The similarity recompute step uses similarity.compute.compute_similarities per-window. The design referenced recompute_all_windows which did not exist in the repo; we use a small loop (shown above) to call compute_similarities per window.
|
||||
|
||||
*** End Plan
|
||||
@@ -0,0 +1,286 @@
|
||||
# StemAtlas Deployment — Implementation Plan
|
||||
|
||||
**Design:** `thoughts/shared/designs/2026-03-22-stematlas-deployment-design.md`
|
||||
**Date:** 2026-03-22
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Four independent batches. Batches A and B can run in parallel. Batch C requires the pipeline to finish first. Batch D is VPS infrastructure (manual steps, done once).
|
||||
|
||||
```
|
||||
Batch A: stemwijzer repo — Streamlit multi-page + Docker
|
||||
Batch B: sgeboers.nl repo — blog/, nav, blog post HTML skeleton
|
||||
Batch C: Charts — generate + embed (after pipeline finishes)
|
||||
Batch D: VPS infrastructure — Nginx vhost + Certbot + /srv/stematlas/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Batch A — stemwijzer repo: Streamlit multi-page + Docker
|
||||
|
||||
### A1. Check Dockerfile
|
||||
Read existing `Dockerfile` — verify it installs all deps from `pyproject.toml` and sets `CMD` to start the app. Note current entrypoint (probably `streamlit run app.py`).
|
||||
|
||||
### A2. Create `Home.py`
|
||||
New file at project root. Streamlit landing/about page:
|
||||
- Title: "StemAtlas"
|
||||
- Brief description of the two pages (quiz + explorer)
|
||||
- Links (Streamlit sidebar nav handles the rest automatically)
|
||||
- `st.page_link()` cards pointing to the two pages
|
||||
|
||||
### A3. Create `pages/1_Stemwijzer.py`
|
||||
Thin wrapper that imports and calls `app.main()`:
|
||||
- Import `from app import main`
|
||||
- Remove the `if __name__ == "__main__": main()` guard from `app.py` (or keep it — Streamlit ignores it when the file is imported)
|
||||
- The page title shown in Streamlit nav comes from the filename: `1_Stemwijzer` → "Stemwijzer"
|
||||
|
||||
### A4. Create `pages/2_Explorer.py`
|
||||
Same pattern:
|
||||
- Import `from explorer import run_app`
|
||||
- Call `run_app()`
|
||||
- Filename → nav label: "Explorer"
|
||||
|
||||
### A5. Update Dockerfile CMD
|
||||
Change entrypoint from `streamlit run app.py` to `streamlit run Home.py --server.port 8501 --server.address 0.0.0.0`.
|
||||
|
||||
### A6. Create `docker-compose.yml`
|
||||
Two services in the stemwijzer repo:
|
||||
|
||||
```yaml
|
||||
version: "3.9"
|
||||
services:
|
||||
stematlas:
|
||||
image: ${DOCKER_REGISTRY}/sgeboers/stemwijzer:latest
|
||||
ports:
|
||||
- "127.0.0.1:8501:8501"
|
||||
volumes:
|
||||
- /srv/stematlas/data:/app/data
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- DB_PATH=/app/data/motions.db
|
||||
|
||||
scheduler:
|
||||
image: ${DOCKER_REGISTRY}/sgeboers/stemwijzer:latest
|
||||
command: python scheduler.py
|
||||
volumes:
|
||||
- /srv/stematlas/data:/app/data
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- DB_PATH=/app/data/motions.db
|
||||
```
|
||||
|
||||
`127.0.0.1:8501` — only accessible from localhost, Nginx proxies externally.
|
||||
|
||||
### A7. Smoke test for `Home.py`
|
||||
Add `tests/test_home_import.py` — same pattern as `test_explorer_import.py`. Verify `Home` module is importable, `run_app` or equivalent callable exists.
|
||||
|
||||
### A8. Run tests
|
||||
`.venv/bin/python -m pytest -q` — all existing + new smoke tests must pass.
|
||||
|
||||
### Verification
|
||||
`docker build -t stematlas-local .` locally to confirm image builds without errors.
|
||||
|
||||
---
|
||||
|
||||
## Batch B — sgeboers.nl repo: blog/ + nav
|
||||
|
||||
> This batch requires access to the sgeboers.nl repo on git.sgeboers.nl.
|
||||
> Steps below assume the repo is cloned locally.
|
||||
|
||||
### B1. Inspect existing site structure
|
||||
Read `index.html` and any existing CSS files to understand:
|
||||
- Current nav structure (header? sidebar? footer?)
|
||||
- CSS class conventions for links/sections
|
||||
- Any existing page patterns to copy for the blog post
|
||||
|
||||
### B2. Create `blog/` directory
|
||||
Add `blog/index.html` — a minimal blog listing page:
|
||||
- Title: "Blog"
|
||||
- One entry: "StemAtlas — Mapping Dutch Democracy" → `blog/stematlas.html`
|
||||
- Matches existing site style
|
||||
|
||||
### B3. Add nav link to main site
|
||||
Update `index.html` (or whichever file contains the nav) to add a "Blog" link pointing to `/blog/`.
|
||||
|
||||
### B4. Create `blog/stematlas.html` skeleton
|
||||
Full blog post HTML based on `thoughts/blog-post-political-compass.md`:
|
||||
- Convert markdown to HTML (headings, paragraphs, code blocks, tables)
|
||||
- Add Plotly CDN `<script>` in `<head>`
|
||||
- **Chart placeholders**: `<!-- CHART: compass_latest -->`, `<!-- CHART: trajectories -->` — to be filled in Batch C
|
||||
- Add two CTAs linking to `stematlas.sgeboers.nl`:
|
||||
- After compass chart: *"Explore every window interactively →"*
|
||||
- At bottom: *"Try the Stemwijzer quiz →"*
|
||||
- Match existing site CSS (link the same stylesheet)
|
||||
|
||||
### B5. Update Drone pipeline (sgeboers.nl repo)
|
||||
Confirm the existing `.drone.yml` in sgeboers.nl picks up new files under `blog/` automatically (it should, if it deploys the whole repo root). No changes needed if it's already a `rsync` or `cp -r` deploy.
|
||||
|
||||
### Verification
|
||||
Open `blog/stematlas.html` locally in browser — post renders correctly with placeholder chart divs, nav works.
|
||||
|
||||
---
|
||||
|
||||
## Batch C — Charts: generate + embed (after pipeline finishes ~21:40)
|
||||
|
||||
> Requires `data/motions.db` to be unlocked (pipeline complete).
|
||||
|
||||
### C1. Run tests
|
||||
`.venv/bin/python -m pytest -q` — confirm all pass now that DB is free.
|
||||
|
||||
### C2. Run similarity cache recompute
|
||||
```
|
||||
.venv/bin/python -m pipeline.run_pipeline \
|
||||
--db-path data/motions.db \
|
||||
--start-date 2019-01-01 --end-date 2025-01-01 \
|
||||
--window-size quarterly \
|
||||
--skip-metadata --skip-extract --skip-svd --skip-text
|
||||
```
|
||||
Fusion only — fills `fused_embeddings` for new 2019–2021 and 2024 windows.
|
||||
|
||||
### C3. Recompute similarity cache
|
||||
```
|
||||
.venv/bin/python -c "
|
||||
from similarity.compute import compute_similarities
|
||||
import duckdb
|
||||
conn = duckdb.connect('data/motions.db', read_only=True)
|
||||
windows = [r[0] for r in conn.execute(\"SELECT DISTINCT window_id FROM fused_embeddings ORDER BY 1\").fetchall()]
|
||||
conn.close()
|
||||
for w in windows:
|
||||
print(f'Computing {w}...')
|
||||
compute_similarities('data/motions.db', w, top_k=20)
|
||||
"
|
||||
```
|
||||
|
||||
### C4. Generate compass HTML files
|
||||
```
|
||||
.venv/bin/python scripts/generate_compass.py \
|
||||
--db data/motions.db \
|
||||
--out outputs/blog-charts \
|
||||
--method pca --pca-residual
|
||||
```
|
||||
|
||||
This produces `outputs/blog-charts/compass_*.html` and `outputs/blog-charts/trajectories_*.html`.
|
||||
|
||||
### C5. Extract Plotly snippets
|
||||
For each chart file, extract the embeddable snippet:
|
||||
```python
|
||||
# Run once per chart to get embeddable HTML
|
||||
import plotly.io as pio
|
||||
# OR: just strip everything outside <div id="..."> and its <script>
|
||||
# The generate_compass.py output is self-contained — use BeautifulSoup or
|
||||
# manual extraction to get just the div+script block
|
||||
```
|
||||
|
||||
Simpler: modify `generate_compass.py` to add a `--partial` flag that calls `fig.to_html(include_plotlyjs=False, full_html=False)` and writes `.partial.html` files alongside the full ones.
|
||||
|
||||
### C6. Fill chart placeholders in blog post
|
||||
Replace `<!-- CHART: compass_latest -->` and `<!-- CHART: trajectories -->` in `blog/stematlas.html` with the extracted Plotly div+script blocks.
|
||||
|
||||
### C7. Update motion count table in blog post
|
||||
Run SQL to get authoritative counts:
|
||||
```sql
|
||||
SELECT strftime(date, '%Y') AS year, COUNT(*) AS motions
|
||||
FROM motions
|
||||
GROUP BY year ORDER BY year;
|
||||
```
|
||||
Replace placeholder numbers in `blog/stematlas.html` table.
|
||||
|
||||
### C8. Push sgeboers.nl repo
|
||||
Commit and push `blog/stematlas.html` + `blog/index.html` + nav changes to git.sgeboers.nl → Drone deploys.
|
||||
|
||||
---
|
||||
|
||||
## Batch D — VPS infrastructure (manual, one-time)
|
||||
|
||||
> SSH into the VPS. Steps are sequential.
|
||||
|
||||
### D1. Create data directory
|
||||
```bash
|
||||
sudo mkdir -p /srv/stematlas/data
|
||||
sudo chown $USER:$USER /srv/stematlas/data
|
||||
```
|
||||
|
||||
### D2. Copy `motions.db` to VPS
|
||||
From local machine:
|
||||
```bash
|
||||
rsync -avz --progress data/motions.db user@vps:/srv/stematlas/data/motions.db
|
||||
```
|
||||
~3.6GB transfer — takes a few minutes.
|
||||
|
||||
### D3. Add Nginx vhost
|
||||
New file `/etc/nginx/sites-available/stematlas`:
|
||||
```nginx
|
||||
server {
|
||||
listen 80;
|
||||
server_name stematlas.sgeboers.nl;
|
||||
return 301 https://$host$request_uri;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl;
|
||||
server_name stematlas.sgeboers.nl;
|
||||
|
||||
# Let's Encrypt certs (Certbot fills these in)
|
||||
ssl_certificate /etc/letsencrypt/live/stematlas.sgeboers.nl/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/stematlas.sgeboers.nl/privkey.pem;
|
||||
|
||||
location / {
|
||||
proxy_pass http://127.0.0.1:8501;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_read_timeout 86400;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Enable: `sudo ln -s /etc/nginx/sites-available/stematlas /etc/nginx/sites-enabled/`
|
||||
|
||||
### D4. Get Let's Encrypt cert
|
||||
```bash
|
||||
sudo certbot --nginx -d stematlas.sgeboers.nl
|
||||
```
|
||||
(Assumes Certbot is already installed and working for other subdomains on this VPS.)
|
||||
|
||||
### D5. First deploy
|
||||
The Drone pipeline for the stemwijzer repo will handle future deploys. For the first deploy, either:
|
||||
- Push a commit to trigger Drone, OR
|
||||
- Manually on VPS: `cd /srv/stematlas && docker-compose pull && docker-compose up -d`
|
||||
|
||||
### D6. Verify
|
||||
- `https://stematlas.sgeboers.nl` → Streamlit loads, shows Home.py
|
||||
- Both pages accessible from Streamlit nav
|
||||
- `docker-compose logs stematlas` — no errors
|
||||
|
||||
---
|
||||
|
||||
## Dependencies Between Batches
|
||||
|
||||
```
|
||||
A (stemwijzer repo) ──► D5 (first deploy) ──► D6 (verify)
|
||||
B (sgeboers.nl repo) ──► C8 (push blog)
|
||||
C (charts) ──► C8 (push blog)
|
||||
D1-D4 (VPS infra) ──► D5 (first deploy)
|
||||
|
||||
Pipeline finish (~21:40) ──► C1 (tests) ──► C2-C7 (charts)
|
||||
```
|
||||
|
||||
Batches A and B are fully independent — can start now.
|
||||
Batch C waits only for the pipeline to finish.
|
||||
Batch D is VPS-side and independent of code changes.
|
||||
|
||||
---
|
||||
|
||||
## Estimated Effort
|
||||
|
||||
| Batch | Tasks | Est. Time |
|
||||
|-------|-------|-----------|
|
||||
| A | Multi-page Streamlit + docker-compose | 45 min |
|
||||
| B | Blog HTML + nav (after inspecting site) | 60 min |
|
||||
| C | Charts + embed (after pipeline) | 30 min |
|
||||
| D | VPS infra (manual SSH) | 30 min |
|
||||
| **Total** | | **~2.5 hours** |
|
||||
Reference in New Issue
Block a user