motief/docs/admin/recompute_similarity.md

Recomputing Similarity (Admin)

This document explains the admin CLI and developer workflows for recomputing similarity scores and running clustering jobs locally.

What this does

• Recompute similarity vectors/scores for existing records in the database.
• (Optionally) run the clusterer job that groups similar items based on recomputed vectors.

These operations are typically run as admin/maintenance tasks after changing the embedding/similarity logic or restoring a database snapshot.

Migration filenames

When adding or running migrations related to similarity or clustering, follow the project's migration filename pattern. Migration files touching similarity will typically include keywords like recompute_similarity or clusterer in the filename, for example:

• 20260101_001_recompute_similarity.py
• 20260215_002_clusterer_migration.py

Check your migrations folder for the exact filenames used in your environment.

Environment variables

When running the CLI locally you may need to set the following environment variables.

• TEST_DB_URL — connection string for a test/development database (used by local runs when you don't want to touch production data).
• AI_PROVIDER_MOCK — when set to a truthy value (1, true, yes) the AI/embedding provider is mocked so you don't make real API calls during development. Treat any non-empty value of AI_PROVIDER_MOCK as truthy.
• SIMILARITY_TOP_N — default number of top similar items to compute/keep for each record. The CLI --top-n flag overrides this value for the duration of the run.

Examples:

• Export in a shell (persistent for your session): export TEST_DB_URL="postgresql://user:pass@localhost:5432/devdb" export AI_PROVIDER_MOCK="true" export SIMILARITY_TOP_N="50"

• Inline for a single command (non-persistent): TEST_DB_URL="postgresql://user:pass@localhost/devdb" AI_PROVIDER_MOCK=1 python -m src.cli.recompute_similarity --batch-size 100

Notes:

• --top-n CLI flag takes precedence over SIMILARITY_TOP_N when both are provided.
• AI_PROVIDER_MOCK should be set to a truthy value (e.g. 1, true, yes) to avoid real external AI calls during local runs.

Running locally (development)

The CLI lives under src/cli. Use the module runner to execute the recompute script. Example commands:

Run a dry-run that doesn't persist changes:

python -m src.cli.recompute_similarity --top-n 10 --batch-size 100 --dry-run

Run for real (writes results to the DB):

python -m src.cli.recompute_similarity --top-n 50 --batch-size 500

Common flags

• --top-n — override SIMILARITY_TOP_N for this run.
• --batch-size — number of records to process per batch.
• --dry-run — inspect what would be changed without writing to the DB.

Notes

• Always point TEST_DB_URL at a non-production database when experimenting.
• Use AI_PROVIDER_MOCK=true to skip external calls and speed up local dev.
• If you change the embedding or similarity algorithm, re-run the recompute job and re-index/cluster as needed.

If you need help or encounter mismatches between migration files and the CLI, check the migrations folder and speak with the team member that authored the change.