pgvector-filterbench

A command-line tool for checking how pgvector search behaves once you add filters. It measures speed, compares approximate results with exact search, and checks query plans to see whether PostgreSQL is using your HNSW index.

DownloadVersion 0.1.2

pgvector-filterbench

About pgvector-filterbench

In an actual application, you usually need results from a particular customer, category or group of active records. Those filters can change how approximate vector search behaves. A query might finish quickly while missing some of the nearest matches that satisfy your conditions.

pgvector-filterbench runs an exact search as a reference, then tests different HNSW strategies and settings against it. You can compare how long each approach takes and how many of the exact search’s top matches it finds. That gives you something concrete to work with when adjusting the search parameters.

It also checks the query plans. PostgreSQL can choose a sequential scan, which makes a supposedly impressive HNSW result a little awkward: the index you thought you were testing wasn’t being used. The benchmark checks for that explicitly.

You can sample query vectors from your database or supply your own, then compare the standard HNSW search with strict and relaxed iterative scans. Results are written to JSON and Markdown. If you want to run it in CI, you can set thresholds so a drop in recall or unexpected index usage fails the check.

There’s a Docker demo with synthetic data if you want to see how it works before setting up a run against your own tables. The tool itself is written in Go, and prebuilt binaries are available.

The benchmark queries are read-only, although exact searches can still give the database a fair amount of work. For a larger dataset, I’d start on a replica or staging database and keep the initial test small.

Useful when you’re tuning filtered vector search and want to see what your settings actually change.

Release history

Version 0.1.218 Sep 2026

Added

Report how many rows every measured query returned: rows_returned (mean and minimum) plus short_result_runs in the JSON report, and Rows (mean/min) plus Short Sets columns in the Markdown report. Recall and same-rank alone cannot distinguish a filter-shortened result set from a complete but wrong one. The exact ground-truth row reports rows returned as well, which is always k.exact for eligible samples.
Optional gate.require_full_results check that fails the gate when any measured ANN configuration returns fewer than k.ann rows. It is disabled by default, because short filtered-HNSW result sets are the condition the tool exists to measure.

Changed

Raise the minimum supported Go version from 1.21 to 1.25 for patched dependencies.

Fixed

Accept --version and -v on the root command. Both previously failed with unknown flag: --version, and the version was absent from --help. The flag is wired after the embedded VERSION file is loaded so it never reports the dev placeholder, and it prints the same string as the version command.
Avoid allocating unused per-run ANN row-count copies while building reports.
Pin CI quality checks to Go 1.27.x and update Staticcheck and govulncheck to compatible versions, fixing analyzer failures after the floating Go stable version advanced to 1.27.
Update golang.org/x/text to v0.39.0 to fix GO-2026-5970, an infinite loop on invalid UTF-8 input in Unicode normalization, and update its golang.org/x/sync dependency.

External download

Download 0.1.2

Version 0.1.119 Jul 2026

Fixed

Calculate recall against the exact top k.ann rows instead of every row returned by a larger k.exact query.
Omit non-applicable plan_valid and index_used claims from the exact JSON result row.
Run index validation and vector-dimension discovery in read-only transactions bounded by the configured statement and lock timeouts.
Redact PostgreSQL server messages from logged and reported measurement errors while retaining operation context and SQLSTATE.
Reject pgx simple-protocol connections, removing the runtime condition required by GO-2026-5004 without dropping Go 1.21 support.
Build the container image with the patched Go 1.26.5 toolchain while continuing to test source compatibility with Go 1.21.

Added

GitHub Actions checks for Go 1.21 compatibility, unit and race tests, vet, staticcheck, govulncheck, and PostgreSQL 16/17/18 pgvector end-to-end gates.

External download

Download 0.1.1

Version 0.1.016 Jul 2026

Initial release of pgvector-filterbench.
Go CLI with run and version commands.
Four benchmark strategies: exact ground truth, bounded ANN, strict iterative scan, and relaxed iterative scan.
Configurable sweep of hnsw.ef_search and hnsw.max_scan_tuples parameters.
Seeded random sampling from the database for reproducible query vectors.
External JSONL query-vector input for workload replay.
Optional self-exclusion for corpus-derived queries, enabled in the sample configurations.
Per-sample exact ground truth with index scans disabled in a read-only transaction.
Read-only sampling and ANN transactions with bounded pool size and transaction-local statement/lock timeouts.
EXPLAIN ANALYZE plan capture and validation: verifies an HNSW Index Scan on the expected schema, table, and index.
PostgreSQL 18 JSON-plan compatibility for decimal Actual Rows values.
Catalog validation for HNSW access method, opclass, table ownership, and valid/ready/live index state.
Recall@k, same-rank, and latency statistics with configurable warmup and measured runs.
Randomised strategy order and counterbalanced exact-first/ANN-first execution to reduce systematic cache bias.
Deterministic ID tie-breaking and functional self-exclusion in exact and ANN queries.
CI gate with meaningful failure on missing measurements, query errors, low recall, sequential scans, or absent index scans.
Privacy-safe JSON and Markdown reports with neutral target labels, opt-in detailed evidence, configurable output formats, and enforced 0600 file permissions.
Strict configuration validation and exact vector-dimension checks for external workloads.
Focused tests for configuration, sampling, recall, gates, plan validation, reporting, version checks, and CLI exit codes.
Go 1.21-compatible module graph and Docker build.
Project .gitignore covering the local IDEA.md brief, reports, binaries, secrets, and IDE files.
Version-selectable, ephemeral Docker Compose demo with pgvector and 10,000 distinct deterministic synthetic vectors at 100%, 10%, 1%, and 0.1% filter selectivity levels.
Full Docker benchmark-gate verification on PostgreSQL 16.14, 17.10, and 18.4 with pgvector 0.8.5.
PostgreSQL 18-compatible parent-directory data mount and TCP-only readiness check that avoids the initialization-server race.
Apache-2.0 license.

External download

Download 0.1.0

pgvector-filterbench

About pgvector-filterbench

Release history

More software