Welcome back to The State of Serene, our monthly note on what shipped and where SereneDB is heading.

July was a big month. Plenty of new features: term-dictionary facets and autocomplete, partial indexes, an online CREATE INDEX that doesn't block writes, real Postgres roles and grants, CREATE SERVER with a ClickHouse connector behind it and a docs search box that runs on SereneDB. Plenty of performance work too: vector search rebuilt on IVF with RaBitQ queries up to 49x faster, filters pushed down into the index scan and FSST+ compression taking 23.8% off URL columns on disk. We also published SearchBench and came out 16.1x ahead of ArangoDB across 92 queries.

All of it is in v26.07.5, so everything below is something you can pull and run today.

What happened in July​

Faceted search and autocomplete without touching a document​

Every search UI needs the same three things next to the results list: a facet sidebar with counts, a type-ahead and a distinct-value list. On a normal database all three mean scanning the matched documents and aggregating them, which is exactly the work you already paid for once when you built the index.

The inverted index already knows every value of every field and how often each one occurs. That's what a term dictionary is. SereneDB now makes it readable from SQL. The ts_dict_* aggregates enumerate a field's dictionary straight off disk, with no document scan and no postings. The list forms come out positionally aligned, so unnest zips them into rows:

SELECT unnest(ts_dict_agg(body))   AS term,
       unnest(ts_dict_count(body)) AS docs,
       unnest(ts_dict_freq(body))  AS freq
FROM   docs_idx
ORDER  BY term;

Most of the time you don't have to spell any of that. On keyword columns the optimizer rewrites ordinary SQL onto the same path: count(DISTINCT col), min, max, array_agg(DISTINCT col) and the plain SELECT col, count(*) ... GROUP BY col facet shape. EXPLAIN prints TsDict: on the scan when it fires and the document lookup disappears from the plan.

The sidebar case gets its own trick. A real facet panel counts four or five dimensions at once and nobody wants five round trips for that, so GROUPING SETS of single-column sets collapse into one dictionary pass:

SELECT category, brand, price_band, count(*)
FROM   products_idx
WHERE  description @@ 'wireless'
GROUP  BY GROUPING SETS ((category), (brand), (price_band));

The plan tells you it worked. No document lookup, no scan node under the aggregate, just the term dictionary being read three ways at once:

╭─ IRESEARCH_SCAN ────────────────────╮
│ Index: products_idx                 │
│ Index Filter:                       │
│ ╭─ Term ─────────────────────╮      │
│ │ Field: description(string) │      │
│ │ Value: wireless            │      │
│ ╰────────────────────────────╯      │
│ TsDict: category, brand, price_band │
│ ~7 rows                             │
╰─────────────────────────────────────╯

That TsDict: line is the tell. Projections trimmed for width, the rest is what EXPLAIN prints.

Facets survive a WHERE on an INCLUDE'd column now too. All of it works the same on view-backed indexes.

Docs: term dictionary and TSQUERY, plus the faceted search, autocomplete, spell correction and tag cloud recipes.

Vector search, rebuilt on IVF​

SereneDB replaced HNSW with a native IVF index built on its own columnstore. Three reasons.

It scales with disk. An HNSW graph wants to live in memory. Every hop is a random access into the graph, so the moment it outgrows RAM your latency falls off a cliff and index size turns into a hardware budget. IVF keeps the coarse centroids resident and leaves everything else on disk, read lazily per probed cluster. The dataset gets to be much bigger than the box.

It's the same machinery as the rest of the index. A cluster is a term. Its members are that term's postings list and the quantized codes ride along in the postings payload stream. IVF isn't a second index bolted onto the side, it's the inverted index doing its usual job with different keys, so it inherits the on-disk format, the segment lifecycle, compaction and recovery instead of reimplementing all four.

Which is why it combines. A vector query is a disjunction over cluster iterators, so it composes with full-text filters, boolean predicates and column filters in the same scan. No candidate-generation step feeding a filtering step in another system. That's what makes filtered kNN and hybrid search one query instead of an integration.

Declare it inline on the column:

CREATE INDEX idx ON t
  USING inverted(pk, emb ivf (metric = 'l2', quant = 'sq8'));

metric is l2, l1, ip or cosine, matched by the <->, <#> and <=> operators. quant is sq8, sq4, pq, rabitq or none, so you can trade precision for a footprint that fits in memory. You don't size the cluster count by hand any more. A multi-level centroid tree gets built from the row count and sdb_ivf_posting_size moves the target leaf size if you want to push on it.

Recall then costs you two session settings at query time rather than a rebuild. sdb_nprobe decides how many clusters a query scans and sdb_rerank_factor sizes the pool that gets re-scored with exact distances, so the same index serves a cheap approximate query and an expensive high-recall one. kNN is still plain SQL. An ORDER BY <distance> LIMIT k gets you the nearest k and a WHERE <distance> < threshold gets you everything inside a radius.

Builds come out around 60x faster than the HNSW index they replace.

RaBitQ was the slowest quantizer. Its 1-bit default was the worst case: one scalar distance call per vector per probed cluster. That's a single PQ4 fast-scan per cluster now, at every bit width. Measured on dbpedia, 100K vectors at 1536 dimensions, l2, k=10:

Recall came out the same or better, build time and index size unchanged.

RaBitQ and PQ both rebuilt their entire scoring pipeline per probed cluster: roughly 10 to 20 µs of setup against 0.07 µs per vector of scanning. The code-dependent half is a lookup table built once per query now, so latency stays nearly flat as nprobe grows. 20% faster at nprobe=128, 24% at 512.

Filtered kNN returns the right rows now. An ORDER BY emb <-> $1 LIMIT k with a WHERE on an INCLUDE'd column pushes the filter into the scan instead of running it above.

Those are SereneDB's own before-and-after numbers, which only tell you it got faster than it was. For where SereneDB lands against somebody else, there are preliminary recall-versus-QPS results against Qdrant at serenedb.github.io/vector-search-benchmark. Treat them as early. Proper vector benchmarks are part of benchmark season and they're coming.

One upgrade note, because it matters: the payload layout changed twice in July. Vector indexes built before v26.07.5 need a rebuild.

Docs: vector search.

CREATE INDEX stopped blocking your writes​

CREATE INDEX ... USING inverted(...) no longer blocks writes. Inserts, updates and deletes run against the table for the whole build and the index that gets published matches the table exactly. The backfill is parallel, so it gets faster with more cores. Deletes that race the build are handled, so nothing removed mid-build survives in the finished index.

Partial indexes handle the case where only some rows are worth indexing:

CREATE INDEX recent_errors ON logs USING inverted(message log_dict)
  WHERE level = 'error';

PostgreSQL semantics all the way down, including NULL counting as non-matching. Rows enter and leave the index as updates move them across the predicate boundary, so membership stays correct without a reindex.

Index tuning is real DDL now:

CREATE INDEX opt_tuned ON opt_t USING inverted(label)
  WITH (segment_memory_max = 67108864, segment_docs_max = 5000,
        compaction_max_segments = 4, compaction_floor_segment_bytes = 4194304);

ALTER INDEX opt_tuned SET (segment_memory_max = 33554432);
ALTER INDEX opt_tuned RESET (segment_memory_max);

ALTER INDEX SET applies live to the running storage, writer limits and task cadences both. pg_class.reloptions lists the persisted set and pg_index.indisvalid / indisready are false while a build is in flight, so anything already speaking Postgres can see what's going on.

Docs: inverted CREATE INDEX.

Filters push all the way into the scan​

"Top 10 relevant errors from the frontend service in the last hour" is two problems glued together: a search and a filter. If the filter runs above the scan you score every match and then throw most of it away, which gets worse the more selective your filter is. That's backwards.

Index scans accept pushed table filters now. Covered columns filter inside the scan at codec level, with a row-group zonemap short-circuit and one decode pass that doubles as materialization, so a filtered top-k only ever collects rows that pass. New BitpackingFilter and AlpFilter skip whole codec groups whose header bounds already refute the predicate, without decoding them at all.

There's one new knob and it's a good one: a static score > c bound is consumed at pushdown. The top-k collector starts at that floor, streaming WAND seeds its threshold from it, the filter vanishes from the plan and EXPLAIN prints Min Score.

The numbers, all measured on our side:

• dense lookups against a parquet-backed view: from 40-66x the cost of a native scan down to 1.6-3.1x
• ORDER BY x ASC LIMIT 5 on 1M rows: 0.219 ms to 0.128 ms, 42% off
• a deliberately adversarial DESC TopN over insert-ordered data: 12 ms to 0.6 ms, 8,480 rows scanned instead of 250,000
• impossible filters fold to an empty result at plan time and never touch storage

None of this is search-specific. Plain analytical scans over columnar storage get the same treatment.

Postgres roles and grants, for real​

Up to now SereneDB was effectively single-user. Fine on a laptop, a non-starter the moment two people or a BI tool share an instance. RBAC is the thing that lets you put it in front of a team. It's PostgreSQL's actual model rather than a lookalike, so GRANT, REVOKE, role membership and column-level privileges behave the way your existing scripts already expect.

CREATE ROLE analyst LOGIN PASSWORD 'secret' VALID UNTIL '2027-01-01 00:00:00+00';
GRANT SELECT (id, title, created_at) ON docs TO analyst;
GRANT analyst TO junior_analyst;

Roles, nested membership through GRANT role TO role, column grants, VALID UNTIL expiry, SET ROLE mid-session and SCRAM over the wire.

The defaults are the part worth reading. The whole policy lives in HBA now instead of a pile of per-connection special cases:

• psql -h 127.0.0.1 -U postgres works with no password on a fresh install and over the unix socket too. First connect stays trivial.
• Superuser over the network requires a password. It used to be silently trusted on 0.0.0.0, which is the kind of default that ends up in an incident report.
• Every non-superuser needs a password, local or remote, same as Postgres.
• SERENEDB_INITIAL_PASSWORD=secret on first boot seeds the superuser password, so a container comes up remote-reachable and authenticated in one step.
• Three superuser loopback trust lines get force-prepended to every ruleset, so a broken HBA config can't lock the admin out of the box.

The pre-RBAC --auth_password, --auth_method and --auth_user flags are gone. --auth_password used to shadow RBAC entirely by pinning the server to one hard-coded user. Per-role catalog passwords are the only auth path now.

There's a whole security chapter now, where every example on the page is a query we actually run: roles, privileges, role membership and client authentication. Statement pages too: CREATE ROLE, GRANT and REVOKE.

Index ClickHouse and Postgres with CREATE SERVER​

The data you want to search is usually already sitting in Postgres or ClickHouse and nobody's excited about maintaining a second copy of it.

CREATE SERVER and CREATE USER MAPPING make a remote system a catalog object that survives restart, so it's infrastructure rather than an ATTACH you have to remember to redo every session. There's a new ClickHouse connector behind it too, speaking the native TCP protocol with columnar blocks and LZ4 or ZSTD, with projection, filter and TopN pushdown, a shared connection pool and read plus write support.

The part that makes this more than plumbing: you can build an inverted index over an attached ClickHouse or Postgres table. Search runs here, matched rows get re-fetched from the remote by its own primary key. Full-text and vector search over a ClickHouse table without moving the ClickHouse table.

Credentials never enter SQL text and pg_foreign_server redacts secret values, which is a deliberate divergence from Postgres.

Docs: CREATE SERVER and external data.

String compression: FSST+ and bitpacked RLE​

On log and URL data the string columns are the storage bill. FSST encodes each dictionary entry whole, so the redundancy between neighbouring entries goes unused: a symbol table is global while shared prefixes are local. FSST+ cleaves each entry into a shared prefix plus a suffix and stores every distinct prefix once, which is exactly the shape URLs, paths and stack traces have. On ClickBench hits_10pct:

Reads came out at parity, within 2%, because the prefixes get FSST-decoded once when the segment initializes. Writes cost about 1.4x.

RLE now bitpacks both its streams, run values and run counts, which is 1.3 to 2.4x smaller on run-heavy data, 19 to 34% cheaper on the write path and 12 to 23% faster to read at short run lengths. It also declines outright when every run is a singleton instead of pretending to help.

Deploy it on Kubernetes, build it on macOS​

There's a Helm chart now, published with every release, so getting SereneDB onto a cluster is one command:

helm install mydb \
  https://github.com/serenedb/serenedb/releases/download/helm-chart-v0.0.5/serenedb-0.0.5.tgz

That gives you a single-node StatefulSet on a persistent volume, client and headless Services, generated superuser credentials and a NetworkPolicy. Upgrades recreate the pod on the same PVC, so serened shuts down gracefully and the new version opens the same datadir. helm test mydb runs a SELECT 1 against the service to confirm it came up.

There's deliberately no replicaCount. SereneDB is a single-machine database and the chart gives you exactly one database pod rather than pretending otherwise.

macOS is a supported build target now, which came out of someone asking whether it builds on Apple silicon. Flex and bison became optional in the same stretch, falling back to a pre-generated parser when they aren't installed, so there's less to set up before building from source.

Benchmark season: SearchBench and the ArangoDB numbers​

We said last month we were calling it benchmark season. Here's the first one.

SearchBench is our open benchmark for search and analytics: 92 queries over OpenTelemetry logs at 100M and 1B records, measuring ingest time, index size and median latency, methodology copied fairly directly from ClickBench. Every adapter is seven shell scripts, so adding an engine is an afternoon.

The first head-to-head is SereneDB against ArangoDB. We started there because ArangoSearch runs on a fork of IResearch, our own search library, taken in 2023. It's the closest thing we'll ever get to measuring ten years of work against its own starting point.

At 100M records on the same machine: ingest and index 20.5x faster, index 5.3x smaller and 16.1x at the median across all 92 queries, 17.2x geometric mean. SereneDB wins 89 of 92. It also answered the full billion with nothing capped and nothing failed, at 46 ms median. ArangoDB's Community Edition caps at 100 GiB of total database size and its 100M index is already 55.3 GiB, so the billion was never on the table for that side.

Which leaves one comparison worth making anyway. Put SereneDB at a billion next to ArangoDB at 100M and SereneDB is still ahead on every task family while carrying ten times the data. Counts land at 20.5 ms against 77 ms, log tailing at 10 ms against 289.5 ms, group-by at 277.5 ms against 1.443 s and joins at 2.1 s against 32.5 s.

Everything is reproducible. Raw per-query results for every engine are at serenedb.com/searchbench. If you think we undertuned something, the adapters are Apache-2.0 and we'd rather take a pull request than defend a guess.

Docs search, powered by SereneDB​

We needed search on our docs. We make a search database. You can see where this went.

docs.serenedb.com and this blog now run hybrid search, BM25 plus vectors, on a SereneDB instance, with streamed AI answers that cite their sources. There's also an MCP endpoint, so an agent can query the docs directly instead of scraping the HTML and guessing.

Then we packaged it, because everyone shipping docs has this problem. Point it at a git repo, a folder, a live website or an S3 bucket and it indexes and re-syncs on its own:

npm install @serenedb/docs-search-react@latest

There's a script-tag embed if you're not on React, a serenedb/docs-search-backend image and a configurator that generates the compose file for you. Run it full-text only or turn on hybrid. AI answers and the MCP server are separate opt-ins.

Docs: Serene Docs Search.

Also shipped​

• You can bind a search condition as a query parameter. TSQUERY is a real value type now, so WHERE col @@ $1 works and an app builds its search query server-side instead of concatenating SQL strings. Prepared statements, arrays of TSQUERY, the operators (||, &&, !!, ##, ^) and 'text'::tokenize('dict') casts all come with it.
• Postgres introspection tells the truth now. pg_depend, pg_rewrite and pg_attrdef were empty stubs, so the ~15 information_schema views that join them silently returned nothing. They're derived from the real dependency graph now and validated byte-for-byte against PostgreSQL 18. Two of them were also handing out oids outside the 32-bit range, which broke any client decoding them as int4. The size functions got the same treatment: pg_relation_size, pg_table_size, pg_total_relation_size, pg_indexes_size, pg_database_size and pg_schema_size return real bytes, where pg_indexes_size used to be hardcoded to zero.
• UNION type end to end: DDL, union_value(), COPY, WAL recovery and use inside inverted indexes.
• Temporal types promoted to first class: TIMESTAMP_S/_MS/_NS, TIMESTAMPTZ_NS, TIMETZ and TIME_NS across the binary wire and index range queries. TIMESTAMPTZ text output follows the session TimeZone the way PG 18 does.
• WAL group commit, with the fsync batched and pipelined across concurrently committing transactions. Commit semantics don't change, only the physical fsync moves. New snapshots stay bounded below the not-yet-durable suffix, so nothing can read a commit a crash could lose.
• sdb_progress, one row per connection with pipeline progress and exact tuple and byte counters for every write statement. pg_stat_activity was an always-empty stub and is now a projection of it, along with every pg_stat_progress_* view. Any query is cancellable with pg_cancel_backend(pid).
• Per-index metrics in sdb_metrics, keyed by relation_id so you can join against pg_class.oid: live docs, segments, files, index size, average commit and consolidation and cleanup times, failure counters.

What to expect in August​

• Sloppy phrase search. A phrase is exact today and the ## gap form pins a fixed distance between two terms. Sloppy phrase gives the whole phrase a slop budget instead, so a query still matches when the text drops a word in the middle.
• Auto-refreshable remote search indices. Today an index over Parquet or Iceberg in a bucket is a snapshot of what was there when you built it. Next month it keeps itself current.
• Logical Postgres replication and triggers. Publications and subscriptions, so an ATTACH'd Postgres can stream changes into SereneDB instead of being re-read.
• Azure. Object storage support beyond S3.
• Faster indexing. More write throughput on top of July's parallel backfill.
• Faster recovery. Less time between starting serened and answering queries after an unclean shutdown.
• More benchmarks. ArangoDB is the first of several. Same rules every time: open adapters, published methodology, raw numbers and the best configuration we can build for every engine on the table.

Kudos​

Some of the most useful input we got in July came from people running SereneDB on their own data and telling us exactly what needs to be improved. A report with a repro is worth as much as a patch and it's usually harder to write.

Thanks to rodion-m for putting inverted indexes under a large code corpus and reporting everything that broke (#970, #971, #972), deadtrickster for a sharp run of reports on vector filtering and BM25 scoring (#961, #962, #964), a-soll for asking about Apple silicon (#883) before SereneDB could build there and emarsden for holding the SQL to the standard (#934, #935). Several of those are fixed in this release. The rest are open with our names on them.

There's also good work in flight we owe review time to: w3lld1 on multi-column search highlights (#932), ivan-digital on an embeddable IResearch-only build target (#976), romanpovol on edge-ngram plus better stemming and normalizing tokenizers (#659) and afigor2701 on pfor encoding (#780).

Want to be in the next one? We tag beginner-friendly work with good first issue, so grab one, ask questions in the issue and we'll get you going.

Everything above is in v26.07.5. Grab it and point it at something. If you'd rather look before you install, the code search demo is live and ⌘K on the docs gets you SereneDB searching SereneDB's documentation. Every raw benchmark result is at serenedb.com/searchbench.

If you like what you see, ⭐ star us on GitHub. Hit a rough edge? Open an issue. The Kudos section above is people who did exactly that.

See you in the next State of Serene.