Welcome back to The State of Serene, our monthly note on what shipped and where SereneDB is heading.

August went into search. New features: sloppy phrase search, a Lucene query parser that reads queries the way Lucene reads them, REINDEX for remote indexes that keep themselves up-to-date, per-branch scoring control, an idf() scorer and Azure blob storage. Plenty of performance work too: the postings read path got rewritten and runs up to 2.9x quicker across 1076 queries, BM25 score computation nearly halved and quant = 'none' vector search became a real quantizer instead of re-reading raw vectors. The catalog also split away from the data, which is what makes index recovery parallel. Outside the repo, SereneDB became a selectable doc store in RAGFlow. And we pointed the whole thing at Dota 2 and shipped Serene Why You Lost.

v26.08.2 is the latest release.

What happened in August​

Sloppy phrase search​

An exact phrase misses the text that drops a word in the middle. "zion machine" does not find "zion sent a machine". Until August the way to ask for that was to build a disjunction of every gap variant by hand.

Slop is a budget for the whole phrase instead:

SELECT id FROM docs_idx WHERE description @@ ts_phrase('zion machine', slop := 3);
SELECT id FROM docs_idx WHERE description @@ ts_phrase('group children')::slop(1);
SELECT id FROM docs_idx WHERE b @@ to_tsquery('"quick fox"~1');

Three spellings for the same thing. slop := sets it on the call, ::slop(N) applies it to a phrase that already exists, including one from phraseto_tsquery. Lucene's "..."~N reaches it through to_tsquery. Declared gaps still work: with ts_phrase('group', 1, 'children', slop := 0) the budget counts deviation from the declared gap rather than from adjacency.

Docs: proximity search with slop and ts_phrase.

Lucene query syntax​

to_tsquery takes a Lucene or Elasticsearch query_string expression. The parser behind it got rebuilt to read one the way Lucene does. What it reads:

Terms are whatever the syntax has not claimed, so café, u.s.a and rock-n-roll are terms. A hyphen or a plus inside a word belongs to the word. A term goes through the analyzer. A prefix, wildcard, fuzziness or range bound gets normalized instead of tokenized, which is the distinction that keeps ["alpha beta" TO gamma] searching from alpha beta rather than from alpha.

Anything SereneDB has no algebra for is refused by name instead of quietly ignored. A query that cannot be read says so instead of answering zero.

Docs: to_tsquery.

Search got faster​

The postings read path, the term-iterator contract and score pruning all got rewritten. Measured over the 1076 queries of search-benchmark-game against the tree before the rewrite:

Computing a BM25 score came out nearly twice as fast on its own.

Where it comes from: two new block encodings that carry no payload at all, so a run of consecutive doc ids decodes from its header and folds into a consumer's mask in one operation. Block consumption moved out of the iterator and into the format layer, so the block disjunction and the deleted-docs mask share one implementation instead of each open-coding it. The in-block bit scatter runs 8 interleaved chains, because consecutive doc ids usually land in the same 64-bit word and serialise the loop on store-to-load forwarding. In-block seek is a branchless fixed-width binary search, so there is no data-dependent branch left to mispredict.

Search over a lakehouse, without a second copy of your data​

Your data in your own bucket stays the source of truth, whether that is an Iceberg table or a directory of Parquet, CSV or JSON. Any engine writes to it, Spark or BigQuery or Flink or SereneDB itself. SereneDB holds only the derived index on local disk: term dictionaries, vector structures, the columns you chose to index. There is no second copy of the corpus, so losing the node means rebuilding an index rather than restoring a database. Columns you did not index are still selectable, materialized from the source for the rows a query actually matched.

The piece missing until August was the barrier between writing and querying.

REINDEX INDEX chunks_idx;

CREATE INDEX rpr_idx ON rpr_v USING inverted(id, body rpr_en)
  WITH (reindex_interval = 100);

One pass compares the source's current committed state against what the index holds, applies the difference and publishes atomically, so readers see the previous complete state or the new one and never a partial index. When it returns, everything committed before it is searchable. That turns a pipeline into three steps: writers commit, one REINDEX, consumers start. reindex_interval runs the pass on a schedule instead. REINDEX INDEX CONCURRENTLY is accepted too, because the pass never blocks readers either way.

How much work a pass does depends on what it is reading:

For a catalog-attached Iceberg table a pass forces a fresh table load even inside the server's staleness window, which is what makes it a barrier rather than a hint.

The whole thing from an empty catalog, including the hybrid queries at the end, is in Search over Iceberg. Reference for the refresh itself: refreshing the index.

Control how a score is evaluated​

Scoring a boolean query is two decisions: how each branch scores and how the branch scores combine. Both are overridable per node now.

::merge(...) sets the combine policy. The default is sum, so a document matching fox and cat gets both contributions added. max takes the best branch instead:

SELECT id, BM25(idx.tableoid) FROM idx
WHERE (body @@ 'fox' OR body @@ 'cat')::merge('max');

It binds to its own node, so nesting means something. max on an inner node still lets the outer node add dog on top. max on the outer node takes the best of all three.

::score(...) overrides the scorer for one subtree while the rest of the query stays on the ORDER BY scorer:

SELECT id, BM25(idx.tableoid) FROM idx
WHERE body @@ 'fox' OR (body @@ 'cat')::score('constant(1)');

::score(NULL) takes a subtree out of scoring entirely. It still selects rows, contributes nothing and asks for no index features, so its postings are read without frequency or norms. That is the shape Elasticsearch spells as a filter clause.

The two compose. A per-branch scorer under a group policy competes on its own terms, so a constant(10) branch survives a max against a BM25 sibling. An unscored branch has nothing to compare, so the other branch wins by default.

max earns its keep on synonyms. Expand car into car OR automobile and the default sum gives a document that happens to use both words two contributions. It then outranks a document that is just as relevant and picked one spelling. Those branches are one concept and max scores them as one.

Both ::score and ::merge ride on the TSQUERY value rather than on the predicate, the same family as ::boost and ::slop, so they reach a group built by ts_any or ts_all as readily as a parenthesised OR:

SELECT id, BM25(idx.tableoid) FROM idx
WHERE body @@ ts_any(['car', 'automobile'])::merge('max');

Riding on the value is also why they survive a prepared statement. WHERE b @@ $1::score('constant(42)') makes the parameter that type, so the modifier arrives with the bound value rather than being lost before the index sees it.

There is also a new idf() scorer: a document scores the inverse document frequency of the matched term alone, with no term frequency and no length normalisation. It works anywhere a scorer works, optimize_top_k = 'idf()' included. It scores columns indexed without a dictionary too. bm25(k1 = 0) is the same formula and used to return zero, which is fixed as well.

Docs: scoring.

Azure, plus credentials for every cloud​

az:// URIs work now in COPY, read_csv, read_parquet and glob, with CREATE SECRET (TYPE azure) for connection strings and scoping. You can build an inverted index over an az:// Parquet or CSV view the same way you would over S3.

Iceberg catalogs got Google service-account auth: CREATE SECRET (TYPE ICEBERG, PROVIDER google) with service-account-key and GCE metadata-server modes, so BigLake catalog tokens renew themselves under a machine identity with no user credentials involved. Verified against a real BigLake catalog with a forced token expiry mid-session.

Object-storage reads got about a fifth quicker in the same stretch: large Azure reads 20% warm and 21% cold, large HTTPS reads 21%.

Docs: AWS, Azure and Google Cloud credentials, plus Iceberg catalog authentication and the BigLake cookbook.

pgstream works against SereneDB, plus a wider Postgres surface​

Point a Postgres logical-replication client at SereneDB and it works now. pgstream is the one we tested with. Three gaps each stopped it cold and silently, all three in how a pg catalog answered a question. All three turned up by diffing its wire capture against the same client aimed at real Postgres.

pg_proc and pg_aggregate are populated. pg_proc listed only macros created in the current database, so 3461 functions existed and exactly one was visible. \df, ORM introspection and information_schema.routines and parameters all saw nothing. Both are filled from a walk over the system catalog now.

Vector search: Panorama pruning and a cheaper descent​

quant = 'none' used to mean no payload at all, so every query re-read raw vectors out of the columnstore to rank them. It is a real quantizer now, built on faiss's Panorama layout: a PCA-rotated basis with per-level suffix norms, which lets the posting scan drop a candidate on an exact Cauchy-Schwarz bound against the running k-th best rather than reading it in full. Same answers, less reading.

Radius queries go through the payload for the same reason, with a constant pruning threshold instead of a raw reranker over the columnstore. Under a lossy quantizer a survivor's reported distance is rescored exactly from the index's own vectors, so the number you get back is the real one.

The quantizer writer and reader protocol is a single block contract now, which is what lets Panorama's batched layout sit alongside PQ and RaBitQ fast-scan groups and flat scalar-quantizer records without each one growing its own hooks.

Then the descent itself got cheaper. Probing N clusters on a deep centroid tree meant expanding every child at every level. Most of that work never reaches a cluster you scan. sdb_ivf_max_search_fanout caps how many children a node expands, defaulting to 16, which decouples descent width from how many clusters you asked for. It raises itself when the cap is too small to supply the requested clusters, so it cannot starve a query. The nprobe setting is now sdb_ivf_search_nprobe. sdb_rerank_factor takes fractional values.

SereneDB is a doc store in RAGFlow now​

RAGFlow is an open-source RAG engine with around 90k stars. Since August SereneDB is one of the document stores you can put behind it, on both the Go and the Python path. DOC_ENGINE=serenedb and it runs. The integration is not ours. Our fellow community member deadtrickster wrote it: 26 files and about 3,900 lines. It merged upstream on 4 August and ships in RAGFlow v0.27.0 onward.

Why it fits is the thing this blog keeps saying. One inverted index carries the scored text column and the IVF vector column, so hybrid retrieval is one SQL statement instead of two systems and a merge step. The connector puts one table per tenant with kb_id as a filter column, which is the Elasticsearch layout rather than a table per dataset, so BM25 statistics stay computed over the whole tenant corpus instead of per knowledge base.

Their numbers, measured on the Python path over a 247,665-chunk multilingual corpus with weighted fusion, against Elasticsearch on the same gold set:

Same MRR, a hair behind on mean rank, roughly 5x quicker. At concurrency 16 it did 514 QPS against Elasticsearch's 241, with a p99 of 42 ms against 108 ms. Building the index over those 247k chunks took 5.6 seconds. Migrating all 247,665 documents through the connector produced zero errors.

Those are the contributor's measurements on their corpus rather than ours. We have not reproduced them. We are quoting them because the methodology is in the PR.

Why You Lost, an analytics showcase built on SereneDB​

Serene Why You Lost takes a finished Dota 2 match, lets you pick which of the ten players is you and tells you what actually cost you the game.

It exists to show what search plus analytics in one engine does to a problem that is not logs. A match is a long event stream with ten people deciding at once and a one-word result hiding all of it. Finding the moments that mattered is a search problem. Judging whether a decision was good is an analytics question over thousands of similar situations. Both run in the same queries, which is the whole argument this database makes, applied to something you can check against your own memory of the game.

A scoreboard says you died six times and finished an item late. It cannot say which of those mattered, what led to it or what the alternative was. Why You Lost answers at three depths. The overview accounts for how the game developed and where momentum shifted. The deep analysis finds the moments that moved your chances of winning and explains why they moved them. The interactive 2D replay puts those moments back in context, with every fight on one timeline and positions, movement and health second by second.

No language model writes the advice. Every comparison comes from real match data and it prints the sample size next to the number, so you can decide for yourself whether to believe it.

It went up as a seven-day test during The International 2026 and it's still online. Whether it stays depends on whether people use it, so go break it. The writeup has the rest.

Search-backed tables take indexes​

A search-backed table is the iresearch columnstore with, as of August, optional indexes on top. WITH (storage = 'search') and the rows go straight into iresearch segments. It is not transactional. What you get back for that is speed.

The indexes are the new half. Until August this was a table you could not index at all, which limited it to whatever a sequential scan could answer. Now it carries inverted indexes with the same syntax a regular table uses. Each indexed field gets a virtual field id that maps back to the real column on materialization, so one column can be indexed several times with different tokenizers while sharing a single store.

The bulk-load path also stopped writing WAL chunk files. It flushes and fsyncs whole index segments and records the segment name in the WAL instead, so recovery adopts the segment when its tick committed and background cleanup wipes it when it did not.

Plain inserts are at parity now. On 10M rows of ClickBench a search-backed table does insert plus commit in 4.89 s where a transactional one takes 5.20 s. It used to be around 30% slower than that. Give both tables inverted indexes on WatchID and URL and it stops being a comparison:

Two limits worth knowing before you reach for it. There is no backfill yet, so indexes have to be added while the table is empty. Only one IVF index per column is allowed. Both are follow-ups rather than design decisions.

Also shipped​

• DDL is transactional now. A CREATE or DROP that fails no longer leaves the catalog half-changed.
• Recovery of inverted indexes runs in parallel and is no longer bound by how much memory the machine has.
• DML against an inverted index runs in parallel too.
• Sequences got faster.
• Fuzzy expansion has its own cap, sdb_levenshtein_max_terms, default 64. It used to take its limit from sdb_scored_terms_limit, which is documented as a scoring-cost knob, so setting that knob changed result sets: over 2001 terms within edit distance 2 of cat, SET sdb_scored_terms_limit = 50 returned exactly 50 rows.

What to expect in September​

• Logical replication subscriptions and triggers. CREATE SUBSCRIPTION so an attached Postgres streams its changes in rather than being re-read. August made SereneDB usable as a replication target; this is the other half.
• Row-level security. Policies per table, on top of July's roles and grants.
• Another round of search performance. The postings work in this issue is not finished. The next round is aiming at 1.5x to 2x.
• Geospatial. A spatial type and index surface.
• Even faster search. We've almost finished a huge rework of our search execution.
• More benchmarks. Still open from last month. Vector benchmarks are the first ones we owe you.

Kudos​

Plenty of this month came from outside the core team.

aksel2904 built sloppy phrase search, both the engine side and the SQL surface. He also contributed the regexp filter we shipped back in June, which makes two features of the search surface now.

deadtrickster did awesome work. Apart from the RAGFlow integration, he also built the tool we should probably have written ourselves. serenedash is a live terminal dashboard for a SereneDB server: storage and the spill split, pool memory against RSS and swap, sessions and how far along their statements are, per-thread CPU, a perf-backed profile and whether index maintenance is keeping up. The same collectors are exposed over MCP, so an agent can read the live server instead of having panels pasted at it. Credentials are optional. A panel that genuinely needs the server says which of "no driver", "no credentials" or "cannot connect" applies rather than drawing a zero and calling it a reading.

On the activity view, one keypress plans a statement that is already running. EXPLAIN has always been something you could type, but aiming it at the query currently burning a core means fetching its text out of pg_stat_activity and quoting it back. On that deployment the text was 68 KB. EXPLAIN does not execute, so it is safe to point at something hung.

He also filed several of the bugs we fixed this month, found by running SereneDB behind a real workload and reporting exactly what broke. Thanks as well to emarsden, who keeps holding the SQL surface to the standard and to deymon-d, who fixed phrase with intervals.

And thank you to everyone with work still in progress: seb-06, w3lld1, ivan-digital, romanpovol and afigor2701.

Want to be in the next one? We tag beginner-friendly work with good first issue, so grab one, ask questions in the issue and we'll get you going.

v26.08.2 is the current release. Grab it and point it at something. If you'd rather look before you install, the code search demo is live and ⌘K on the docs gets you SereneDB searching SereneDB's documentation. Every raw benchmark result is at serenedb.com/searchbench.

If you like what you see, ⭐ star us on GitHub. Hit a rough edge? Open an issue. The Kudos section above is people who did exactly that.

See you in the next State of Serene.