Most data never moves. It just sits as files in object storage: Parquet and JSON in S3, datasets on Hugging Face. A lot of it, in open formats. That's a data lake: cheap to store, but you can't search it on its own.

Search and analytics tools can't read the lake directly; they need their own copy of the data. So you set up a separate system, write code to copy the data into it, keep that copy up to date when the source changes and add a warehouse on the side for the numbers. That's a lot of extra work, just to move data you'd rather leave where it is.

SereneDB lets you skip that. You point it at the files where they are, build an index and search them in place. Because the index also holds your columns, one query can both search and compute statistics, and the data never leaves storage. And it isn't only for code search: the same view-and-index steps work on any Parquet in any bucket.

To show what that looks like we built a code-search system on it, in about 15 minutes: roughly 14,000 programming problems and 11.5 million accepted solutions, about 14 GB of text in all, searchable by keyword, by code substring and by meaning, with live statistics over whatever you searched for. The data is Parquet on Hugging Face; we put SQL views over the files and built indexes on the views. SereneDB speaks the Postgres wire protocol, so psql and any Postgres client just work.

Index remote sources, store only the index​

A view is just a SQL definition. Ours read Parquet straight off Hugging Face, reshaping each source into a common shape and gluing different sources together with UNION ALL:

CREATE VIEW solutions_v AS
  -- Codeforces: ~11.4M accepted submissions, with a natural numeric id
  SELECT submission_id::BIGINT      AS id, problem_id AS task_id, source AS code, ...
  FROM read_parquet('hf://datasets/open-r1/codeforces-submissions/...')
UNION ALL
  -- MBPP reference solutions (no numeric id -> hash the natural key)
  SELECT hash('mbpp/' || task_id)   AS id, 'mbpp/' || task_id AS task_id, code, ...
  FROM read_parquet('hf://datasets/google-research-datasets/mbpp/...')
UNION ALL
  -- HumanEval reference solutions
  SELECT hash('he/' || task_id)     AS id, task_id, prompt || canonical_solution AS code, ...
  FROM read_parquet('hf://datasets/openai/openai_humaneval/...')
UNION ALL
  -- Rosetta Code: per-language source
  SELECT hash('rc/' || task_name)   AS id, task_name AS task_id, code, ...
  FROM read_parquet('hf://datasets/christopher/rosetta-code/...');

CREATE INDEX solutions_idx ON solutions_v
  USING inverted(id, task_id, lang, code code_grams);

solutions_v unions four sources this way; tasks_v unions six. That's seven distinct Hugging Face datasets across the two indexes, each with its own schema, flattened into one searchable relation.

That's the whole ingest. No table is created and nothing is downloaded — there's no wget, no COPY, no "first pull it onto a box" step. The view reads the remote Parquet in place (pulling only the columns and rows it touches), and the index build streams from there. The single thing we store is the index. Adding a source is one more SELECT, and keeping the index current as those files change is what we're building next.

So there's no search stack to stand up: no Elasticsearch, no ETL job, no copy to keep in sync. You point SereneDB at data already sitting in S3, write CREATE INDEX, and you have search.

Keep columns in the index with INCLUDE​

An inverted index maps each term to the documents that contain it: a posting list of doc ids. By default a match hands you those ids, and you'd look the rest of each row up in the source. INCLUDE(...) changes that: the listed columns are stored in the index, right next to the postings, so a result comes back whole, with no lookup into the source.

CREATE INDEX solutions_idx ON solutions_v
  USING inverted(id, task_id, lang, code code_grams)
  INCLUDE (id, task_id, code, code_len, lang, time_ms, memory_kb);

So the code, the language and the judge's time_ms / memory_kb measurements all ride along in the index. EXPLAIN shows it: a single IRESEARCH_SCAN, no TABLE_SCAN, with the INCLUDE columns projected straight from the index:

EXPLAIN SELECT id, task_id, code_len, time_ms, memory_kb
FROM solutions_idx
WHERE code @@ ts_all(ts_tokenize(ARRAY['priority_queue'], 'code_grams_q'));

No TABLE_SCAN anywhere. Once built, the index never reads the Parquet again. The corpus keeps serving with the network unplugged, and statistics over time_ms / memory_kb are pure index reads.

Because those measurements ride in the index, a query can search and aggregate in one pass. The next section does exactly that.

Tokenizers: turning text into terms​

The inverted index is generic; a tokenizer decides what counts as a term. For problem statements that's a normal language analyzer (lowercasing, stemming, stop-words, BM25). For source code we use a sparse n-gram tokenizer, which is what lets you grep code the way GitHub code search does.

It runs a monotonic stack over bigram hashes and picks a sparse set of variable-length n-grams (3 bytes and up) from each line. Indexing emits grams that cover every substring; a query only needs a short covering chain. So a substring search turns into a conjunction: tokenize the query and require every gram to be present.

  "for (int i"  →  [ "for ", "r (in", "(int", "nt i" ]

  WHERE code @@ ts_all(ts_tokenize(ARRAY['for (int i'], 'code_grams_q'))

The index cuts millions of rows down to a few candidates; a LIKE checks them for an exact match. Replace ts_all with ts_any(..., k) and you get fuzzy "roughly this shape" search instead, ranked by BM25.

Queries you can run​

Five ways to ask the same two indexes (solutions_idx, tasks_idx) a question. This is the real SQL behind the search on the site.

Substring grep​

Find accepted solutions that literally contain a code fragment. The sparse-ngram tokenizer turns the fragment into grams, the index returns the rows that contain all of them, ranked by BM25.

SELECT task_id, lang, code_len
FROM solutions_idx
WHERE code @@ ts_all(ts_tokenize(ARRAY['priority_queue'], 'code_grams_q'))
ORDER BY bm25(solutions_idx.tableoid) DESC
LIMIT 20;

Analytics over the matches​

SereneDB does analytics too, over the rows you just searched. Because time_ms and memory_kb live in the index, the matches get aggregated in the same query: p50/p95 run time and memory over the full match set. Search and analytics in one shot, no export to a notebook.

SELECT count(*)                              AS matches,
       round(approx_quantile(time_ms, 0.5))  AS p50_ms,
       round(approx_quantile(time_ms, 0.95)) AS p95_ms,
       round(approx_quantile(memory_kb, 0.5)) AS p50_kb
FROM solutions_idx
WHERE code @@ ts_all(ts_tokenize(ARRAY['from functools import lru_cache'], 'code_grams_q'));

Keyword relevance (BM25)​

Classic full-text ranking over problem statements: the scoring behind most search boxes.

SELECT id, title, round(bm25(tasks_idx.tableoid)::numeric, 2) AS score
FROM tasks_idx
WHERE statement @@ ts_phrase('shortest path')
ORDER BY bm25(tasks_idx.tableoid) DESC
LIMIT 10;

Semantic search​

Match by meaning instead of words: nearest-neighbour over the embedding vectors (HNSW), so "dsu" finds "disjoint set union" with no word in common.

SELECT id FROM task_vec
ORDER BY embedding <=> $query_embedding
LIMIT 10;

Hybrid (reciprocal rank fusion)​

BM25 matches on words and vectors match on meaning; they disagree often enough that you want both. RRF merges two rankings without needing their scores to be comparable: keep each result's rank in each list and add up 1 / (60 + rank), so anything ranked high in either list bubbles up. It's how the site's Explore blends keyword and semantic search behind one slider.

WITH kw AS (
  SELECT id, row_number() OVER (ORDER BY bm25(tasks_idx.tableoid) DESC) AS rank
  FROM tasks_idx WHERE statement @@ ts_phrase('disjoint set') LIMIT 100),
sem AS (
  SELECT id, row_number() OVER (ORDER BY embedding <=> $query_embedding) AS rank
  FROM task_vec ORDER BY embedding <=> $query_embedding LIMIT 100)
SELECT id, sum(1.0 / (60 + rank)) AS score
FROM (SELECT * FROM kw UNION ALL SELECT * FROM sem)
GROUP BY id ORDER BY score DESC LIMIT 10;

Try it yourself​

Open the web UI and search the corpus directly, or point your Claude at the hosted MCP endpoint and let it run the searches as tools:

claude mcp add codesearch --transport http https://codesearch.serenedb.com/mcp

codex mcp add codesearch --url https://codesearch.serenedb.com/mcp

Then ask it something like "find accepted solutions using a monotonic stack and chart their run times" and it calls search_code, code_analytics, explore and a few others for you, all backed by the same indexes.