Skip to content
ferroseekdocumentation homemain, packages at version v0.1.0
Ferroseek on GitHub

How it is verified

Ferroseek claims Elasticsearch compatibility, so the reference is a real Elasticsearch: whatever Elasticsearch 7.17.4 answers is treated as correct, and where the project’s own contract and the server disagree, the server wins. The checks below, all in the repository, hold the engine to that.

The oracle is a single-node Elasticsearch 7.17.4 with security off, started from conformance/docker-compose.yml on port 9200. It gets the same sample catalog, in an index products with one shard, no replicas and the default analyzer.

docker compose -f conformance/docker-compose.yml up -d
npm run es:load # load the catalog into Elasticsearch
npm run conformance:capture # record Elasticsearch's answers as golden files
TARGET_URL=http://localhost:4090 npm run conformance:compare -- --all

The corpus is a directory of request bodies, one per file, and conformance:capture stores the oracle’s response to each. conformance:compare sends the same requests to Ferroseek and compares:

Suite Directory Requests
Main conformance/queries/ 106 searches: queries, filters, aggregations, sorting, paging
Stretch conformance/queries-stretch/ 55: phrases, phrase prefixes, typo tolerance, date_histogram, search_after, _msearch, …
Errors conformance/errors/ 24 malformed or invalid requests, compared by status and error type
Routes defined in conformance/corpus.ts 11: /, _mapping, _doc found and not found, _count, unknown index

Two more suites check custom analyzers (accent folding). Each runs against its own index, which the oracle and a Ferroseek build both serve under the same name, so each needs a Ferroseek built from the same definition:

Suite Index Requests
Folding (--folding) products_folded: the 100k catalog with an asciifolding analyzer on name and description (data/folding/) 36 searches and 2 routes: unaccented queries finding accented names and the reverse, phrases, prefixes, typo tolerance, facets and sorting
Preserve (--preserve) products_preserve: 40 hand-written products with preserve_original on name and the standard analyzer on description (data/preserve/) 19 searches and 2 routes: exact accents outranking folded matches, phrases over stacked terms, multi_match across differently analyzed fields
npm run es:load -- data/generated/products-100k.ndjson --index products_folded \
--mapping data/folding/mapping.json --settings data/folding/settings.json
npm run es:load -- data/preserve/products.ndjson --index products_preserve \
--mapping data/preserve/mapping.json --settings data/preserve/settings.json
npm run conformance:capture -- --folding
cargo run --release --manifest-path engine/Cargo.toml --bin build-index -- \
--mapping data/folding/mapping.json --settings data/folding/settings.json \
--input data/generated/products-100k.ndjson --name products_folded --out /tmp/folded
cargo run --release --manifest-path engine/Cargo.toml --bin serve -- --index /tmp/folded --port 4093 &
TARGET_URL=http://localhost:4093 npm run conformance:compare -- --folding

The loader keeps refresh off while it loads, so that the single segment holds the documents in file order, and checks that it does: Elasticsearch breaks score ties by that internal order and Ferroseek by file order.

The comparison rules:

  • took is ignored. timed_out, _shards and hits.total must be exact.
  • Aggregations must be exact in structure, keys, bucket order and counts; floating-point metric values within 1e-9 relative.
  • Hits must have the same _id sequence, and the same _index, _source and sort values. _score must be within 1e-4 relative. Two adjacent hits may swap only if the oracle’s scores for them differ by less than that tolerance.
  • Every corpus query that sorts by a field ends its sort with sku, so the expected order is total.

conformance/hostile/ holds 65 requests built to hurt: date histograms with overflowing intervals, nested aggregations that would create millions of buckets, hundreds of ranges, very deep or very wide queries, oversized bodies. Each comes with an expectation: the exact error status and type, or, for accepted requests, an upper bound on the response size.

TARGET_URL=http://localhost:4090 npm run conformance:hostile

After the run, the suite sends a sanity query and checks that the engine is still alive and still answers correctly. A native counterpart, engine/tests/hostile_memory.rs, replays the same corpus against the engine with a counting allocator and checks per-request heap and time. It is meant to run in an optimised build:

cd engine
cargo test --profile hostile --test hostile_memory -- --nocapture

Do not point this suite at the Elasticsearch oracle: several requests are accepted there and are expensive by design; one ran the 1 GB heap out of memory.

npm run client:check points the unmodified official Elasticsearch clients, both the 7.17 line and the 8.x line, at Ferroseek and at Elasticsearch, and makes the same calls: info, a search with an aggregation, count, get, msearch, and three failing calls (an unknown document, an unknown query type, an unknown index). It reports any call whose outcome differs between the two servers. This checks that the clients accept Ferroseek at all (product header, content types) and that errors surface as ResponseError with the right status and type.

npm run client:check -- http://localhost:4090 http://localhost:9200

The claim that switching is a connection-settings change is tested end to end. npm run swap:test runs the example shop Worker three times, with byte-identical sources, changing only environment variables:

  1. embedded: no SEARCH_URL; the engine runs inside the shop Worker;
  2. remote Ferroseek: SEARCH_URL=http://localhost:4090, the standalone Worker;
  3. remote Elasticsearch: SEARCH_URL=http://localhost:9200.

It replays 17 listing URLs (searches, category pages, multi-select facets, sorting, paging, a query with no results, an invalid page) against each and requires the shop’s JSON responses to be identical apart from two timing fields. It hashes the application and library sources before each run, to prove that nothing but the environment changed, and then prints warm latency per configuration.

npm run shop:index
npm run swap:test # needs :4090 (or SWAP_EDGE_URL) and :9200 running and :4092 free

npm run live:write runs the full write path — the standalone Worker under wrangler dev --env local, with the writer Durable Object and a local R2 bucket — next to the oracle, from fresh local state each time (--port picks the first port, default 4105; it also uses the next two).

npm run wasm:build && npm run wasm:build:writer && npm run index:build:100k
npm run live:write # all three parts below
npm run live:write -- probe # or one part: probe, catalog, measure

probe. The write sequence of conformance/write-probe.ts (index management, single-document writes with every refresh mode, deletes, updates and upserts, optimistic concurrency, external versions, document validation and coercion, ids, _bulk including its malformed forms, a scoring sequence) is sent to Ferroseek and to Elasticsearch, and every response is compared shape for shape: generated ids mapped by position, uuids and creation dates replaced by placeholders, took ignored, everything else exact, key order included. After every step that refreshes, both are searched — _search with facets, with field sorts and with a text query, _count, _doc and _mget with realtime=false — and compared. The four steps where Ferroseek deliberately differs (a write to a missing index, an array on a numeric field, two malformed bulk action lines) are sent to Ferroseek only and checked against the documented answer, so both sides keep the same documents and sequence numbers.

catalog. The 100k catalog: the offline build is adopted (upload to the local bucket, POST /_ferroseek/adopt); the whole read corpus (conformance:compare --all and the hostile suite) runs against that R2 generation; 4,800 changes go through _bulk (3,000 updates of price, name, brand, tags and stock, 300 second updates, 1,000 deletes, 500 new documents; seeded); after _refresh, the 160 searches of the main and stretch corpus are compared with an Elasticsearch index loaded fresh with the final documents in the same order. Totals, aggregations, field-sorted hits and the hit sets of score-ranked searches must be exact; scores are reported. Then a compaction runs, the search Worker’s base swap is timed while queries keep arriving, and the corpus is compared again, scores included.

measure. Write latency, refresh=true and wait_for latency, the time until a write is visible in a second Worker process reading the same local bucket (a different isolate), whether a search in that isolate after a refresh=true answer sees the write, and the cold load of the 100k index from R2 and from static assets.

The last run’s results:

Check Result
Write-probe responses 145 of 145 identical to Elasticsearch, key order included
Documented deviations 4 of 4 answered as designed
Searches after refresh points 168 of 168 equal in totals, facets, sorts and hit sets; 160 also in scores
Read corpus on the adopted R2 base compare 196 of 196, hostile 66 of 66
100k corpus after 4,800 changes 160 of 160 equal in totals, facets, sorts and hit sets, against an Elasticsearch index loaded fresh; 97 of 160 also in scores (see Known scoring differences)
100k corpus after compaction 160 of 160, scores included
Search in another isolate after refresh=true 20 of 20 saw the write

These figures come from local runs: the Worker, R2 and the Durable Objects ran under wrangler dev on one machine, not on Cloudflare.

npm run conformance:hostile-write starts its own Worker with read-only and read-write credentials and sends 31 hostile requests to the write side: bodies just under and over each size limit, with and without Content-Length (refused by the Worker before anything reaches the writer), missing and wrong credentials, read-only credentials on every kind of write and on the internal /_ferroseek/* routes, multi-index and wildcard deletes, index-less _refresh, and internal headers sent by the client. Each case names the expected status and error type, and some read back afterwards to prove nothing was written.

npm run shop:indexing-test runs the example shop twice, with its static index and with --env indexing, adopts the same index into the second, requires the 17 listing URLs of the swap test to answer identically, then updates a product through /_search-admin/ and checks that the shop’s listing shows it at once.

segmented-corpus (engine/src/bin/segmented_corpus.rs) builds an index the way the writer leaves it between compactions: the 100k catalog as a base, plus deltas and tombstone files made by the writer’s own delta builder, with updates, deletes and additions in every delta. engine/tests/segmented/oracle.ts loads the same operations into Elasticsearch, and engine/tests/segmented/compare.ts runs the main and stretch corpus against both: 161 of 161 searches agree.

npm test runs the TypeScript tests (Vitest), including the comparison rules themselves, the reader library’s generation logic (manifest validation, the freshness rule, budget and swap decisions against a fake engine, routing, the credential split, write forwarding and size limits) and generations loaded into the real WebAssembly engine from an in-memory bucket, and the writer’s Durable Object tests, and npm run engine:test runs the Rust tests: analyzer parity with Elasticsearch’s standard analyzer and with the folding analyzers, query and aggregation behaviour, index validation, a mutation fuzzer and the hostile-memory check.

The folding parity test (engine/tests/folding_parity.rs) uses a fixture captured from Elasticsearch by tools/capture-folding-fixture.ts: every one of the 1,242 characters of the folding table through asciifolding, and awkward texts plus the whole table through the standard tokenizer with each supported filter chain, with token positions. The capture tool also checks that Elasticsearch leaves every other character of the Basic Multilingual Plane unchanged. The table itself is generated from Lucene 8.11.1’s ASCIIFoldingFilter.java by tools/gen-ascii-folding.ts.