How it is verified

Ferroseek claims Elasticsearch compatibility, so the reference is a real Elasticsearch: whatever Elasticsearch 7.17.4 answers is treated as correct, and where the project’s own contract and the server disagree, the server wins. Four checks in the repository hold the engine to that.

Conformance against Elasticsearch 7.17.4

The oracle is a single-node Elasticsearch 7.17.4 with security off, started from conformance/docker-compose.yml on port 9200. It gets the same sample catalog, in an index products with one shard, no replicas and the default analyzer.

docker compose -f conformance/docker-compose.yml up -d
npm run es:load                 # load the catalog into Elasticsearch
npm run conformance:capture     # record Elasticsearch's answers as golden files
TARGET_URL=http://localhost:4090 npm run conformance:compare -- --all

The corpus is a directory of request bodies, one per file, and conformance:capture stores the oracle’s response to each. conformance:compare sends the same requests to Ferroseek and compares:

Suite	Directory	Requests
Main	conformance/queries/	106 searches: queries, filters, aggregations, sorting, paging
Stretch	conformance/queries-stretch/	55: phrases, phrase prefixes, typo tolerance, date_histogram, search_after, _msearch, …
Errors	conformance/errors/	24 malformed or invalid requests, compared by status and error type
Routes	defined in conformance/corpus.ts	11: /, _mapping, _doc found and not found, _count, unknown index

The comparison rules:

• took is ignored. timed_out, _shards and hits.total must be exact.
• Aggregations must be exact in structure, keys, bucket order and counts; floating-point metric values within 1e-9 relative.
• Hits must have the same _id sequence, and the same _index, _source and sort values. _score must be within 1e-4 relative. Two adjacent hits may swap only if the oracle’s scores for them differ by less than that tolerance.
• Every corpus query that sorts by a field ends its sort with sku, so the expected order is total.

The hostile-request suite

conformance/hostile/ holds 65 requests built to hurt: date histograms with overflowing intervals, nested aggregations that would create millions of buckets, hundreds of ranges, very deep or very wide queries, oversized bodies. Each comes with an expectation: the exact error status and type, or, for accepted requests, an upper bound on the response size.

TARGET_URL=http://localhost:4090 npm run conformance:hostile

After the run, the suite sends a sanity query and checks that the engine is still alive and still answers correctly. A native counterpart, engine/tests/hostile_memory.rs, replays the same corpus against the engine with a counting allocator and checks per-request heap and time. It is meant to run in an optimised build:

cd engine
cargo test --profile hostile --test hostile_memory -- --nocapture

Do not point this suite at the Elasticsearch oracle: several requests are accepted there and are expensive by design; one ran the 1 GB heap out of memory.

The official clients

npm run client:check points the unmodified official Elasticsearch clients, both the 7.17 line and the 8.x line, at Ferroseek and at Elasticsearch, and makes the same calls: info, a search with an aggregation, count, get, msearch, and three failing calls (an unknown document, an unknown query type, an unknown index). It reports any call whose outcome differs between the two servers. This checks that the clients accept Ferroseek at all (product header, content types) and that errors surface as ResponseError with the right status and type.

npm run client:check -- http://localhost:4090 http://localhost:9200

The swap test

The claim that switching is a connection-settings change is tested end to end. npm run swap:test runs the example shop Worker three times, with byte-identical sources, changing only environment variables:

1. embedded: no SEARCH_URL; the engine runs inside the shop Worker;
2. remote Ferroseek: SEARCH_URL=http://localhost:4090, the standalone Worker;
3. remote Elasticsearch: SEARCH_URL=http://localhost:9200.

It replays 17 listing URLs (searches, category pages, multi-select facets, sorting, paging, a query with no results, an invalid page) against each and requires the shop’s JSON responses to be identical apart from two timing fields. It hashes the application and library sources before each run, to prove that nothing but the environment changed, and then prints warm latency per configuration.

npm run shop:index
npm run swap:test   # needs :4090 and :9200 running and :4092 free

Unit and engine tests

npm test runs the TypeScript tests (Vitest), including the comparison rules themselves, and npm run engine:test runs the Rust tests: analyzer parity with Elasticsearch’s standard analyzer, query and aggregation behaviour, index validation, a mutation fuzzer and the hostile-memory check.