Build and update the index
There are two ways to get documents into Ferroseek, and they combine:
- Offline build. A native Rust program turns an NDJSON catalog into Ferroseek’s binary index. Deployed as static files with the Worker, it is read-only: to change it, you rebuild and redeploy.
- The indexing API. With an R2 bucket and the writer Durable Object configured (see Deploy to Cloudflare), the Worker accepts Elasticsearch writes:
PUT /{index},PUT /{index}/_doc/{id},_update,DELETE,_bulkand_refresh. An offline build can be adopted as the starting point of such an index.
The first half of this page is the offline build; the second half, from Write through the API, is the indexing API.
Prepare the input
Section titled “Prepare the input”The builder reads two files, and optionally a third.
A mapping, as the body of an Elasticsearch mappings object. Only flat properties with a type are accepted:
| Mapping type | Stored as |
|---|---|
text |
Analyzed with the standard analyzer, or the field’s analyzer, with positions for phrase queries |
keyword |
Exact values, for filters, facets and sorting |
double, float, half_float |
64-bit floating point |
integer, long, short, byte |
64-bit integer |
boolean |
Boolean |
date |
Milliseconds since the epoch |
A text field may also name an analyzer and a search_analyzer (see Accent-insensitive search). Any other key in a field definition, such as fields or normalizer, is rejected, as is any other type.
Documents as NDJSON, one per line, each with its id and source:
{"_id":"SKU-000001","_source":{"sku":"SKU-000001","name":"Sonece Wireless Crib A100","brand":"Sonece","price":44.99,"in_stock":true,"created_at":"2023-10-09T18:57:56Z"}}keyword and text fields may hold arrays (the sample catalog’s tags does). Numeric, boolean and date fields must be single-valued; the builder rejects arrays there.
Index settings, optional, as the body of an Elasticsearch settings object. Only analysis is read; see Accent-insensitive search.
Run the builder
Section titled “Run the builder”cargo run --release --manifest-path engine/Cargo.toml --bin build-index -- \ --mapping data/mapping.json \ --input data/generated/products-100k.ndjson \ --name products \ --out worker/assets/indexAdd --settings <file> when the mapping names custom analyzers. The root scripts npm run index:build:2k and npm run index:build:100k run exactly the command above for the two sample catalogs. --name is the index name the engine answers to: requests for any other index get 404 index_not_found_exception. One Worker serves one index.
The output directory gets a manifest.json and part files:
{ "format": "ferroseek-index", "version": 4, "name": "products", "doc_count": 100000, "total_bytes": 61101888, "parts": [ { "file": "part-000.bin", "offset": 0, "bytes": 20971520 }, { "file": "part-001.bin", "offset": 20971520, "bytes": 20971520 }, { "file": "part-002.bin", "offset": 41943040, "bytes": 19158848 } ]}The format is binary and usable in place: loading copies each part’s bytes into WebAssembly memory at its offset, without parsing. Parts are at most 20 MiB because Workers static assets accept files up to 25 MiB.
Accent-insensitive search
Section titled “Accent-insensitive search”By default a text field keeps accents: nordvik does not find “Nørdvik”, and cafe does not find “Café”. Elasticsearch solves this with a custom analyzer that adds the asciifolding token filter, declared in the index settings and set per field in the mapping. Ferroseek reads the same two definitions, so one settings file and one mapping behave the same on both.
A complete example. The settings declare an analyzer folding:
{ "analysis": { "analyzer": { "folding": { "type": "custom", "tokenizer": "standard", "filter": ["lowercase", "asciifolding"] } } }}The mapping uses it on name:
{ "properties": { "sku": { "type": "keyword" }, "name": { "type": "text", "analyzer": "folding" }, "brand": { "type": "keyword" } }}With three documents:
{"_id":"1","_source":{"sku":"1","name":"Nørdvik Desk Lamp","brand":"Nørdvik"}}{"_id":"2","_source":{"sku":"2","name":"Café Crème Espresso Cup","brand":"Café Roma"}}{"_id":"3","_source":{"sku":"3","name":"Plain Wooden Spoon","brand":"Atelier"}}Build with --settings:
cargo run --release --manifest-path engine/Cargo.toml --bin build-index -- \ --mapping mapping.json --settings settings.json \ --input products.ndjson --name products --out indexNow {"query": {"match": {"name": "nordvik"}}} finds “Nørdvik Desk Lamp”, with the same score as Elasticsearch gives it for the same definition (1.0226655), and {"match_phrase": {"name": "cafe creme"}} finds the cup. GET /products/_mapping returns the analyzer attribute as Elasticsearch does. Without --settings, a mapping that names folding is rejected: field [name]: analyzer [folding] has not been configured in mappings.
The --mapping file may instead be a whole index-creation body, {"settings": …, "mappings": …}; the builder then takes the settings from it, and --settings must be left out.
What the settings may contain:
| Setting | Accepted |
|---|---|
analysis.analyzer.<name> |
"type": "custom" (or no type) with "tokenizer": "standard" and filter: a list of filter names, applied in order. "type": "standard" with no options is the standard analyzer under another name |
analysis.filter.<name> |
"type": "asciifolding" with preserve_original (true or false, default false), or "type": "lowercase" with no options |
| Filter names in an analyzer | lowercase, asciifolding, and the names defined under analysis.filter |
The order of filters matters, as in Elasticsearch: asciifolding turns some characters into capitals (the small capital ᴀ becomes A), which only a later lowercase lowers. Put lowercase first unless you have a reason not to.
With preserve_original: true, a folded token keeps its original next to it at the same position, so café is indexed as both cafe and café. A query for café then matches both forms, and documents spelled with the accent score higher than those without; a query for cafe scores both spellings alike. The filter must be the last one of its analyzer; a filter after it could turn the two tokens into the same term.
{ "analysis": { "filter": { "folding_keep_original": { "type": "asciifolding", "preserve_original": true } }, "analyzer": { "folding_preserve": { "type": "custom", "tokenizer": "standard", "filter": ["lowercase", "folding_keep_original"] } } }}A field’s search_analyzer, if given, analyzes query text instead of its analyzer; it needs analyzer to be set as well.
Folding follows Lucene’s ASCIIFoldingFilter exactly: a fixed table of 1,242 characters (Latin letters with diacritics, ligatures, small capitals, full-width forms, some punctuation and enclosed numbers), so ø, ß, æ, đ, ł, ș and ş fold to o, ss, ae, d, l, s and s. Characters outside the table, such as Greek or Cyrillic letters, are kept. It is not a transliteration: mueller does not find “Müller” (that folds to muller).
Everything else is rejected when the index is built, with a message naming it: other tokenizers, character filters and token filters (stemmers, stop words, synonyms, ICU), normalizer on keyword fields, search_quote_analyzer, analyzers named default, and any setting other than analysis, including number_of_shards. For example:
build-index: settings: [ferroseek] unsupported: token filter [porter_stem] in analyzer [folding]; only lowercase, asciifolding and filters of those two types defined under analysis.filter are supportedEach text field’s analyzers are recorded in the index itself; the Worker needs no settings file.
Folding applies to text fields only. keyword fields always match exactly, because normalizers are not supported: with brand a keyword, a term filter for Nordvik does not find documents whose brand is “Nørdvik”, and the facet buckets keep the accent.
Size it
Section titled “Size it”The 100k-product sample catalog (about 400 to 500 bytes of JSON per product) gives a 58.3 MiB index, and a Worker that has loaded it uses about 62 MB of its 128 MB. The index holds the original _source of every document as well as the search structures, so the size grows with both the number of products and the size of each one. Check total_bytes in the manifest against the memory your Worker needs for everything else.
Update a static index
Section titled “Update a static index”- Export the current catalog to NDJSON.
- Rebuild the index into the Worker’s assets directory.
- Deploy the Worker (see Deploy to Cloudflare).
Isolates that start after the deployment load the new index on their first request. A static index has no partial update: every change, even of one product, means a full rebuild and redeploy. For frequent changes, use the indexing API below.
For an embedded engine, copy the built index to wherever its assetsIndexSource reads from; the example shop’s npm run shop:index copies worker/assets/index/ to examples/shop-worker/public/search-index/.
Write through the API
Section titled “Write through the API”With the indexing API configured, the Worker serves the Elasticsearch write routes and keeps every index in R2: a base image (an offline build, or the output of a compaction) plus deltas that each refresh adds. One Durable Object per index, the writer, accepts the writes, stores them in its SQLite database and publishes new generations to R2; every isolate that serves searches loads the latest generation.
The routes and their parameters are listed in Routes, the size and count limits in Limits and differences, and the bindings in Deploy with the indexing API. The indexing API is new and has been measured locally only; see Writes, measured locally.
The examples below ran against npm run dev:indexing (local R2 and Durable Objects) on port 4105, with read-write credentials configured, which curl -u writer:write-secret supplies.
Create an index
Section titled “Create an index”curl -s -u writer:write-secret -X PUT http://localhost:4105/shop \ -H 'Content-Type: application/json' -d '{ "settings": { "number_of_replicas": 0, "refresh_interval": "-1" }, "mappings": { "dynamic": "strict", "properties": { "sku": { "type": "keyword" }, "name": { "type": "text" }, "price": { "type": "double" } } }}'{"acknowledged":true,"shards_acknowledged":true,"index":"shop"}The mapping and the analysis settings follow the same rules as the offline builder. Two things differ from the builder because documents arrive one at a time:
dynamic."strict"rejects a document with an unmapped field, as Elasticsearch does.falsekeeps unmapped fields in_sourcewithout indexing them. Withoutdynamic, unmapped fields are rejected too, with a[ferroseek]reason: Ferroseek has no dynamic mapping, so"dynamic": trueis refused when the index is created.- Settings. Besides
analysis:number_of_shards(only1),number_of_replicas(stored; it only changes_shards.totalin write responses) andrefresh_interval(default1s;-1turns scheduled refreshes off).PUT /{index}/_settingschanges the last two.
There is no automatic index creation: a write to an index that does not exist answers 404 index_not_found_exception, as Elasticsearch does with action.auto_create_index: false.
The mapping and the analyzers are fixed once the index exists: PUT /{index}/_mapping is rejected, so no field can be added later. To change them, create another index with the new definition and write the documents into it.
Write documents
Section titled “Write documents”curl -s -u writer:write-secret -X PUT http://localhost:4105/shop/_doc/1 \ -H 'Content-Type: application/json' -d '{"sku":"1","name":"Red Mug","price":7}'{"_index":"shop","_type":"_doc","_id":"1","_version":1,"result":"created","_shards":{"total":1,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}Versions, sequence numbers, results, 409 conflicts, if_seq_no/if_primary_term, external versions, _create, _update with doc, doc_as_upsert and upsert, and deletes of missing documents answer exactly as Elasticsearch 7.17 does; the live comparison checks them response by response.
_bulk takes NDJSON with index, create, update and delete actions. Items are applied in order; a failed item does not stop the others:
printf '%s\n' \ '{"index":{"_id":"3"}}' '{"sku":"3","name":"Green Mug","price":9}' \ '{"update":{"_id":"1"}}' '{"doc":{"price":6.5}}' \ '{"delete":{"_id":"2"}}' \ '{"index":{"_id":"4"}}' '{"sku":"4","colour":"red"}' \| curl -s -u writer:write-secret -X POST 'http://localhost:4105/shop/_bulk?refresh=true' \ -H 'Content-Type: application/x-ndjson' --data-binary @-The fourth item fails (strict_dynamic_mapping_exception, "status":400), the response has "errors":true, and the other three are applied. A /_bulk body may address several indices through _index; each index’s writer applies its own items.
When writes become searchable
Section titled “When writes become searchable”A write is acknowledged once it is stored in the writer’s database. It becomes searchable when a refresh publishes a generation that contains it and the isolate answering the search has loaded that generation:
| You send | The write is searchable |
|---|---|
nothing (refresh absent or false) |
after the next scheduled refresh: within refresh_interval plus build, upload and the freshness window F, about 1.4 s with the defaults in the local measurement |
?refresh=true (or bare ?refresh) |
when the response arrives: the writer publishes now and answers after publishing plus F; the response has "forced_refresh": true |
?refresh=wait_for |
when the response arrives: the writer answers after the next refresh that includes the write, plus F |
POST /{index}/_refresh |
when the response arrives, for everything written before it |
The writer also refreshes on its own when more than 25,000 documents or 16 MiB of source are pending, so documents may become visible earlier than a client with refresh_interval: -1 expects, never later. Publishes are at least one second apart (R2 accepts one write per second to the manifest); refresh requests in that second share one publish.
wait_for differs from Elasticsearch in one way: with refresh_interval: -1, Elasticsearch waits until something else refreshes, possibly forever. Ferroseek waits at most 30 s, then refreshes and answers with "forced_refresh": true. Prefer refresh=true or an explicit _refresh when refreshes are off.
What is promised:
- After the response to a write with
refresh=trueorwait_for, or to_refresh,PUT /{index}orDELETE /{index}, every search started afterwards sees the write, in any isolate. - In the isolate that forwarded the write (the standalone Worker that received it, or the embedding application’s isolate), searches after the response see it immediately: the isolate loads the new generation before it returns the response.
GET /{index}/_doc/{id}(real time, the Elasticsearch default) sees every acknowledged write, refreshed or not.?realtime=falsereads the searchable generation instead.- An isolate never goes back to an older generation, and a generation that shows a write shows every write before it.
What is not promised: without a refresh parameter there is no fixed instant at which a write becomes visible, and two isolates may serve different generations for up to F (1 s by default, FERROSEEK_FRESHNESS_MS). F bounds how old a manifest check may be, not how long loading takes: a very large new delta or a new base takes longer to load.
Real-time GETs go to the index’s writer, which handles about 1,000 requests per second. A read-heavy site that fetches product pages by id can set FERROSEEK_REALTIME_GET=reader (option realtimeGet: "reader") to serve every GET from the searchable generation, or use _search with an ids query.
Adopt an offline build
Section titled “Adopt an offline build”A large catalog is faster to start from an offline build than to send through _bulk. Upload the image as one object, then ask the writer to adopt it:
npm run index:build:100knpm run adopt -- --url http://localhost:4105 --index products --local --persist-to worker/.wrangler/state --env localnpm run adopt (worker/scripts/adopt-base.ts) concatenates the parts into one object, uploads it with wrangler r2 object put (--remote for a deployed bucket), and sends POST /_ferroseek/adopt with {"index", "settings", "mappings", "base_key"}. The writer checks that the mapping and analyzers match the ones recorded in the image, registers every document with _version 1 and _seq_no equal to its position, and publishes generation 1. For the 100k catalog this took 3.7 s locally (0.5 s reading the image, 2.0 s registering the documents). The index then accepts writes like any other.
Compaction
Section titled “Compaction”Every refresh adds a delta. Deltas and the tombstones that mask replaced and deleted documents cost memory in every isolate, so the writer merges them into a new base when they reach half of the delta allowance (the memory budget minus the base), when 20 % of the base is dead, or 12 hours after the first delete. The merge runs in the compactor, a Durable Object in its own Worker script; it does not re-analyse text, and its output is identical to an offline build of the same documents. POST /_ferroseek/compact?index=<name> (write credentials) starts one now and answers when it is committed.
When the writer cannot keep up, writes answer 429 es_rejected_execution_exception, as an overloaded Elasticsearch does; retry after a moment.
The compactor merges up to 80 MiB of input (the base plus its deltas) inside a Worker. The design’s fallback for larger merges, a Cloudflare Container running the native merger, is not built: a larger compaction fails, and once the delta allowance is full, writes answer 429. An index within the default 74 MiB memory budget stays below the ceiling; it matters if you raise FERROSEEK_MEMORY_BUDGET_BYTES. For the 100k sample catalog (a 61 MB base) the delta allowance is about 10 MiB, so compactions start after about 5 MiB of deltas.
After a compaction each isolate swaps in the new base. When the old and the new base do not both fit in memory (the 100k catalog: two 61 MB images in a 128 MB isolate), requests for that index wait while the old base is freed and the new one loads; no request fails. In the local measurement the pause was 284 ms for the 100k base.
Scoring after writes
Section titled “Scoring after writes”Hit sets, totals, facets and field sorts are exact at every generation. Scores are not always identical to Elasticsearch’s between compactions: both engines keep replaced and deleted documents in their term statistics until a merge, but not the same ones (Ferroseek keeps one superseded copy per refresh, Elasticsearch one per write). After a compaction Ferroseek’s scores equal those of an Elasticsearch index loaded fresh with the same documents.
Other index sources
Section titled “Other index sources”The static loader reads the index through a small interface, IndexSource: a manifest() that returns the parsed manifest and a part(file) that returns a stream of one part’s bytes; assetsIndexSource reads static assets. Generations are read through GenerationSource, implemented for R2 by r2GenerationSource. See Library API.