Skip to content
ferroseekdocumentation homemain, packages at version v0.1.0
Ferroseek on GitHub

Build and update the index

There are two ways to get documents into Ferroseek, and they combine:

  • Offline build. A native Rust program turns an NDJSON catalog into Ferroseek’s binary index. Deployed as static files with the Worker, it is read-only: to change it, you rebuild and redeploy.
  • The indexing API. With an R2 bucket and the writer Durable Object configured (see Deploy to Cloudflare), the Worker accepts Elasticsearch writes: PUT /{index}, PUT /{index}/_doc/{id}, _update, DELETE, _bulk and _refresh. An offline build can be adopted as the starting point of such an index.

The first half of this page is the offline build; the second half, from Write through the API, is the indexing API.

The builder reads two files, and optionally a third.

A mapping, as the body of an Elasticsearch mappings object. Only flat properties with a type are accepted:

Mapping type Stored as
text Analyzed with the standard analyzer, or the field’s analyzer, with positions for phrase queries
keyword Exact values, for filters, facets and sorting
double, float, half_float 64-bit floating point
integer, long, short, byte 64-bit integer
boolean Boolean
date Milliseconds since the epoch

A text field may also name an analyzer and a search_analyzer (see Accent-insensitive search). Any other key in a field definition, such as fields or normalizer, is rejected, as is any other type.

Documents as NDJSON, one per line, each with its id and source:

{"_id":"SKU-000001","_source":{"sku":"SKU-000001","name":"Sonece Wireless Crib A100","brand":"Sonece","price":44.99,"in_stock":true,"created_at":"2023-10-09T18:57:56Z"}}

keyword and text fields may hold arrays (the sample catalog’s tags does). Numeric, boolean and date fields must be single-valued; the builder rejects arrays there.

Index settings, optional, as the body of an Elasticsearch settings object. Only analysis is read; see Accent-insensitive search.

cargo run --release --manifest-path engine/Cargo.toml --bin build-index -- \
--mapping data/mapping.json \
--input data/generated/products-100k.ndjson \
--name products \
--out worker/assets/index

Add --settings <file> when the mapping names custom analyzers. The root scripts npm run index:build:2k and npm run index:build:100k run exactly the command above for the two sample catalogs. --name is the index name the engine answers to: requests for any other index get 404 index_not_found_exception. One Worker serves one index.

The output directory gets a manifest.json and part files:

worker/assets/index/manifest.json
{
"format": "ferroseek-index",
"version": 4,
"name": "products",
"doc_count": 100000,
"total_bytes": 61101888,
"parts": [
{ "file": "part-000.bin", "offset": 0, "bytes": 20971520 },
{ "file": "part-001.bin", "offset": 20971520, "bytes": 20971520 },
{ "file": "part-002.bin", "offset": 41943040, "bytes": 19158848 }
]
}

The format is binary and usable in place: loading copies each part’s bytes into WebAssembly memory at its offset, without parsing. Parts are at most 20 MiB because Workers static assets accept files up to 25 MiB.

By default a text field keeps accents: nordvik does not find “Nørdvik”, and cafe does not find “Café”. Elasticsearch solves this with a custom analyzer that adds the asciifolding token filter, declared in the index settings and set per field in the mapping. Ferroseek reads the same two definitions, so one settings file and one mapping behave the same on both.

A complete example. The settings declare an analyzer folding:

settings.json
{
"analysis": {
"analyzer": {
"folding": {
"type": "custom",
"tokenizer": "standard",
"filter": ["lowercase", "asciifolding"]
}
}
}
}

The mapping uses it on name:

mapping.json
{
"properties": {
"sku": { "type": "keyword" },
"name": { "type": "text", "analyzer": "folding" },
"brand": { "type": "keyword" }
}
}

With three documents:

products.ndjson
{"_id":"1","_source":{"sku":"1","name":"Nørdvik Desk Lamp","brand":"Nørdvik"}}
{"_id":"2","_source":{"sku":"2","name":"Café Crème Espresso Cup","brand":"Café Roma"}}
{"_id":"3","_source":{"sku":"3","name":"Plain Wooden Spoon","brand":"Atelier"}}

Build with --settings:

cargo run --release --manifest-path engine/Cargo.toml --bin build-index -- \
--mapping mapping.json --settings settings.json \
--input products.ndjson --name products --out index

Now {"query": {"match": {"name": "nordvik"}}} finds “Nørdvik Desk Lamp”, with the same score as Elasticsearch gives it for the same definition (1.0226655), and {"match_phrase": {"name": "cafe creme"}} finds the cup. GET /products/_mapping returns the analyzer attribute as Elasticsearch does. Without --settings, a mapping that names folding is rejected: field [name]: analyzer [folding] has not been configured in mappings.

The --mapping file may instead be a whole index-creation body, {"settings": …, "mappings": …}; the builder then takes the settings from it, and --settings must be left out.

What the settings may contain:

Setting Accepted
analysis.analyzer.<name> "type": "custom" (or no type) with "tokenizer": "standard" and filter: a list of filter names, applied in order. "type": "standard" with no options is the standard analyzer under another name
analysis.filter.<name> "type": "asciifolding" with preserve_original (true or false, default false), or "type": "lowercase" with no options
Filter names in an analyzer lowercase, asciifolding, and the names defined under analysis.filter

The order of filters matters, as in Elasticsearch: asciifolding turns some characters into capitals (the small capital ᴀ becomes A), which only a later lowercase lowers. Put lowercase first unless you have a reason not to.

With preserve_original: true, a folded token keeps its original next to it at the same position, so café is indexed as both cafe and café. A query for café then matches both forms, and documents spelled with the accent score higher than those without; a query for cafe scores both spellings alike. The filter must be the last one of its analyzer; a filter after it could turn the two tokens into the same term.

{
"analysis": {
"filter": {
"folding_keep_original": { "type": "asciifolding", "preserve_original": true }
},
"analyzer": {
"folding_preserve": {
"type": "custom",
"tokenizer": "standard",
"filter": ["lowercase", "folding_keep_original"]
}
}
}
}

A field’s search_analyzer, if given, analyzes query text instead of its analyzer; it needs analyzer to be set as well.

Folding follows Lucene’s ASCIIFoldingFilter exactly: a fixed table of 1,242 characters (Latin letters with diacritics, ligatures, small capitals, full-width forms, some punctuation and enclosed numbers), so ø, ß, æ, đ, ł, ș and ş fold to o, ss, ae, d, l, s and s. Characters outside the table, such as Greek or Cyrillic letters, are kept. It is not a transliteration: mueller does not find “Müller” (that folds to muller).

Everything else is rejected when the index is built, with a message naming it: other tokenizers, character filters and token filters (stemmers, stop words, synonyms, ICU), normalizer on keyword fields, search_quote_analyzer, analyzers named default, and any setting other than analysis, including number_of_shards. For example:

build-index: settings: [ferroseek] unsupported: token filter [porter_stem] in analyzer [folding]; only lowercase, asciifolding and filters of those two types defined under analysis.filter are supported

Each text field’s analyzers are recorded in the index itself; the Worker needs no settings file.

Folding applies to text fields only. keyword fields always match exactly, because normalizers are not supported: with brand a keyword, a term filter for Nordvik does not find documents whose brand is “Nørdvik”, and the facet buckets keep the accent.

The 100k-product sample catalog (about 400 to 500 bytes of JSON per product) gives a 58.3 MiB index, and a Worker that has loaded it uses about 62 MB of its 128 MB. The index holds the original _source of every document as well as the search structures, so the size grows with both the number of products and the size of each one. Check total_bytes in the manifest against the memory your Worker needs for everything else.

  1. Export the current catalog to NDJSON.
  2. Rebuild the index into the Worker’s assets directory.
  3. Deploy the Worker (see Deploy to Cloudflare).

Isolates that start after the deployment load the new index on their first request. A static index has no partial update: every change, even of one product, means a full rebuild and redeploy. For frequent changes, use the indexing API below.

For an embedded engine, copy the built index to wherever its assetsIndexSource reads from; the example shop’s npm run shop:index copies worker/assets/index/ to examples/shop-worker/public/search-index/.

With the indexing API configured, the Worker serves the Elasticsearch write routes and keeps every index in R2: a base image (an offline build, or the output of a compaction) plus deltas that each refresh adds. One Durable Object per index, the writer, accepts the writes, stores them in its SQLite database and publishes new generations to R2; every isolate that serves searches loads the latest generation.

The routes and their parameters are listed in Routes, the size and count limits in Limits and differences, and the bindings in Deploy with the indexing API. The indexing API is new and has been measured locally only; see Writes, measured locally.

The examples below ran against npm run dev:indexing (local R2 and Durable Objects) on port 4105, with read-write credentials configured, which curl -u writer:write-secret supplies.

curl -s -u writer:write-secret -X PUT http://localhost:4105/shop \
-H 'Content-Type: application/json' -d '{
"settings": { "number_of_replicas": 0, "refresh_interval": "-1" },
"mappings": { "dynamic": "strict", "properties": {
"sku": { "type": "keyword" }, "name": { "type": "text" }, "price": { "type": "double" } } }
}'
{"acknowledged":true,"shards_acknowledged":true,"index":"shop"}

The mapping and the analysis settings follow the same rules as the offline builder. Two things differ from the builder because documents arrive one at a time:

  • dynamic. "strict" rejects a document with an unmapped field, as Elasticsearch does. false keeps unmapped fields in _source without indexing them. Without dynamic, unmapped fields are rejected too, with a [ferroseek] reason: Ferroseek has no dynamic mapping, so "dynamic": true is refused when the index is created.
  • Settings. Besides analysis: number_of_shards (only 1), number_of_replicas (stored; it only changes _shards.total in write responses) and refresh_interval (default 1s; -1 turns scheduled refreshes off). PUT /{index}/_settings changes the last two.

There is no automatic index creation: a write to an index that does not exist answers 404 index_not_found_exception, as Elasticsearch does with action.auto_create_index: false.

The mapping and the analyzers are fixed once the index exists: PUT /{index}/_mapping is rejected, so no field can be added later. To change them, create another index with the new definition and write the documents into it.

curl -s -u writer:write-secret -X PUT http://localhost:4105/shop/_doc/1 \
-H 'Content-Type: application/json' -d '{"sku":"1","name":"Red Mug","price":7}'
{"_index":"shop","_type":"_doc","_id":"1","_version":1,"result":"created","_shards":{"total":1,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}

Versions, sequence numbers, results, 409 conflicts, if_seq_no/if_primary_term, external versions, _create, _update with doc, doc_as_upsert and upsert, and deletes of missing documents answer exactly as Elasticsearch 7.17 does; the live comparison checks them response by response.

_bulk takes NDJSON with index, create, update and delete actions. Items are applied in order; a failed item does not stop the others:

printf '%s\n' \
'{"index":{"_id":"3"}}' '{"sku":"3","name":"Green Mug","price":9}' \
'{"update":{"_id":"1"}}' '{"doc":{"price":6.5}}' \
'{"delete":{"_id":"2"}}' \
'{"index":{"_id":"4"}}' '{"sku":"4","colour":"red"}' \
| curl -s -u writer:write-secret -X POST 'http://localhost:4105/shop/_bulk?refresh=true' \
-H 'Content-Type: application/x-ndjson' --data-binary @-

The fourth item fails (strict_dynamic_mapping_exception, "status":400), the response has "errors":true, and the other three are applied. A /_bulk body may address several indices through _index; each index’s writer applies its own items.

A write is acknowledged once it is stored in the writer’s database. It becomes searchable when a refresh publishes a generation that contains it and the isolate answering the search has loaded that generation:

You send The write is searchable
nothing (refresh absent or false) after the next scheduled refresh: within refresh_interval plus build, upload and the freshness window F, about 1.4 s with the defaults in the local measurement
?refresh=true (or bare ?refresh) when the response arrives: the writer publishes now and answers after publishing plus F; the response has "forced_refresh": true
?refresh=wait_for when the response arrives: the writer answers after the next refresh that includes the write, plus F
POST /{index}/_refresh when the response arrives, for everything written before it

The writer also refreshes on its own when more than 25,000 documents or 16 MiB of source are pending, so documents may become visible earlier than a client with refresh_interval: -1 expects, never later. Publishes are at least one second apart (R2 accepts one write per second to the manifest); refresh requests in that second share one publish.

wait_for differs from Elasticsearch in one way: with refresh_interval: -1, Elasticsearch waits until something else refreshes, possibly forever. Ferroseek waits at most 30 s, then refreshes and answers with "forced_refresh": true. Prefer refresh=true or an explicit _refresh when refreshes are off.

What is promised:

  1. After the response to a write with refresh=true or wait_for, or to _refresh, PUT /{index} or DELETE /{index}, every search started afterwards sees the write, in any isolate.
  2. In the isolate that forwarded the write (the standalone Worker that received it, or the embedding application’s isolate), searches after the response see it immediately: the isolate loads the new generation before it returns the response.
  3. GET /{index}/_doc/{id} (real time, the Elasticsearch default) sees every acknowledged write, refreshed or not. ?realtime=false reads the searchable generation instead.
  4. An isolate never goes back to an older generation, and a generation that shows a write shows every write before it.

What is not promised: without a refresh parameter there is no fixed instant at which a write becomes visible, and two isolates may serve different generations for up to F (1 s by default, FERROSEEK_FRESHNESS_MS). F bounds how old a manifest check may be, not how long loading takes: a very large new delta or a new base takes longer to load.

Real-time GETs go to the index’s writer, which handles about 1,000 requests per second. A read-heavy site that fetches product pages by id can set FERROSEEK_REALTIME_GET=reader (option realtimeGet: "reader") to serve every GET from the searchable generation, or use _search with an ids query.

A large catalog is faster to start from an offline build than to send through _bulk. Upload the image as one object, then ask the writer to adopt it:

npm run index:build:100k
npm run adopt -- --url http://localhost:4105 --index products --local --persist-to worker/.wrangler/state --env local

npm run adopt (worker/scripts/adopt-base.ts) concatenates the parts into one object, uploads it with wrangler r2 object put (--remote for a deployed bucket), and sends POST /_ferroseek/adopt with {"index", "settings", "mappings", "base_key"}. The writer checks that the mapping and analyzers match the ones recorded in the image, registers every document with _version 1 and _seq_no equal to its position, and publishes generation 1. For the 100k catalog this took 3.7 s locally (0.5 s reading the image, 2.0 s registering the documents). The index then accepts writes like any other.

Every refresh adds a delta. Deltas and the tombstones that mask replaced and deleted documents cost memory in every isolate, so the writer merges them into a new base when they reach half of the delta allowance (the memory budget minus the base), when 20 % of the base is dead, or 12 hours after the first delete. The merge runs in the compactor, a Durable Object in its own Worker script; it does not re-analyse text, and its output is identical to an offline build of the same documents. POST /_ferroseek/compact?index=<name> (write credentials) starts one now and answers when it is committed.

When the writer cannot keep up, writes answer 429 es_rejected_execution_exception, as an overloaded Elasticsearch does; retry after a moment.

The compactor merges up to 80 MiB of input (the base plus its deltas) inside a Worker. The design’s fallback for larger merges, a Cloudflare Container running the native merger, is not built: a larger compaction fails, and once the delta allowance is full, writes answer 429. An index within the default 74 MiB memory budget stays below the ceiling; it matters if you raise FERROSEEK_MEMORY_BUDGET_BYTES. For the 100k sample catalog (a 61 MB base) the delta allowance is about 10 MiB, so compactions start after about 5 MiB of deltas.

After a compaction each isolate swaps in the new base. When the old and the new base do not both fit in memory (the 100k catalog: two 61 MB images in a 128 MB isolate), requests for that index wait while the old base is freed and the new one loads; no request fails. In the local measurement the pause was 284 ms for the 100k base.

Hit sets, totals, facets and field sorts are exact at every generation. Scores are not always identical to Elasticsearch’s between compactions: both engines keep replaced and deleted documents in their term statistics until a merge, but not the same ones (Ferroseek keeps one superseded copy per refresh, Elasticsearch one per write). After a compaction Ferroseek’s scores equal those of an Elasticsearch index loaded fresh with the same documents.

The static loader reads the index through a small interface, IndexSource: a manifest() that returns the parsed manifest and a part(file) that returns a stream of one part’s bytes; assetsIndexSource reads static assets. Generations are read through GenerationSource, implemented for R2 by r2GenerationSource. See Library API.