Limits and differences
Ferroseek runs inside a 128 MB Worker isolate whose WebAssembly memory already holds the index (about 60 MB for 100,000 products). WebAssembly memory never shrinks, and the engine aborts if an allocation fails. So every request must stay within roughly 16 MiB of extra memory and roughly 100 ms of CPU on a 100k-document index, or be rejected before the work starts, with an Elasticsearch-shaped error.
All limits live in one file, engine/src/limits.rs. They are checked when the request is parsed, or before the work they bound begins. Budgets for clauses, buckets, work and response bytes are shared across the whole request, including all searches of an _msearch.
Request limits
Section titled “Request limits”Where Elasticsearch has the same limit, Ferroseek uses its value and its error. The others are Ferroseek’s own and answer 400 illegal_argument_exception with a reason of the form [ferroseek] … exceeds the limit of [N].
| Limit | Value | Error | Same as Elasticsearch? |
|---|---|---|---|
| Request body size | 256 KiB | 413 illegal_argument_exception |
Elasticsearch’s http.max_content_length is 100 MB, also 413 |
| Estimated heap of the parsed body | 8 MiB | 400 illegal_argument_exception |
Ferroseek only |
| Term clauses per request | 1,024 | 400, too_many_clauses under a shard failure |
indices.query.bool.max_clause_count, but counted per request (see below) |
bool nesting depth |
20 | 400 x_content_parse_exception |
indices.query.bool.max_nested_depth |
Values in one terms query |
65,536 | 400, shard failure |
index.max_terms_count |
Values in one ids query |
65,536 | 400 illegal_argument_exception |
Ferroseek only |
| Buckets per response, all levels | 65,536 | 503 too_many_buckets_exception |
search.max_buckets |
Ranges in one range aggregation |
1,024 | 400 illegal_argument_exception |
Ferroseek only |
Queries in filter and filters aggregations |
256 | 400 illegal_argument_exception |
Ferroseek only |
Length of a date_histogram format |
128 characters | 400 illegal_argument_exception |
Ferroseek only |
| Sort keys | 8 | 400 illegal_argument_exception |
Ferroseek only |
_source include and exclude patterns |
1,024 | 400 illegal_argument_exception |
Ferroseek only |
Searches in one _msearch |
32 | 400 illegal_argument_exception |
Ferroseek only |
| Response body | 8 MiB, also for ?pretty |
400 illegal_argument_exception |
Ferroseek only |
aggregations section of one search |
4 MiB less 64 KiB | 400 illegal_argument_exception |
Ferroseek only |
| CPU work per request | 30,000,000 work units | 400 illegal_argument_exception |
Ferroseek only |
from + size |
10,000 | 400, shard failure |
index.max_result_window |
Some context for the values:
- Body size. A realistic search body is a few KiB. 256 KiB still fits a
termsfilter of about 15,000 SKUs, or a 32-search_msearch. - Parsed size. Parsed JSON can take from about 2 times to about 90 times its text size (
[[1],[1],…]is the worst case), so the byte limit alone does not bound memory. The engine estimates the parsed size with a byte scan before parsing. - Clauses.
matchcounts one clause per analyzed term,multi_matchone per term per field. - Work units. A unit is one posting decoded, one column value scanned or one document visited; natively a unit costs about 1.2 to 1.7 ns, so the budget is roughly 40 to 55 ms. The contract’s faceted benchmark query uses well under one million.
- Response size. A full
size: 10000page of the sample catalog is about 4.9 MB compact and 7.5 MB with?pretty, so the largest page the result window allows still fits.
Write limits
Section titled “Write limits”With the indexing API. Body sizes are checked by the Worker before anything is forwarded to the writer: first the declared Content-Length, then by counting while the body streams through, so a body without a length is cut off at the limit too. The other limits are checked by the writer, per document with the same code the offline builder uses.
| Limit | Value | Error | Same as Elasticsearch? |
|---|---|---|---|
_bulk body |
10 MiB | 413 illegal_argument_exception “[ferroseek] bulk body exceeds the limit of [10485760] bytes” |
http.max_content_length is 100 MB, also 413 |
| Body of any other write | 1 MiB | 413 illegal_argument_exception |
as above |
POST /_ferroseek/adopt body |
64 KiB | 413 illegal_argument_exception |
Ferroseek only |
Items per _bulk |
10,000 | 400 illegal_argument_exception |
Ferroseek only |
_source of one document |
1 MiB | item 400 |
Ferroseek only (a Durable Object SQLite row holds 2 MB) |
_id |
512 bytes | 400 action_request_validation_exception; in _bulk the whole request |
Same |
| Mapped fields per index | 1,000 | 400 illegal_argument_exception |
index.mapping.total_fields.limit |
| Keyword term | 32,766 bytes | item 400 “Document contains at least one immense term” |
Lucene’s term limit |
| Values in one field of one document | 1,024 | item 400 |
Ferroseek only |
| JSON nesting in a document | 64 | item 400 |
Ferroseek only |
| Text tokens / distinct terms in one document | 100,000 / 40,000 | item 400 |
Ferroseek only |
| Pending documents before a forced refresh | 25,000 or 16 MiB of source | none: the writer refreshes | Ferroseek only |
| Deltas per index | 32, and the delta allowance of the memory budget | 429 es_rejected_execution_exception while a compaction catches up |
thread-pool rejections |
refresh=wait_for |
30 s, or 1,000 waiting requests, then a forced refresh | none: answers with "forced_refresh": true |
Elasticsearch waits indefinitely when refreshes are off |
| Compaction input | 80 MiB of base plus deltas | the compaction fails; once the delta allowance is full, writes answer 429 |
Ferroseek only (the design’s Container fallback for larger merges is not built; under the default memory budget an index stays below it) |
| Memory budget of an isolate | 74 MiB of index data across all indices (FERROSEEK_MEMORY_BUDGET_BYTES) |
a write that would make the index outgrow it: 400; an index that does not fit when first requested: 503 index_unavailable_exception “[ferroseek] index [I] needs N bytes, M left in the memory budget” |
Ferroseek only |
The delta allowance is the memory budget minus the base and 2 MiB of headroom: for the 100k sample catalog (a 61 MB base) about 10 MiB of deltas, and the writer starts a compaction at half of it. See Compaction.
Writes per index go through one Durable Object, which handles about 1,000 requests per second; use _bulk for loads.
Rejected features
Section titled “Rejected features”These are answered with 400 illegal_argument_exception and a reason starting [ferroseek] unsupported:.
Search body options: highlight, suggest, explain, version, seq_no_primary_term, stored_fields, docvalue_fields, script_fields, collapse, rescore, min_score, timeout, terminate_after, indices_boost, stats, profile, pit, runtime_mappings, fields, slice, ext.
Queries: fuzzy, wildcard, regexp, dis_max, function_score, query_string, simple_query_string, nested, script, script_score, boosting, combined_fields, geo_distance, more_like_this; named queries (_name); the analyzer parameter of a query; multi_match types most_fields, cross_fields and bool_prefix. Parameter-level details are on Queries.
Aggregations: see Aggregations and facets.
Sorting: custom missing values, mode other than min and max, unmapped_type, numeric_type, format, nested.
APIs: anything not in the route tables: no scroll, no point in time, no cluster management. Without the indexing API, no writes and no index management. With it: no scripts (_update, bulk update), no _update_by_query, _delete_by_query or _reindex, no mapping changes after the index is created, no aliases, no dynamic mapping ("dynamic": true), one shard per index.
Mappings: only text, keyword, numeric, boolean and date fields; numeric, boolean and date fields must be single-valued. The only field options are analyzer and search_analyzer on text fields (search_analyzer needs analyzer). Rejected when the index is built: normalizer (keyword fields match exactly; there are no normalizers), search_quote_analyzer, fields, copy_to and every other option.
Analysis settings (rejected when the index is built or created, see Accent-insensitive search): tokenizers other than standard; character filters; token filters other than lowercase and asciifolding (stemmers, stop words, synonyms, icu_folding and the other ICU analysis); lowercase with a language; an asciifolding filter with preserve_original that is not the last filter of its analyzer; built-in analyzers other than standard (simple, whitespace, the language analyzers, …); analyzers named default, default_search or default_search_quoted; every index setting other than analysis (the indexing API also accepts number_of_shards: 1, number_of_replicas and refresh_interval). GET /{index}/_settings is available only with the indexing API.
Where Ferroseek is stricter
Section titled “Where Ferroseek is stricter”Some requests that Elasticsearch accepts are rejected:
- URL parameters. Only
?prettyis accepted. Elasticsearch also takes search options such assizeorqin the query string; Ferroseek rejects them so that a request means the same thing everywhere. Put options in the body. - Clause count per request. Lucene 8 checks the 1,024-clause limit per
boolquery; Ferroseek counts every clause in the request, acrossquery,post_filterand aggregation filters. Very wide query trees that Elasticsearch accepts can be rejected. - Body size of 256 KiB against Elasticsearch’s 100 MB.
- Every limit marked “Ferroseek only” in the table above.
- Date math. Dates must be ISO 8601 or epoch milliseconds;
now-7dand similar expressions are not parsed. _msearchwithout an index. A search whose header names no index needs exactly one index from the path, the other headers or the Worker; Elasticsearch would search every index. See Routes.- Keyword fields are never accent-folded or otherwise normalized;
normalizeris rejected.
On the write side:
- No automatic index creation. A write to a missing index answers
404 index_not_found_exception; Elasticsearch creates the index (with dynamic mapping) by default. - No dynamic mapping. An unmapped field is rejected unless the mapping says
"dynamic": false;"dynamic": trueis refused at index creation. - Multi-valued numeric, date and boolean fields are rejected (
400 mapper_parsing_exception,[ferroseek] unsupported: multi-valued field …); Elasticsearch accepts arrays there. - Bulk parsing. An action line Elasticsearch silently repairs (a missing closing brace) or reads partly (two actions on one line, of which it takes the first) is rejected with
400 illegal_argument_exception“Malformed action/metadata line [N], …”. - Wildcards and lists in index management.
DELETE /logs-*andDELETE /a,bare refused. refresh=wait_foris bounded at 30 s (see the table).- Body sizes: 10 MiB for
_bulkand 1 MiB otherwise, against Elasticsearch’s 100 MB. - Every write limit marked “Ferroseek only” above.
Known scoring differences
Section titled “Known scoring differences”Between compactions, both Ferroseek and Elasticsearch count replaced and deleted documents in their scoring statistics, but not the same copies: a document replaced several times before one refresh leaves one superseded copy in Ferroseek (one per refresh) and one per write in Elasticsearch. Scores, and the order of hits ranked by score, can then differ slightly; hit sets, totals, facets and field sorts do not. After a compaction, scores equal those of an Elasticsearch index loaded fresh with the same documents.
With typo tolerance, one match query with several terms can expand two different terms to the same dictionary term. For example, on name with fuzziness: "AUTO", "toe tte" expands to {hoe, tee, tote} and {tee, tote}. Lucene merges the duplicate clauses while rewriting the query and, when it does, rebuilds the clause list in HashMap order, which depends on JVM identity hash codes. Ferroseek uses first-occurrence order instead.
The only thing this can change is which document frequency a merged term is scored with, and only when two other terms of the same query also share an expansion. In that case scores can differ slightly from Elasticsearch’s, and the order of hits with close scores can differ with them. The conformance corpus includes queries with shared expansions, such as conformance/queries-stretch/320-fuzzy-two-tokens-shared-expansions.json.