Skip to content
ferroseekdocumentation homemain, packages at version v0.1.0
Ferroseek on GitHub

Library API

Two TypeScript packages live in this repository. They are workspace packages, not published to npm; see Embed the engine in a Worker for how to depend on them.

Package Directory Purpose
@ferroseek/engine packages/ferroseek/ The WebAssembly engine and its loader, auth and response handling
@ferroseek/client packages/search-client/ The official Elasticsearch client, connected by settings only
function createFerroseek(options: FerroseekOptions): Ferroseek;
interface FerroseekOptions {
readonly wasm: WebAssembly.Module;
readonly source?: IndexSource; // a static index — or:
readonly generations?: GenerationSource; // indices published to R2 by the writer
readonly writer?: DurableObjectNamespace; // env.FERROSEEK_WRITER; needs generations
readonly auth?: AuthConfig; // read-only credentials
readonly writeAuth?: AuthConfig; // read-write credentials
readonly freshnessMs?: number; // F, default 1,000
readonly memoryBudgetBytes?: number; // default 74 MiB
readonly realtimeGet?: "writer" | "reader";
readonly verifyHashes?: "all" | "small" | "none";
readonly diagnostics?: boolean;
readonly onError?: (error: unknown, phase: "load" | "request" | "generation" | "write") => void;
}
interface Ferroseek {
fetch(request: Request, ctx?: { waitUntil(p: Promise<unknown>): void }): Promise<Response>;
ready(): Promise<LoadStats>;
}
interface LoadStats {
readonly loadMs: number;
readonly indexBytes: number;
readonly docCount: number;
}

Set exactly one of source and generations.

With source, fetch answers the search routes listed in Routes from one immutable index. It loads the index on first use; concurrent first calls share one load, and a failed load is retried by the next call. ready() loads it now and resolves with its statistics.

With generations, each index is loaded on first use and its manifest is checked again at most once per freshnessMs: a request answered within F/2 of the last check is served at once, one within F is served and checks in the background (through ctx.waitUntil, so pass the Worker’s ctx), a later one waits for the check. A new generation that only adds deltas is loaded beside the active one while requests keep being served; one that cannot fit beside it (a new base after compaction) is swapped in place, and requests for that index wait. All indices share memoryBudgetBytes. If R2 cannot be reached, the loaded generation keeps being served and onError(err, "generation") is called; an index that is not loaded yet answers 503 index_unavailable_exception. Responses served while a newer generation is known but not loaded carry x-ferroseek-stale-ms. verifyHashes (default "small") checks the SHA-256 of tombstone files and of segments up to 8 MiB against the manifest.

With writer, write, index-management and /_ferroseek/* routes are forwarded to the index’s writer Durable Object (see Write routes); without it they answer 400 “writes are not enabled”. When a write answer names a newer generation, the call loads it before resolving, so this isolate’s next search sees the write.

auth holds read-only credentials and writeAuth read-write ones; requests are checked before anything else. Request bodies of search routes are read with a 256 KiB cap; write bodies are streamed to the writer and cut off at their limit.

With diagnostics, responses carry x-engine-time-us, x-wasm-memory-bytes, on a request that waited for a load x-index-load-ms, and with generations also x-ferroseek-served (index:generation) and x-ferroseek-swap (index:kind:generation:ms:footprint of the last swap).

interface GenerationSource {
readonly description: string;
manifest(index: string, ifNoneMatch?: string): Promise<ManifestRead | "unchanged" | null>;
object(index: string, uuid: string, key: string, range?: { offset: number; length: number }): Promise<ReadableStream<Uint8Array>>;
}
function r2GenerationSource(bucket: R2Bucket, prefix?: string): GenerationSource; // prefix default "ferroseek/"
function generationOptionsFromEnv(env: IndexingEnv): Pick<FerroseekOptions, "generations" | "writer" | …> | undefined;

r2GenerationSource reads <prefix>indices/<index>/manifest.json with a conditional get (etagDoesNotMatch), validates it, and reads segment and tombstone objects under <prefix>indices/<index>/<uuid>/, large ones as concurrent 20 MiB ranges. generationOptionsFromEnv builds the options from the bindings FERROSEEK_BUCKET and FERROSEEK_WRITER and the variables FERROSEEK_PREFIX, FERROSEEK_FRESHNESS_MS, FERROSEEK_MEMORY_BUDGET_BYTES, FERROSEEK_REALTIME_GET and FERROSEEK_VERIFY_HASHES; it returns undefined when no bucket is bound. writeAuthFromEnv(env) reads FERROSEEK_WRITE_USERNAME, FERROSEEK_WRITE_PASSWORD and FERROSEEK_WRITE_API_KEY as authFromEnv reads the read-only ones.

The Durable Object classes come from @ferroseek/writer: export FerroseekWriter (and, where the compactor runs, FerroseekCompactor) from the Worker’s main module, after setWriterWasm(module) with packages/ferroseek-writer/wasm/ferroseek-writer.wasm (built by npm run wasm:build:writer).

import wasm from "@ferroseek/engine/wasm";

The compiled engine as a WebAssembly.Module, for Wrangler’s CompiledWasm rule. It is a separate entry point so that the main entry stays importable in Node and in tests. The file it imports, packages/ferroseek/wasm/ferroseek.wasm, is produced by npm run wasm:build.

interface IndexSource {
readonly description: string;
manifest(): Promise<unknown>;
part(file: string): Promise<ReadableStream<Uint8Array>>;
}
function assetsIndexSource(assets: AssetsFetcher, basePath?: string): IndexSource;
interface AssetsFetcher {
fetch(request: Request): Promise<Response>;
}

assetsIndexSource(env.ASSETS, "index") reads index/manifest.json and the part files through a Workers static-assets binding. basePath defaults to "index" and may only contain letters, digits, ., _, - and /. The loader validates the manifest, then streams each part into WebAssembly memory at its offset and checks its length.

interface AuthSettings {
readonly username?: string;
readonly password?: string;
readonly apiKey?: string; // base64("id:api_key")
}
interface AuthConfig {
readonly basic?: { readonly username: string; readonly password: string };
readonly apiKey?: { readonly id: string; readonly key: string };
}
function authFromEnv(env: Readonly<Record<string, unknown>>): AuthConfig | undefined;
function parseAuth(settings: AuthSettings, labels?: Readonly<Record<keyof AuthSettings, string>>): AuthConfig;
function checkAuth(request: Request, config: AuthConfig): Promise<Response | null>;

authFromEnv reads FERROSEEK_USERNAME, FERROSEEK_PASSWORD and FERROSEEK_API_KEY, and returns undefined when none is set. parseAuth validates settings and throws FerroseekConfigError on an inconsistent combination. checkAuth returns null for an authenticated request, otherwise the 401 response to send.

Thrown when the engine is configured inconsistently: missing or malformed settings.

function createSearchClient(config: SearchClientConfig): SearchClient; // the @elastic/elasticsearch Client
interface SearchClientConfig {
readonly node: SearchNode;
readonly auth?: SearchAuth;
readonly options?: SearchClientOptions;
}
type SearchNode = string | URL | InProcessNode;
interface InProcessNode {
fetch(request: Request): Promise<Response>;
}
type SearchAuth =
| { readonly username: string; readonly password: string }
| { readonly apiKey: string };

SearchClientOptions is the official client’s ClientOptions without node, nodes, auth, Connection, cloud and ConnectionPool.

function searchConfigFromEnv(
env: Readonly<Record<string, unknown>>,
embedded: () => InProcessNode,
): ResolvedSearchConfig;
interface ResolvedSearchConfig {
readonly mode: SearchMode; // "embedded" | "remote"
readonly config: SearchClientConfig;
}

Reads SEARCH_URL, SEARCH_USERNAME, SEARCH_PASSWORD and SEARCH_API_KEY. embedded is called only when SEARCH_URL is unset. See Connect with the client.

  • SearchConfigError: thrown for missing, malformed or inconsistent connection settings.
  • errors: the official client’s error classes (ResponseError, ConnectionError, TimeoutError, …).
  • estypes: the official client’s request and response types.