Blog

Run a decision model in the browser tab on WebGPU

What ardana.ai/playground downloads into your tab, how onnxruntime-web runs decider:0.8b on WebGPU or WASM, and why the tab answers exactly like the server.

A small groundhog sits inside a browser window on a laptop screen holding a tiny model box, with an unplugged power cord lying on the floor.

Half a gigabyte, downloaded only after you tap Run

I have opened enough “try it in your browser” demos that start pulling hundreds of megabytes the moment the page loads. Nobody asked me. The tab sits there with a spinner while the laptop fan spins up. For a decision model that is the wrong way to start, because the visitor has to trust what runs in their tab.

So ardana.ai/playground/ runs the model in your tab, but nothing downloads without a tap. The page reads /models.json on load and lists three browser models under “In browser · runs in this tab”: decider:0.8b at int4, the browser default, 467.7 MB; decider:2b at int4, 1.1 GB; and qwen3.5:0.8b at int8, 904.6 MB. We kept qwen3.5:0.8b at int8 because at int4 the spike saw 4 of its 6 preset answers flip. decider:4b and smollm3:3b stay native-only. Even a ?autorun=1 share link waits for a tap when the files are not already in the browser.

Each size is the sum of three files: model.onnx, model.onnx.data and tokenizer.json. Before a first run the top bar says exactly this:

Run downloads decider:0.8b into this tab once: 467.7 MB, kept by this browser. Running it takes 3 to 4 times that in memory.

We measured that “3 to 4 times” in Chrome 154, reading the Performance API’s own memory measurement plus the GPU process. On WebGPU decider:0.8b takes 3.4 times its size: 585 MB in the page and 1.0 GB more in the GPU process. On WASM it is 4.1 times, 1.94 GB. The page’s WebAssembly memory never shrinks, so only a reload frees all of it, and the model note says so.

The order of the download matters too. The tokenizer comes first and plans the request before the weights are fetched, so a request the model would refuse downloads only the tokenizer. If you tap Stop halfway, the page keeps every whole 4 MiB part that arrived and says “Stopped, not answered. The next run resumes the download.” That promise holds on HTTPS only, and that is the next section.

Where the files live, and why HTTPS matters three times

Where do 467.7 MB go once the tab has them? Into Cache Storage. The page opens one cache per model and version, named ardana-browser <model> <version>, and stores each file in it under /v1/browser/<name>/<file>, whichever source sent it. MDN describes the Cache interface as persistent storage for request and response pairs, and its items do not expire unless deleted. The files stay until a newer version replaces them.

The first catch: window.caches exists only in a secure context. An insecure page reads it as undefined, and a browser may withhold it. That is why the banner on a plain http page says the download happens on each visit. The page has nowhere to keep the bytes.

What is a secure context? MDN defines it as a Window or Worker that meets a defined standard of authentication and confidentiality. In practice it means a trustworthy origin: the schemes https, wss and file, or the hosts 127.0.0.0/8, ::1/128 and localhost. So ardana serve on 127.0.0.1 counts, and the same page over plain http on a LAN address does not.

The second catch is WebGPU. MDN lists it as available only in secure contexts, and it is not Baseline yet. An insecure page gets no GPU.

The third catch is threads. The WASM backend runs on threads only in a cross-origin isolated page, where crossOriginIsolated is true. That takes headers, and both sources send them. ardana serve puts these on every response:

cross-origin-opener-policy: same-origin
cross-origin-embedder-policy: require-corp
cross-origin-resource-policy: same-origin

Our landing’s _headers file gives /playground/* the same opener and embedder policies. Without them, the WASM backend gets no threads.

onnxruntime-web on WebGPU, else WASM

The runtime under all of this is onnxruntime-web 1.30.0. We vendor it from the npm tarball into playground/ort/1.30.0/, and the page imports it from its own origin, /ort/1.30.0/, never from a CDN. It is also imported on first use, not at page load. Picking a row under In browser starts the import; a page that only runs server models loads no runtime at all.

There are two builds in that folder. ort.webgpu.bundle.min.mjs with ort-wasm-simd-threaded.asyncify.wasm serves a webgpu session, and ort.wasm.bundle.min.mjs with ort-wasm-simd-threaded.wasm serves a wasm one. Why two? The WebGPU build has no CPU kernel for com.microsoft.GatherBlockQuantized, which is the shared int4 embedding, so a session that runs on CPU needs the plain WASM build.

Which one a visitor gets comes down to one check:

Boolean(globalThis.navigator && navigator.gpu);

A browser with navigator.gpu gets a webgpu session. If the WebGPU session fails, or there is no navigator.gpu, it gets a wasm one. The page tells you which it picked: the banner reads Running decider:0.8b in this tab on WebGPU, or on WASM after a fallback.

The session is created from the bytes the tab already holds:

ort.InferenceSession.create(model, {
	executionProviders: [device],
	externalData: [{ path: 'model.onnx.data', data }],
	logSeverityLevel: 3
});

Both files go in as bytes, never as a URL, so onnxruntime fetches nothing on its own. Each planned row then gets one session.run(feeds, ['logits']), and we read the fp16 logits of the last position at that row’s label ids.

The fallback matters because WebGPU is not everywhere yet. Per caniuse it ships in Chrome 113+, Edge 113+, Opera 99+, Samsung Internet 24+, Safari 26.0+ (partial), Safari on iOS 26.0+ and Chrome on Android 154, with global usage at 87.82%; Firefox keeps it disabled by default through v160. The API is a low-level one, built on D3D12, Vulkan and Metal, and meant to supersede WebGL. Everyone outside that table still gets an answer from the WASM session.

The tab answers with the server’s own bytes

Running a model in a tab is easy. I cared about getting the same answer the server gives, so I did not write a second planner in JavaScript. The crate ardana-engine is ardana-core’s tokenizer, planner and readout, compiled to its own WebAssembly module. It plans a request the way the server does, hands out the rows to decode, and answers with the server’s own bytes.

Refusals too. A request the API refuses with 422 or 413 gets the same status and the same body text in the tab, serialised compact through axum’s Json, exactly as the API does it. The context window is LoadOptions::default().n_ctx, the default the server loads with, so a request too long for it is refused the same way. The test runs_answer_as_the_server pins this down.

Why do the token ids match? The ONNX repositories carry the source’s files, so browser and native read the same ids. The export caps max_position_embeddings at 40,960, which is ardana-core’s default context window. That gives smaller position tables and identical answers.

A run in the tab gives the same Exchange a server run would: the exact body sent, plus the status and body text the API answers that request with.

The exchange panel says Sent · in this tab, and after a run the page says the model answered “in this tab on WebGPU”, or “on WASM”. The latency_ms it shows covers the plan, the decodes and the readout, not the download of the files. So the number is honest about the model and says nothing about your connection.

The same tab from your own ardana serve

The page at ardana.ai/playground/ is the same page ardana serve serves at /, on the same origin as /v1/systemone. There the tab fetches its files from GET /v1/browser/<name>/<file>: model.onnx, model.onnx.data, tokenizer.json, plus profile as JSON, and anything else is 404. The first request pulls the variant into the server’s hub cache, and it stays there while the server runs. Files go out uncompressed with Content-Length, Accept-Ranges: bytes, a 206 on Range and If-Range, an ETag equal to the snapshot commit, and Cache-Control: no-store. The log shows two lines:

ardana serve: pulling the browser variant of {name} ({repo}, {size})
ardana serve: serving the browser variant of {name} ({reference} at {commit})

GET /v1/models grows three fields: x_browser with the download bytes, x_browser_pulled, and x_browser_default. A server with nothing pulled opens on decider:0.8b in the tab.

The standalone page needs no server at all:

cargo xtask build-playground --public-url /playground/ [--hub <url>] [--library <url>]

It reads the library document from --library (default /models.json) at run time and its browser files from --hub (default https://huggingface.co) at this path:

<hub>/<org>/<repo>/resolve/<version>/<file>

It fetches them with Range alone, since the Hub’s ETag is no commit. The landing’s npm run build-playground runs exactly this.

So when does in-tab beat an API call? The onnxruntime-web docs put it in two lines: “Since the data never leaves the device for inferencing, it is a safer method of doing inferencing”, and “If you lose internet connection, the model will still be able to inference.” A request that must not leave the device, or a page that must keep working offline, runs in the tab. Everything else goes to /v1/systemone.

Open ardana.ai/playground/ and tap Run. Or run ardana serve, open http://127.0.0.1:8000, and tell me which backend your browser picked.