oriel.chat answers a conversation with a local LLM (llama.cpp), offline, on every platform, with the tuning done: a catalog of small instruct models downloaded from Hugging Face, the model’s own chat template, replies streamed to the page token by token, a JSON schema the reply must follow, KV-cache reuse across turns, and the CPU or the GPU per device. You give it { role, content } messages and get the reply.

Use it for chat, summaries, rewriting, extraction. For the phone’s built-in model with nothing to download, use System AI; for vision models, embeddings or a sampling loop of your own, use oriel.llama directly (Local AI).

Availability

Turned on by .llama = true (off by default). No permission; downloads need network access. GPU backends come from the Local AI build options.

Quick start

build.zig:

const dep = b.dependency("oriel", .{
    .target = target,
    .optimize = optimize,
    .llama = true,
});

src/main.zig: initialize the module before oriel.main, and expose it with commands of your own. generate blocks, so its command is async; cancel must run while generate is busy, so it isn’t.

const std = @import("std");
const oriel = @import("oriel");
const app = @import("oriel_app");
const chat = oriel.chat;

pub const Events = struct {
    @"chat:download": @FieldType(chat.Events, "chat:download"),
    @"chat:token": @FieldType(chat.Events, "chat:token"),
};

pub const Commands = struct {
    pub const async_commands = .{ "chat_download", "chat_send" };

    pub fn chat_status(_: std.mem.Allocator) chat.Status {
        return chat.status();
    }
    pub fn chat_download(_: std.mem.Allocator, args: struct { model: []const u8 }) !void {
        try chat.download(args.model);
    }
    pub fn chat_send(gpa: std.mem.Allocator, args: struct { messages: []const chat.Message, options: chat.Options = .{} }) !chat.Result {
        return chat.generate(gpa, args.messages, args.options);
    }
    pub fn chat_stop(_: std.mem.Allocator, args: struct { reply: u64 = 0 }) void {
        if (args.reply != 0) chat.cancelReply(args.reply) else chat.cancel();
    }
};

pub fn main(init: std.process.Init) !u8 {
    const data = try oriel.paths.appData(init.gpa, .{ .name = "MyApp" });
    const models_dir = try std.fs.path.join(init.gpa, &.{ data, "models" });
    chat.init(init.io, init.gpa, models_dir); // both strings live as long as the app

    return oriel.main(init, .{ .commands = Commands, .events = Events }, .{
        .app = app.build,
        .window = .{ .title = "Chat", .width = 800, .height = 600 },
    });
}

The page:

const { invoke, listen } = window.oriel;

await invoke("chat_download", { model: "qwen2.5-0.5b" }); // once per device
listen("chat:token", ({ text }) => { reply.textContent += text; });
const r = await invoke("chat_send", {
  messages: [
    { role: "system", content: "Answer briefly." },
    { role: "user", content: "What is Zig?" },
  ],
});
console.log(r.text, r.tokens_per_s);

How it works

sequenceDiagram
  participant P as Page
  participant W as Command worker
  participant C as oriel.chat
  participant L as llama.cpp + ggml
  P->>W: invoke("chat_send", messages)
  W->>C: generate(messages, options)
  C->>L: load model + context (first use)
  C->>C: chat template, drop oldest turns to fit
  C->>L: decode only the tokens not in the KV cache
  loop each token, until EOS, max_tokens or cancel
    L-->>C: logits
    C->>C: sample (grammar when .schema)
    C-->>P: chat:token { text }
  end
  C-->>W: Result
  W-->>P: resolve(Result)
  P->>C: chat_stop (any thread): cancel()

src/modules/chat.zig keeps one model and one context loaded, behind one mutex: a generate holds it from start to end, so replies run one at a time and status, delete and compare wait for the reply in progress.

Which model. .model = "auto" takes the largest catalog model present on the device; a name takes that model (error.NoModel when nothing is downloaded). The model loads on first use and stays loaded; another model, the other backend, or a larger .context reloads it.

Loading follows llama-server and GhostPen:

• On the GPU, every layer is offloaded. A model the GPU can’t hold falls back to the CPU (logged as cannot load <model> on the GPU (...): using the CPU). On the CPU no GPU device is used at all, so the CPU’s repacked ARM kernels get the weights (with a GPU backend’s host buffer in between, ChromeOS’s virtio-gpu made the CPU 14 times slower).
• Phones read the weights into memory instead of mapping the file: models live behind Android’s FUSE layer, where a mapped file the CPU reads every token is slow.
• The context is capped at what the model was trained for, with a q8_0 KV cache and flash attention where the backend has it (else an f16 cache); out of memory, it halves the context, down to 2048 tokens (error.ContextFailed below that).
• A resident threadpool: one thread per core on phones up to 4 (the big cores), every core on desktops, half of them on CPUs with 16 or more (SMT siblings share the math units). Prompts are evaluated in batches of 2048 tokens on desktops, 512 on phones.
• A model loaded on the GPU gets one untimed decode (Vulkan builds its pipelines on first use).

The prompt. Qwen3/3.5 and Gemma 4 templates are rendered by Oriel (src/modules/chat/format.zig), so their reasoning can be switched off in the prompt; every other family goes through llama.cpp’s built-in templates (ChatML, Llama 3, Gemma, Mistral, Phi…), and ChatML when the GGUF’s template isn’t recognized. A conversation that doesn’t fit context - max_tokens drops its oldest turns (a user message and its answer); a leading system message always stays. error.PromptTooLong when the last message alone doesn’t fit.

KV-cache reuse. The context keeps the tokens of the last turn. The next generate evaluates only what follows their common prefix, so a long chat doesn’t get slower per turn (Result.reused_tokens). Earlier Qwen replies keep the empty reasoning block the prompt gave them, so the text is exactly what the cache holds: Qwen3.5 (recurrent layers) and Gemma 4 (sliding-window attention) can only extend their cache.

Sampling. Greedy at temperature = 0; otherwise llama.cpp’s defaults: a light repetition penalty (1.05 over the last 64 tokens: small models loop without it), top-k 40, top_p, min-p 0.05, then the temperature. With .schema, the schema’s grammar binds the answer (after any reasoning); each token is drawn as usual and checked against the grammar, and only when it breaks it does the grammar filter the whole vocabulary, much faster than filtering every token.

Streaming. Every token’s text is appended to the raw output; the part outside the reasoning block goes to the page as chat:token events, cut at whole UTF-8 characters (invalid bytes are replaced).

Cancel. cancel() sets a flag the token loop checks, and llama.cpp’s abort callback checks it during long prompt evaluations. During the reply, generate returns what it has with stop = "cancelled"; during the prompt, it returns error.Cancelled. The flag is cleared when a reply ends, so a cancel() while none runs stops the next generate to start: a Stop pressed before the request reached a worker still counts. Each reply has an id, the reply of its chat:token events; cancelReply(id) stops that reply only while it runs. Tokens can reach the page in a burst after the model finished, so a Stop pressed during that burst would otherwise stop the next reply: send the reply of the tokens showing, and cancel() only before the first one.

Locking. A reply holds its own lock from start to end (replies run one at a time; another generate waits), apart from the lock on the module’s state, which it takes only to load the model. So status, download, delete and compare don’t wait for a reply; status().generating is true while one runs, and delete of the model answering fails with error.Generating.

CPU or GPU. .backend = .auto uses what compare() measured for this model on this device (saved in chat-backends.txt in the models directory); until then, the GPU on desktops and the CPU on phones. compare runs the same short prompt with a 64-token greedy reply on the GPU, then on the CPU; the GPU wins on desktops when it is at least as fast, on phones only when it is 1.3 times faster (phone CPUs vary with heat, and a barely faster GPU isn’t worth its battery).

Memory. unload_on (critical memory pressure by default) frees the model through oriel.lifecycle unless a reply is running (Local AI).

Examples

A chat that streams into the page

The page keeps the conversation and sends it whole each turn; the KV cache makes that cheap.

const { invoke, listen } = window.oriel;
const history = [{ role: "system", content: "You are a helpful assistant." }];
let live = null, replyId = 0;
listen("chat:token", ({ text, reply }) => { if (live) { live.textContent += text; replyId = reply; } });

async function send(text) {
  history.push({ role: "user", content: text });
  live = addBubble("assistant");
  replyId = 0;
  try {
    const r = await invoke("chat_send", { messages: history, options: { max_tokens: 1024 } });
    history.push({ role: "assistant", content: r.text });
    status.textContent = `${r.tokens} tokens, ${r.tokens_per_s.toFixed(1)}/s on ${r.backend}` +
      (r.stop === "length" ? " (cut at max_tokens)" : "");
  } finally {
    live = null;
  }
}
stopButton.onclick = () => invoke("chat_stop", { reply: replyId });

examples/showcase’s Chat tab does this, with a model picker, downloads, Compare, and a microphone that dictates the message (Dictation).

Structured output

const r = try oriel.chat.generate(gpa, &.{
    .{ .role = "system", .content = "Extract the recipe." },
    .{ .role = "user", .content = recipe_text },
}, .{
    .temperature = 0,
    .schema =
    \\{"type":"object","properties":{
    \\  "title":{"type":"string"},
    \\  "ingredients":{"type":"array","items":{"type":"string"}},
    \\  "minutes":{"type":"integer"}},
    \\ "required":["title","ingredients"]}
    ,
});
const Recipe = struct { title: []const u8, ingredients: []const []const u8, minutes: ?i64 = null };
const recipe = try std.json.parseFromSliceLeaky(Recipe, arena, r.text, .{ .ignore_unknown_fields = true });

A reply cut by max_tokens (stop = "length") is incomplete JSON: leave room.

Let a reasoning model think

const r = try oriel.chat.generate(gpa, messages, .{ .model = "qwen3.5-9b", .think = true, .max_tokens = 4096 });
// r.text is the answer only; the thinking counts in r.tokens and max_tokens.

Pick the backend once per device

pub fn chat_compare(_: std.mem.Allocator, args: oriel.chat.Options) !oriel.chat.Comparison {
    return oriel.chat.compare(args); // async command: it runs two replies
}

The page shows each run from chat:compare events ({ backend, tokens_per_s }); from then on .backend = .auto uses the faster one for that model.

A headless check

examples/showcase answers --chat "<message>" from the command line on the desktop, and a follow-up that reuses the KV cache: src/main.zig’s headlessChat is a complete program around chat.generate.

Reference

Functions

The catalog

All speak English, Spanish, German, French and more. The small ones are q4_0, which ggml repacks for ARM’s dotprod and i8mm kernels and the GPU backends run natively. Files are fetched from https://huggingface.co/<repo>/resolve/main/<file>; copying the file into the models directory works too.

Types

pub const Message = struct {
    role: []const u8, // "system", "user" or "assistant"
    content: []const u8,
};

pub const Backend = enum { auto, cpu, gpu };

pub const Model = struct {
    name: []const u8,
    label: []const u8, // "Qwen2.5 1.5B"
    repo: []const u8, // Hugging Face repository
    file: []const u8,
    mb: u32,
    desktop_only: bool = false,
};

Options:

Result (strings allocated with generate’s allocator, model/backend static):

Status:

Comparison: { model, gpu: ?Run, cpu: Run, faster: []const u8 }, with Run = { tokens_per_s: f32, prompt_ms: u64, tokens: u32 }; gpu is null without a GPU (then nothing is remembered).

Events

Emitted to every window; list them in your Events struct to type them in the page (oriel.chat.Events has the payload types).

Errors

Migrating from 0.11

• find(name) !*const Model and present(m) are findModel(name) ?*const Model and modelPresent(m), as in oriel.tts and oriel.dictation.
• cancel() with no reply running now stops the next generate instead of doing nothing.

Troubleshooting

• NoModel. Download one first (download("qwen2.5-0.5b")), or copy the GGUF file named as in the catalog into the models directory.
• Replies are slow on a phone. Run compare once per model: phones start on the CPU, and some GPUs win. Use the 0.5B or 1B model.
• The reply stops mid-sentence. stop = "length": raise max_tokens (and context).
• Stop takes a while to act. cancel() returns at once, so its command needn’t be async; listed in async_commands, it waits for a free worker (they may all be busy with replies, downloads and comparisons).
• The reply after a Stop comes back empty. The Stop reached no reply (the last tokens were still arriving after it ended), and cancel() stopped the next one. Stop with cancelReply and the reply of the tokens showing.
• The log says cannot load ... on the GPU. The model didn’t fit in GPU memory; it runs on the CPU. Use a smaller model or quantization.

Related

• Local AI: llama.cpp, GPU backends, downloads and memory.
• System AI: the phone’s own model.
• Dictation and Text to speech: voice in and out for a chat.
• Commands: async commands and events.