oriel.speakers tells voices apart, offline. A speaker encoder (WeSpeaker ResNet34 by default, or ECAPA-TDNN, run by voice-detect.cpp on the shared ggml build) turns a stretch of speech into a short vector, its embedding; two stretches of the same voice give similar vectors. diarize embeds overlapping windows over a recording, clusters them and returns the speaker turns (“speaker 0 from 0.0 to 12.4 s”) with one centroid per speaker, which match compares with voices the app stored before (“that’s Ana”). Pair it with Dictation to label a transcript.

Availability

Turned on by .speakers = true, which needs .llama = true or .whisper = true (they supply the shared ggml build; otherwise: speakers needs llama or whisper). voice-detect.cpp is fetched lazily and compiled into the app. No permission; downloading a model needs network access. The GPU flags are the Local AI ones.

Quick start

build.zig:

const dep = b.dependency("oriel", .{
    .target = target,
    .optimize = optimize,
    .speakers = true,
    .whisper = true, // or .llama = true
});

Zig, on a worker (a long recording takes a while):

const speakers = oriel.speakers;

// A model from speakers.catalog, downloaded with model_download.fetchVerified.
const enc = try speakers.Encoder.load(model_path);
defer enc.deinit();

// 16 kHz mono samples, and where there is speech (whisper's VAD, or the
// whole recording as one region).
var progress: speakers.Progress = .{};
var d = try speakers.diarize(gpa, enc, samples, regions, .{ .progress = &progress });
defer d.deinit(gpa);
for (d.turns) |t| std.log.info("{d:.1}-{d:.1} s: speaker {d}", .{ t.start, t.end, t.speaker });

// Known voices: centroids saved earlier, with enc.family().
for (d.centroids, 0..) |c, i| {
    if (speakers.match(c, known_voices, 0.6)) |who| std.log.info("speaker {d} is {s}", .{ i, names[who] });
}

How it works

1. Windows. Each speech region is cut into window-long windows (3 s) every step (1.5 s), plus one ending at the region’s end. A region shorter than a window gets one window over all of it if it is at least 0.5 s; shorter ones are skipped.
2. Embeddings. Each window becomes an L2-normalized vector (192 values for ECAPA-TDNN, 256 for WeSpeaker). progress.done counts them up to progress.total; progress.cancel stops with error.Canceled.
3. Clustering. Average-linkage agglomerative clustering on cosine similarity: the two most similar clusters merge until the best pair is below threshold (or, with speakers set, until that many are left). Then clusters smaller than min_size windows fold into their nearest big one: a single window always, a larger one only when its centroid is at least fold_above (0.3) similar, so a voice heard for two windows in a short meeting stays a speaker of its own. Clusters whose centroids are above merge_above merge. Past 2048 windows (about 25 minutes of speech) the clustering runs on 2048 evenly spaced ones and every window goes to the nearest resulting centroid, so memory stays at 16 MB.
4. Turns. Overlapping or touching windows of one speaker become one turn; where two speakers’ windows overlap, the boundary is the middle. Speakers are numbered by first appearance.

Short clips favor speakers when the app knows the count: with only a window or two per voice, clustering has little to go on.

Reference

Build options

Functions

An encoder is not thread-safe: voice-detect.cpp keeps one backend for the process, so run one embed or diarize at a time. Its environment variables apply: VOICEDETECT_DEVICE=cpu keeps it on the CPU, VOICEDETECT_THREADS sets its threads.

Models

speakers.catalog, from Hugging Face mudler/voice-detect-gguf, the default first:

Embeddings from different models are not comparable: store enc.family() with a saved voice and compare it before matching.

Similarities measured with Oriel’s tests (one speaker, 3 s halves of a 6 s clip; the second half sped up 1.3x as a stand-in for another voice):

Real voices tell them apart better (LibriSpeech, 3 s halves):

ECAPA’s different people stay too alike to cluster: SpeechBrain trained it on its own features, not the Kaldi fbank voice-detect.cpp feeds every model. WeSpeaker is the default. On three one-minute LibriSpeech conversations (GhostPen’s meetings: windows inside whisper’s segments, clustered, a segment taking its windows’ majority) it found 3, 4 and 1 speakers for 3, 3 and 1, with 10/12, 12/14 and 9/9 segments right.

Licenses

voice-detect.cpp is MIT (NOTICE, licenses/speakers). The models are downloaded by the app at run time, not distributed with Oriel; their licenses are upstream’s (Apache-2.0, CC-BY-4.0: attribution).