Voices and speakers

Three different things that are easy to confuse:

• A speaker of a recording is the voice number inside one recording: segments[].speaker = 0, 1, 2… In the next recording the same person may sit under another number.
• A project voice is a person's permanent number across all recordings of the project: #100, #101… It appears when "Recognise by voice" is on.
• A name is what you called a project voice: "Azimbek (manager)". A name is optional: an unnamed voice is recognised too.

How voices get numbers

When a recording is processed with "Recognise by voice" on:

1. A voice print is computed for every speaker of the recording.
2. The print is compared with the project's voices. A match: the speaker gets that voice's number, status matched.
3. No match, and the speaker has 10 seconds of speech or more: a new project voice is registered with the next number, status new.
4. Less than 10 seconds of speech: status unknown, no number. A print from a few seconds is unreliable, and we do not open a voice on it.

The link between speakers of the recording and project voices is in voices:

"voices": [
  {"label": "SPEAKER_1", "speaker": 0, "voice_id": 100, "name": null, "status": "new"},
  {"label": "SPEAKER_2", "speaker": 1, "voice_id": 101, "name": "Azimbek", "status": "matched", "score": 0.97}
]

speaker is the number from segments[].speaker, voice_id the project voice number, speaker_id its identifier for the speakers API.

How long a voice lives

Not every voice heard once is worth keeping forever, so a voice has a kind, visible in kind:

Kind	What it is	How long it lives
candidate	Heard in one recording	30 days from the last time it was heard, then it goes
recurring	Heard in three recordings or more	Kept until deleted
named	A voice someone named	Kept until deleted

A candidate heard again refreshes last_seen_at and the count starts over; on the third recording it becomes recurring and the expiry is lifted. A name lifts it at once: name a voice and it stays. So a colleague's voice is better named the first time round than left to a third recording.

expires_at says when the voice will go; for recurring and named it is empty.

Give a name

The cabinet's "Speakers" tab lists every project voice: named ones and "Voice #100" ones recognised automatically. "Name it" turns the number into a name. Through the API: PATCH /v1/speakers/{id} with {"name": "…"}.

Mark pieces as one person

Sometimes speaker separation gets it wrong: one person lands in two numbers, or the other way round. A recording's result has a "Mark one person's pieces" mode: select the utterances that surely belong to one person and say who it is: an existing voice or a new name. A print is computed from the selected utterances and added to that voice. The more speech selected, the more reliable; the cabinet warns below 10 seconds.

Through the API: POST /v1/speakers/{id}/enroll with {"job_id": "…", "segment_indexes": [3, 5, 8]}. The utterance numbers are enough: the print is computed on our side from the stored recording. The prints themselves are not in the result — they are biometric data and hundreds of numbers per voice, with nothing to do with them directly.

Upload a voice sample

On the "Speakers" tab you can upload a short recording of one person (say 20–30 seconds of a manager speaking) and attach it to a voice right away. The recording is recognised as a normal job, so it spends balance like any other.

Several businesses in one project

An integrator that serves many businesses through one project names the business of each recording as its tenant: "tenant": "biz-86233275". Voices of two tenants are never compared or numbered together: for voices, a tenant is a project of its own, and each one numbers from #100. Without tenant a recording uses the project's own voices, as before.

A similarity threshold cannot do this. The same kind of office line, headset and codec make staff of different businesses sound alike: in our measurement on real calls, managers of another business came as close as 0.87 to an enrolled manager's voice.

Staff only

When you know which employee's line took the call (the user in your CRM), send it with the recording:

"settings": {
  "speaker_roles": true,
  "tenant": "biz-86233275",
  "known_speakers": [{"external_id": "crm-9612766", "role": "manager"}]
}

external_id is your own id of the line, 1–128 characters of A–Z a–z 0–9 . _ : @ -. A line may be one person, or up to four people who share it on shifts. role is manager (also operator or supervisor), name is optional, and a recording names at most four lines. known_speakers needs tenant: without it the upload is refused with 400 tenant_required, so that a forgotten tenant never compares one business's callers with another's staff.

With known_speakers the recording is compared with the tenant's staff only:

• Only staff are recognised and get a number. A client is never registered, and the voice prints of the recording are dropped before its result is stored.
• A speaker is the line's staff when the closeness to one of the line's voices is at least 0.65 and at least 0.05 above the other speaker of the call.
• identity says who it is: {"status": "matched", "external_id": "crm-9612766", "role": "manager", "voice": 2, "expected": true, "score": 0.91, "threshold": 0.65, "runner_up": 0.39, "voice_id": 100}. voice says which of the line's people it was, 1 to 4; their names we do not know. expected: false means the voice belongs to another line of the same business than the one named. Treat it as a hint to check the CRM, not a verdict: colleagues are told apart much less reliably than staff from clients.
• Other statuses: not_enrolled (the named line has no voice yet), not_matched (nobody of the staff), insufficient_speech.
• Roles are turned on with known_speakers. A two-party call in which exactly one party is recognised takes its roles from the voice: that party gets the line's role and the other one is client. options_report.roles.source says what decided (voice, text or none), and roles.text keeps the text model's own answer beside it, with agrees.

"enroll": "off" recognises the voices already learned and learns nothing new.

How a line's voices are learned

Nobody has to upload samples. Of each call that names a line we take the side the text model calls the manager. A voice heard that way in two of the line's last 20 calls becomes one of its voices; a line keeps up to four, the ones heard most. A client's voice does not come back call after call, so it does not become a voice even when the text model takes the client for the manager. Someone who joins the line is recognised from their third call, and someone who stops taking its calls drops out as the window moves on. Until the line has a voice, roles come from the text, as without voices.

What we keep: for each of the line's last 20 calls, the job number and where that side spoke, and a voice print only once it belongs to one of the line's voices. A side that has not come back yet is printed again from the kept recording when it is needed, so the print of a client the text model picked by mistake is not stored.

If no voice comes back in 20 calls, the line shows inconsistent. In our measurement on real calls a line's staff were recognised in about 95 calls of 100, whether the line was one person or a few people on shifts, on stereo and on mono, and the voice took a customer for staff in fewer than 1 call of 100. On CRM accounts whose people kept changing, recognition falls to about 80 of 100, a customer is taken for staff in 7 to 10 calls of 100, and the voice fixes roles no more often than it breaks them. So send known_speakers only for personal accounts and fixed shifts.

To correct a call where the text model chose the wrong side: POST /v1/tenants/{tenant}/staff/{external_id}/enroll with {"job_id": "…", "speaker_index": 1}. The list of lines with their voices, and deletion: API.

A voice is biometric data

Add voices only with the person's consent. Deleting a voice deletes all its prints: DELETE /v1/speakers/{id}. A voice keeps at most 10 prints, a project at most 500 voices.

Deleting a line, DELETE /v1/tenants/{tenant}/staff/{external_id}, removes all of its voices, their prints and the remembered calls at once; …/voices/{n} removes one of its people. DELETE /v1/tenants/{tenant} removes everything of one business, including the voice prints kept inside its stored results. A deleted recording leaves every line's window. The recordings themselves are kept as for any job, and a result keeps the identity that names the line's id until the result expires after 30 days.