Lip-sync visemes

Lip-sync returns a timeline of visemes (mouth shapes) alongside the audio, so you can animate a character's mouth as the speech plays. Visemes are computed from the synthesized audio itself, not from the text, so they:

• add no latency: every audio chunk carries the visemes for its own audio, and audio is never held back to compute them;
• work with every model and language, and with any input, including SSML, inline tags, and text that word timestamps cannot align;
• are independent of timestamps: request them on their own, or together with word or character timestamps.

Request visemes

Add lipsyncConfig to the request. Its presence turns lip-sync on; an empty object is enough:

{
  "text": "Hello world, this will be saved.",
  "voiceId": "Ashley",
  "modelId": "inworld-tts-2",
  "lipsyncConfig": {}
}

Omitting lipsyncConfig, or sending "lipsyncConfig": null, leaves lip-sync off, and responses carry no lipsyncAlignment.

lipsyncConfig is accepted by every synthesis method: Synthesize speech, Synthesize speech (streaming), Synthesize speech (WebSocket), Synthesize speech (async) and Synthesize speech (batch).

Response

Visemes arrive in timestampInfo.lipsyncAlignment as three arrays of equal length:

{
  "audioContent": "UklGRi...",
  "timestampInfo": {
    "lipsyncAlignment": {
      "visemes": ["sil", "kk", "E", "nn", "oh", "ou", "RR", "nn", "DD", "sil"],
      "visemeStartTimeSeconds": [0, 0.24, 0.31, 0.39, 0.47, 0.59, 0.69, 0.72, 0.95, 1.09],
      "visemeEndTimeSeconds": [0.24, 0.31, 0.39, 0.47, 0.59, 0.69, 0.72, 0.95, 1.09, 1.29]
    }
  }
}

The spans tile the audio with no gaps: each span starts where the previous one ended, and together they cover the whole audio, including silence. Adjacent identical visemes are merged into one span. To animate, show visemes[i] while the playback position is between visemeStartTimeSeconds[i] and visemeEndTimeSeconds[i].

When you also request timestampType, lipsyncAlignment sits next to wordAlignment or characterAlignment in the same timestampInfo. The two are independent: word timestamps keep their own phoneticDetails[].phones[].visemeSymbol, which uses a different, text-derived viseme vocabulary.

Viseme set

Visemes use a standard set of 15 mouth shapes. Avatars rigged with these blend shapes can map them directly.

Pauses you insert with SSML <break> tags are always sil for their exact duration.

Streaming

In Synthesize speech (streaming) and Synthesize speech (WebSocket) responses, every message that carries audio also carries the lipsyncAlignment spans for that audio. A span that continues across two messages is split at the boundary, so concatenating the arrays from every message in order gives the full timeline. Times are measured from the start of the stream, on the same timeline as word timestamps.

Timestamp transport

Lip-sync changes the default timestampTransportStrategy. When the request sets lipsyncConfig and leaves the transport unspecified, word and character timestamps use ASYNC: they follow in trailing messages, so aligning the text never delays the audio or its visemes. Visemes always travel with their audio, whichever transport is used. Set timestampTransportStrategy: SYNC explicitly if you need word timestamps in the same messages as their audio, at the cost of the extra latency described on the Timestamps page.

WebSocket contexts

Set lipsyncConfig when you create a context; it applies to the whole context. An updateConfig that omits it keeps the context's setting, and one that sends a different value is rejected, so lip-sync cannot be turned on or off mid-context. As with word timestamps, viseme times restart at zero after each flushCompleted.

ws.send(JSON.stringify({
  contextId: "avatar-1",
  create: {
    voiceId: "Ashley",
    modelId: "inworld-tts-2",
    lipsyncConfig: {}
  }
}));

Animate an avatar

Recommended setup

For a character that speaks an LLM's replies:

• Use Synthesize speech (WebSocket) with one context per reply. Create the context with lipsyncConfig when the user's turn ends, in parallel with the LLM request.
• Drive the mouth from the visemes. They arrive with their audio, so the mouth is never late.
• If you also want captions or cues on words, add timestampType: WORD. With lip-sync on, the words follow in trailing messages (ASYNC), so first audio isn't held back for them. Don't drive the mouth from the words' phoneticDetails: under ASYNC they arrive after their audio has started playing.

Play the visemes on your audio clock

Viseme times are relative to the start of each synthesis: the stream, or each flush on a WebSocket context. To show the right shape:

1. Concatenate the spans of every message of a synthesis, re-joining a span that was split across two messages.
2. Note the time on your audio clock at which that synthesis's first audio chunk was scheduled to play.
3. On every frame, take the position being heard and look up the span that contains it. The position is the clock time minus the synthesis's start time, minus the output latency (AudioContext.outputLatency in a browser).

Spans can be as short as 10 ms, so switching shapes at span edges looks jittery. Instead, blend into each shape over a few tens of milliseconds, slightly ahead of its span. Scaling how far the mouth opens by the loudness of the audio being played lets loud syllables open wider than quiet ones.

Avatars with five vowel shapes

Some avatar formats have few expressions. VRM, for example, has only the vowel expressions A, I, U, E and O. Mix these to form each viseme, adding a little A to every vowel to lower the jaw. These weights are a starting point, tuned on VRoid models:

Keep each expression at most 1. Beyond that the shape extrapolates: seen from the side, the jaw pushes out past the chin.

When lipsyncAlignment is missing

Check each response for the field (see the note below). To keep the mouth moving when it is absent, also request timestampType: WORD and fall back to the phones in wordAlignment.phoneticDetails, mapped to the same mouth shapes. Under ASYNC they arrive after their audio, so that fallback lags slightly.

Non-streaming methods

• Synthesize speech returns one lipsyncAlignment covering the whole response.

• Synthesize speech (async) and Synthesize speech (batch) jobs write visemes into the timestamps document (timestampsUri), which is produced whenever the job returns visemes or timestamps. Read the lipsyncAlignment entries of that array in order.

• Scripts return one timeline for combined output. With outputMode: PER_UTTERANCE, each utterance has its own timeline starting at zero.