> ## Documentation Index
> Fetch the complete documentation index at: https://docs.inworld.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Lip-sync visemes

> Mouth-shape timing derived from the synthesized audio, for animating avatars in real time.

Lip-sync returns a timeline of **visemes** (mouth shapes) alongside the audio, so you can animate a character's mouth as the speech plays. Visemes are computed from the synthesized audio itself, not from the text, so they:

- add **no latency**: every audio chunk carries the visemes for its own audio, and audio is never held back to compute them;
- work with **every model and language**, and with any input, including SSML, inline tags, and text that word timestamps cannot align;
- are **independent of [timestamps](https://docs.inworld.ai/tts/capabilities/timestamps.md)**: request them on their own, or together with word or character timestamps.

## Request visemes

Add `lipsyncConfig` to the request. Its presence turns lip-sync on; an empty object is enough:

```json
{
  "text": "Hello world, this will be saved.",
  "voiceId": "Ashley",
  "modelId": "inworld-tts-2",
  "lipsyncConfig": {}
}
```

| Field | Description |
| :---- | :---- |
| `lipsyncConfig.visemeSet` | The viseme vocabulary. Optional; the only supported value is `OPENXR`, which is also the default. Other values are rejected with `INVALID_ARGUMENT`. |

Omitting `lipsyncConfig`, or sending `"lipsyncConfig": null`, leaves lip-sync off, and responses carry no `lipsyncAlignment`.

`lipsyncConfig` is accepted by every synthesis method: [Synthesize speech](https://docs.inworld.ai/tts/synthesize-speech.md), [Synthesize speech (streaming)](https://docs.inworld.ai/tts/synthesize-speech-streaming.md), [Synthesize speech (WebSocket)](https://docs.inworld.ai/tts/synthesize-speech-websocket.md), [Synthesize speech (async)](https://docs.inworld.ai/tts/synthesize-speech-async.md) and [Synthesize speech (batch)](https://docs.inworld.ai/tts/synthesize-speech-batch.md).

## Response

Visemes arrive in `timestampInfo.lipsyncAlignment` as three arrays of equal length:

```json
{
  "audioContent": "UklGRi...",
  "timestampInfo": {
    "lipsyncAlignment": {
      "visemes": ["sil", "kk", "E", "nn", "oh", "ou", "RR", "nn", "DD", "sil"],
      "visemeStartTimeSeconds": [0, 0.24, 0.31, 0.39, 0.47, 0.59, 0.69, 0.72, 0.95, 1.09],
      "visemeEndTimeSeconds": [0.24, 0.31, 0.39, 0.47, 0.59, 0.69, 0.72, 0.95, 1.09, 1.29]
    }
  }
}
```

| Field | Description |
| :---- | :---- |
| `visemes` | Viseme names from the [viseme set](#viseme-set). |
| `visemeStartTimeSeconds` | Start of each viseme span, in seconds. |
| `visemeEndTimeSeconds` | End of each viseme span, in seconds. |

The spans tile the audio with no gaps: each span starts where the previous one ended, and together they cover the whole audio, including silence. Adjacent identical visemes are merged into one span. To animate, show `visemes[i]` while the playback position is between `visemeStartTimeSeconds[i]` and `visemeEndTimeSeconds[i]`.

When you also request `timestampType`, `lipsyncAlignment` sits next to `wordAlignment` or `characterAlignment` in the same `timestampInfo`. The two are independent: word timestamps keep their own `phoneticDetails[].phones[].visemeSymbol`, which uses a different, text-derived viseme vocabulary.

## Viseme set

Visemes use a standard set of 15 mouth shapes. Avatars rigged with these blend shapes can map them directly.

<Note>
**Compatibility:** the viseme names and their order follow the [OpenXR specification](https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#XR_FACE_TRACKING_VISEME_COUNT_META), and the shapes are the 15 visemes of the MPEG-4 facial animation standard, so you can use them in existing projects built on either.
</Note>

| Viseme | Sounds | Example |
| :---- | :---- | :---- |
| `sil` | Silence and pauses | |
| `PP` | p, b, m | **p**ut, **b**at, **m**at |
| `FF` | f, v | **f**at, **v**at |
| `TH` | θ, ð | **th**ink, **th**at |
| `DD` | t, d | **t**ip, **d**oll |
| `kk` | k, g | **c**all, **g**as |
| `CH` | tʃ, dʒ, ʃ | **ch**air, **j**oin, **sh**e |
| `SS` | s, z | **s**ir, **z**eal |
| `nn` | n, l | **n**ot, **l**ot |
| `RR` | r | **r**ed |
| `aa` | ɑ | c**a**r |
| `E` | ɛ | b**e**d |
| `ih` | ɪ | t**i**p |
| `oh` | o | t**oe** |
| `ou` | u | b**oo**t |

Pauses you insert with SSML `<break>` tags are always `sil` for their exact duration.

## Streaming

In [Synthesize speech (streaming)](https://docs.inworld.ai/tts/synthesize-speech-streaming.md) and [Synthesize speech (WebSocket)](https://docs.inworld.ai/tts/synthesize-speech-websocket.md) responses, every message that carries audio also carries the `lipsyncAlignment` spans for that audio. A span that continues across two messages is split at the boundary, so concatenating the arrays from every message in order gives the full timeline. Times are measured from the start of the stream, on the same timeline as word timestamps.

### Timestamp transport

Lip-sync changes the default [`timestampTransportStrategy`](https://docs.inworld.ai/tts/capabilities/timestamps.md#streaming-behavior). When the request sets `lipsyncConfig` and leaves the transport unspecified, word and character timestamps use `ASYNC`: they follow in trailing messages, so aligning the text never delays the audio or its visemes. Visemes always travel with their audio, whichever transport is used. Set `timestampTransportStrategy: SYNC` explicitly if you need word timestamps in the same messages as their audio, at the cost of the extra latency described on the [Timestamps](https://docs.inworld.ai/tts/capabilities/timestamps.md#sync-default) page.

### WebSocket contexts

Set `lipsyncConfig` when you create a context; it applies to the whole context. An `updateConfig` that omits it keeps the context's setting, and one that sends a different value is rejected, so lip-sync cannot be turned on or off mid-context. As with word timestamps, viseme times restart at zero after each `flushCompleted`.

```javascript
ws.send(JSON.stringify({
  contextId: "avatar-1",
  create: {
    voiceId: "Ashley",
    modelId: "inworld-tts-2",
    lipsyncConfig: {}
  }
}));
```

## Animate an avatar

### Recommended setup

For a character that speaks an LLM's replies:

- Use [Synthesize speech (WebSocket)](https://docs.inworld.ai/tts/synthesize-speech-websocket.md) with one context per reply. Create the context with `lipsyncConfig` when the user's turn ends, in parallel with the LLM request.
- Drive the **mouth from the visemes**. They arrive with their audio, so the mouth is never late.
- If you also want captions or cues on words, add `timestampType: WORD`. With lip-sync on, the words follow in trailing messages (ASYNC), so first audio isn't held back for them. Don't drive the mouth from the words' `phoneticDetails`: under ASYNC they arrive after their audio has started playing.

### Play the visemes on your audio clock

Viseme times are relative to the start of each synthesis: the stream, or each flush on a WebSocket context. To show the right shape:

1. Concatenate the spans of every message of a synthesis, re-joining a span that was split across two messages.
2. Note the time on your audio clock at which that synthesis's first audio chunk was scheduled to play.
3. On every frame, take the position being heard and look up the span that contains it. The position is the clock time minus the synthesis's start time, minus the output latency (`AudioContext.outputLatency` in a browser).

Spans can be as short as 10 ms, so switching shapes at span edges looks jittery. Instead, blend into each shape over a few tens of milliseconds, slightly ahead of its span. Scaling how far the mouth opens by the loudness of the audio being played lets loud syllables open wider than quiet ones.

### Avatars with five vowel shapes

Some avatar formats have few expressions. VRM, for example, has only the vowel expressions A, I, U, E and O. Mix these to form each viseme, adding a little A to every vowel to lower the jaw. These weights are a starting point, tuned on VRoid models:

| Viseme | Mix |
| :---- | :---- |
| `sil`, `PP` | closed (no vowel) |
| `FF` | I 0.25, E 0.1 |
| `TH` | A 0.3, E 0.15 |
| `DD` | A 0.3, I 0.25 |
| `kk` | A 0.4, I 0.15 |
| `CH` | U 0.35, I 0.2, A 0.15 |
| `SS` | I 0.4, E 0.1, A 0.1 |
| `nn` | A 0.25, I 0.3 |
| `RR` | U 0.35, O 0.2, A 0.1 |
| `aa` | A 1 |
| `E` | E 0.8, A 0.35 |
| `ih` | I 0.8, A 0.25 |
| `oh` | O 0.9, A 0.35 |
| `ou` | U 1, A 0.15 |

Keep each expression at most 1. Beyond that the shape extrapolates: seen from the side, the jaw pushes out past the chin.

### When `lipsyncAlignment` is missing

Check each response for the field (see the note below). To keep the mouth moving when it is absent, also request `timestampType: WORD` and fall back to the phones in `wordAlignment.phoneticDetails`, mapped to the same mouth shapes. Under ASYNC they arrive after their audio, so that fallback lags slightly.

## Non-streaming methods

- **[Synthesize speech](https://docs.inworld.ai/tts/synthesize-speech.md)** returns one `lipsyncAlignment` covering the whole response.
- **[Synthesize speech (async)](https://docs.inworld.ai/tts/synthesize-speech-async.md)** and **[Synthesize speech (batch)](https://docs.inworld.ai/tts/synthesize-speech-batch.md)** jobs write visemes into the timestamps document (`timestampsUri`), which is produced whenever the job returns visemes or timestamps. Read the `lipsyncAlignment` entries of that array in order.

- **Scripts** return one timeline for combined output. With `outputMode: PER_UTTERANCE`, each utterance has its own timeline starting at zero.

<Note>
If lip-sync is temporarily unavailable, synthesis still succeeds: the audio is returned without `lipsyncAlignment`. Check for the field rather than assuming it is present. On a WebSocket, `contextCreated` echoes the `lipsyncConfig` you sent, so it doesn't tell you whether visemes will follow; the responses do.
</Note>