Use cases
- Voice agents and NPCs — Adapt responses based on the speaker’s detected emotion or vocal style (e.g., respond empathetically to a sad tone).
- Accessibility — Detect age category or vocal style to adjust UI, pacing, or interaction complexity.
- Content moderation — Flag unusual vocal patterns (shouting, crying) for escalation or review.
- Analytics and insights — Aggregate emotion and vocal style data across sessions for user experience analysis.
- Localization — Use accent detection to dynamically select language models or localized content.
How it works
Voice Profile analysis runs when enabled viavoiceProfileConfig in your transcribeConfig request (HTTP or WebSocket). Set enableVoiceProfile to true to activate the feature. Optionally use topN to control how many top labels per category are returned.
Classification categories
Age
Estimates the speaker’s age category. Returns an array of labels sorted by descending confidence.Emotion
Detects emotional tone in the speaker’s voice. Returns an array of labels sorted by descending confidence.Pitch
Classifies the speaker’s vocal pitch. Pitch shifts during a conversation can serve as a real-time emotional signal — a voice moving from lower to higher pitch can correlate with rising stress, excitement, or urgency, while a dropping pitch may indicate the speaker is becoming more withdrawn, tired, or deflated. Returns an array of labels sorted by descending confidence.Vocal Style
Identifies the speaker’s manner of delivery. Returns an array of labels sorted by descending confidence.Accent
Detects the speaker’s accent or regional dialect using BCP-47 locale codes. Returns an array of labels sorted by descending confidence.Additional accent locales may be returned beyond those listed above. The model supports a broad range of BCP-47 codes.
Configuration
The STT API accepts both camelCase and snake_case field names (e.g.,transcribeConfig / transcribe_config, voiceProfileConfig / voice_profile_config). The examples below use camelCase.
Set voiceProfileConfig inside transcribeConfig:
groq/whisper-large-v3 for synchronous HTTP, or the assemblyai/... streaming models listed in the STT overview).
WebSocket (Streaming)
IncludevoiceProfileConfig in the first WebSocket message:
Configuration parameters
Response structure
ThevoiceProfile object is returned alongside transcription and usage in both sync and streaming responses. Each category is an array of { label, confidence } objects, sorted by descending confidence.
The JSON below shows the normalized response shape (camelCase throughout). Raw API payloads may use snake_case for the same fields (for example vocal_style, transcribed_audio_ms, model_id). Prefer representing one layer per example in your own docs and client code — either the raw API shape or the normalized shape — not a mix of both.
Example response (sync, normalized shape)
Response fields
Each
ClassLabel object contains:
- label (string) — The predicted class name
- confidence (float) — Score from 0.0 to 1.0
Best practices
- Start with the default
topN(10) — This returns up to 10 labels per category, sorted by descending confidence. LowertopNif you only need the most confident predictions; raise it if you need broader signal. - Use emotion and vocal style together — Combining both categories gives a richer picture. A “tender” emotion with “whispering” vocal style tells a different story than “tender” with “normal” style.
- Handle missing fields gracefully — Voice Profile fields may be absent if the model cannot make a confident classification or if the audio quality is insufficient. Always check for presence before accessing.
- Accent is probabilistic — Accent detection returns the most likely locale, not a definitive answer. Use it as a signal rather than a hard routing decision.
- Test with representative audio — Classification accuracy depends on audio quality, background noise, and speech duration. Test with samples that reflect your production environment.