Sep 16, 2026 Interviews Self-host orchestration ElevenLabs TTS

Roll-your-own interview agent with a live talking avatar

Open-source framework you actually deploy. Pair it with ElevenLabs for voice. Put a lip-synced face on the stream. Keep turn latency in the “conversation” band, not the “wait for a clip” band.

The short version

There is no single Apache-licensed box that is “ElevenLabs Agents + a photoreal talking video, fully free, fully local, sub-second.” The thing that is free and self-deployed is the orchestration + WebRTC room. Voice and face are plugins you swap.

Recommended for interviews: LiveKit Agents (or Pipecat if you already like Daily) as the agent runtime → ElevenLabs streaming TTS for the voice you already pay for → a speech-to-video avatar plugin (Simli / Beyond Presence / HeyGen LiveAvatar LITE / Anam) that joins the same room as a second participant. That is the pattern LiveKit and ElevenLabs both document.[10][8][16]

If you need the face to be open-source too: LiveTalking or OpenAvatarChat on a GPU, driven by the same TTS audio. Quality and ops jump; latency is still real-time if the GPU holds ≥25 fps.[5][3][6]

Honest cost: the frameworks are free. Minutes are not. ElevenLabs Conversational AI is ~$0.08–$0.10/min plus LLM.[9] A cascading LiveKit stack is commonly quoted ~$0.06–$0.15/min all-in without a photoreal face.[13] Adding a managed avatar is another ~$0.10–$0.20/min on HeyGen LiveAvatar Lite/Full math.[15] Self-hosting the face trades vendor minutes for a GPU that must stay hot.

1. Piece it together — recommended architecture

Think in five stages, not one product. An interactive avatar is a conversation loop, not an MP4.[12]

TransportWebRTC room. Candidate browser ↔ SFU. Self-host livekit-server or use LiveKit Cloud / Daily. Pipecat talks to both.[17]

VAD / turnsSilero VAD + a turn detector. Interruptions are mandatory for interviews — the candidate will talk over the avatar.

STTDeepgram Nova (hosted) or faster-whisper / SenseVoice (local). Streaming partials, not batch files.

BrainAny OpenAI-compatible LLM. Keep replies short. Tool calls = score rubric, next question, clock.

TTSElevenLabs streaming (Flash / low-latency tier). LiveKit has a first-party ElevenLabs plugin.[16] PCM 24 kHz if you later attach HeyGen LiveAvatar.[8]

FaceAvatar worker consumes TTS audio, publishes a video track into the same room. Do not mux through your bot process — LiveKit’s advice is: avatar joins as its own participant to avoid extra hops.[11]

Concrete default (hybrid — closest to “ElevenLabs Agents with a face”)

Browser
  └─ LiveKit room (self-host or Cloud)
       ├─ candidate mic/cam
       ├─ agent worker  (LiveKit Agents or Pipecat)
       │     VAD → Deepgram STT → LLM → ElevenLabs TTS
       └─ avatar participant
             Simli / Beyond Presence / HeyGen LiveAvatar LITE
             (audio in → lip-synced video out)

ElevenLabs’ own live-avatar path is HeyGen LiveAvatar in LITE mode: ElevenLabs owns audio, HeyGen owns the face. Separate bills.[8]

Concrete default (max open-source face)

Browser / WHIP-WHEP
  └─ LiveTalking  (Apache-2.0)  or  OpenAvatarChat (Apache-2.0)
       ASR: SenseVoice / faster-whisper
       LLM: vLLM / Ollama OpenAI-compatible
       TTS: ElevenLabs stream  OR  local CosyVoice / Piper
       Face: MuseTalk or wav2lip on NVIDIA GPU
             LiveTalking: wav2lip256 ≈ 60 fps on RTX 3060;
             MuseTalk ≈ 42 fps on 3080 Ti, 72 fps on 4090.[5]

This is the “we own the pixels” path. It is not a weekend toy if you need concurrent interview rooms — each speaking session eats GPU; idle sessions eat CPU for encode.[5]

2. The free frameworks (orchestration)

Framework	What it is	Stars / license	Use it when
Pipecat	Python frame pipeline: transport → STT → LLM → TTS → out. Daily-maintained. Tavus video service built in.[1][7]	~15.6k · BSD-2-Clause	You want the most swapped-in voice/video connectors and Daily rooms.
LiveKit Agents	Agent joins a WebRTC room. Plugins for ElevenLabs, Simli, many avatars. Same code on Cloud or self-host.[10][13]	Apache-2.0 server + agents	Interviews — already a video call. Recording, multi-party, SIP later.
TEN Framework	Agora-backed multimodal real-time agents. Graph of extensions, not a linear pipe.[2][18]	~11.1k · Apache-2.0 + extra restrictions	You want Agora RTC + a heavier extension runtime.
OpenAvatarChat	Full talking-head demo: ASR+LLM+TTS+avatar, duplex interrupt, MuseTalk / FlashHead / lite-avatar.[3]	~3.8k · Apache-2.0	You want one repo that already composites a face, and you’ll replace handlers.
LiveTalking	Streaming digital human: wav2lip / MuseTalk / ultralight, WebRTC/WHEP/RTMP.[5]	~9.5k · Apache-2.0	Self-host the face renderer; plug your own LLM/TTS.
Open-LLM-VTuber	Local voice + Live2D, interruptions, MCP.[4]	~13.8k	Cartoon / companion UX — not a photoreal interviewer.

If you only pick one OSS runtime for this use case: LiveKit Agents. Interviews already look like a video call; the avatar is just another publisher in the room.[10][13]

3. The face layer — pair with ElevenLabs

ElevenLabs is excellent at voice. Their “avatar” product in Creative is talking-video generation; their live conversational face is an integration (HeyGen LiveAvatar), not a self-hosted renderer.[8] You keep ElevenLabs for TTS (and optionally their agent audio), and attach a face that eats that audio.

Face	Self-host?	Latency claims	Notes
Simli	No (API)	Marketed real-time; LiveKit plugin is first-class.[10]	Smallest glue if you already have a LiveKit agent.
Beyond Presence	No (EU-hosted)	Site: global avatar ≤250 ms; Genesis streaming <100 ms.[14]	LiveKit + Pipecat examples. Speech-to-video API takes your LiveKit token.
HeyGen LiveAvatar	No	“One of the fastest”; 1080p costs latency.[15]	Official ElevenLabs LITE path. ~$0.10/min Lite, ~$0.20/min Full on published credit math.[8][15]
Anam	No	180 ms average claimed.[15]	Often picked for coaching/interview-practice feel.
Tavus	No	<500 ms e2e advertised.[15] Pipecat service.[7]	Replica cloning. Perception model. Dual-room with Daily in Pipecat.[7]
D-ID	No	Own bench: conversational <500 ms, some figures <120 ms.[20]	Vendor-run comparison vs HeyGen/Tavus/Anam — treat as marketing, not gospel.
MuseTalk + LiveTalking / OpenAvatarChat	Yes	Must hold ≥25 fps; LiveTalking publishes GPU fps tables.[5][6]	MIT code / Apache stack. Photoreal-ish 2D neural. GPU required.
Live2D (Open-LLM-VTuber)	Yes	Cheap on CPU	Wrong aesthetic for most corporate interviews.

4. Latency — what “short and sweet” actually means

Humans gap ~0–200 ms between turns. Avatars cannot hit that end-to-end; the practical line is first audio/video under ~1 s. Two seconds feels broken.[12]

A well-built streaming cascade (not wait-for-full-transcript):

ASR endpoint+partials     ~150 ms
LLM time-to-first-token   ~300 ms
TTS first audio chunk     ~150 ms
Avatar first video frame  ~200 ms
WebRTC / jitter           ~100 ms
--------------------------------
≈ 900 ms to first frame[12]

LiveKit’s own cascading budget is similar: VAD 50 + STT 150 + LLM TTFT 400 + TTS 150 + net 50 ≈ under 800 ms to first audio if you don’t blow a stage. Speech-to-speech models can go <300 ms if you accept one vendor.[13]

Co-locate agent and models. LiveKit’s India write-up: same stack, wrong region vs Mumbai co-located Deepgram, ~1 s extra (example ~1.67 s e2e with GPT-4o + Cartesia + Deepgram).[19] Avatar extra hop is a common self-inflicted wound — publish the avatar as its own participant.[11]

Interview-specific:

Stream every stage. Never wait for the full LLM paragraph before TTS.
Keep system prompts and retrieved rubric small — context length is latency.
Cap tool steps. A “thinking” filler is better than silence.
Don’t run 1080p avatar in Full mode unless the wow is the product; Lite / 720p wins interviews.
Warm the worker. Cold start is not conversation latency, but candidates feel it.

5. Alternatives you can actually deploy

A. Hybrid (recommended). Self-host LiveKit + your LLM/STT. Pay ElevenLabs TTS. Pay Simli or Beyond Presence for the face. Fastest path to “dynamic conversation with a talking video.”
B. Stay on ElevenLabs Agents, add HeyGen LITE. Least engineering. Still two vendors, still per-minute. This is what you already have, with a face bolted on.[8]
C. Pipecat + TavusVideoService. If Daily is your room. Dual-room architecture; interruptions handled in the service.[7]
D. Full OSS talking head. OpenAvatarChat or LiveTalking + MuseTalk on a rented GPU. ElevenLabs still optional for voice quality. Highest ops.
E. Voice-only first. LiveKit/Pipecat + ElevenLabs, no face. If the interview outcome doesn’t change when a face appears, skip the GPU/vendor. Fora Soft’s rule: if you can’t name the moment the face changes the outcome, don’t pay for it yet.[12]
F. TEN + Agora. If you already sit on Agora RTC. Heavier runtime, extra license caveats on the core.[2]
G. Realtime speech-to-speech (OpenAI Realtime / Gemini Live) + avatar plugin. Lowest latency, least provider freedom.[13]

6. Complexity — what you are signing up for

Layer	Easy	The actual pain
Hello-world agent	A dozen lines of `AgentSession`[13]	Turn-taking, barge-in, endpointing short answers (“yes”), accented STT.
WebRTC	LiveKit Cloud hides ICE	Self-host: TURN, UDP, corporate candidate networks, Safari.
TTS sync	Streaming ElevenLabs	Clock drift vs video; viseme vs waveform; interrupt = drop in-flight audio and freeze last video frame.
Avatar GPU	One 4090 demo	Concurrent rooms. LiveTalking: speaking concurrency = GPU; idle = CPU encode. Need ≥25 inferfps and finalfps.[5]
Interviews product	Prompt + rubric	Recording consent, PII, score calibration, no-face fallback, time-boxed sections, human takeover.
Cost control	One vendor invoice	Four meters: STT, LLM, TTS, avatar, plus egress. Silence still bills some vendors.

Blast radius: start with voice-only LiveKit + ElevenLabs in one region, measure e2e, then add Simli/Beyond Presence as a second participant. Do not start by training a MuseTalk identity.

7. What others are using

LiveKit plugin catalog is the industry switchboard: Anam, Avatario, Beyond Presence, bitHuman, D-ID, LiveAvatar, Simli, Tavus, TruGen, … (Hedra deprecated).[15]
Interview / coaching products (Tough Tongue and similar) treat the avatar as a commodity layer and own the rubric, scoring, and scenario tools on top.[15]
Pipecat is the Daily-world default for composable voice+video agents; Tavus is a documented video service in-tree.[1][7]
China OSS cluster: LiveTalking, OpenAvatarChat, MuseTalk — digital-human streaming on owned NVIDIA boxes, often SenseVoice + CosyVoice, not ElevenLabs.[3][5][6]
Live2D / VTuber stack (Open-LLM-VTuber) is huge on GitHub and the wrong look for most hiring loops.[4]
Buy vs build: managed avatar APIs commonly cited ~$0.10–$0.37/min; a custom render loop ~$0.06–$0.12 in raw compute — the gap is convenience and concurrency, not magic.[12]

8. Cost sketch (order of magnitude)

Piece	Ballpark
ElevenLabs Conversational AI minutes	~$0.10/min Creator/Pro; ~$0.08/min annual Business; LLM extra.[9]
Cascading LiveKit voice (STT+LLM+TTS+transport)	~$0.06–$0.15/min typical; TTS+LLM dominate; LiveKit Cloud often 10–20%.[13]
HeyGen LiveAvatar	~$0.10/min Lite, ~$0.20/min Full on published credits.[15] Plus ElevenLabs audio if LITE.[8]
Self-host face	GPU rent (e.g. 4090/H100 class) + engineer time. Pays off only at high concurrent minutes or data-residency.
Self-host LiveKit SFU	Worth considering past ~500k agent-minutes/month or for residency.[13]

If interviews are getting expensive on ElevenLabs Agents, the usual win is not killing ElevenLabs voice — it’s (1) shorter turns, (2) Flash/low-latency TTS instead of the agent bundle, (3) a cheaper LLM, (4) not paying a second photoreal vendor until the face is proven to change completion rates.

9. If you build this next

Stand up LiveKit (Cloud is fine for a pilot) + Agents worker in the same region as Deepgram/ElevenLabs.
Voice-only interview loop. Record. Measure time-to-first-audio.
Keep ElevenLabs streaming TTS. Confirm PCM 24 kHz if HeyGen is on the roadmap.[8]
Add Simli or Beyond Presence as a room participant. A/B face vs voice-only on the same rubric.
Only then consider MuseTalk/LiveTalking if volume or privacy says you must own the renderer.

This is a map, not a production SLA. Vendor latency numbers are their claims unless noted as a third-party bench. D-ID’s lip-sync table is their own test.[20] Anam’s 180 ms is Anam’s marketing.[15]

Sources

Sources