DigiDoc
A telehealth prototype: talk to a 3D AI doctor, a physician signs off
01 The story
A telehealth prototype. A patient fills in an intake form and has a live voice and video conversation with a lip-synced 3D doctor avatar on Gemini Live. It drafts a clinical summary with ICD-10 codes, which a human physician reviews, edits and approves before the patient sees the plan.
DigiDoc explores a two-step model of remote care: an AI handles the first conversation, and a licensed doctor makes the decisions. The patient side covers sign-in, a health profile, an intake form with a chief complaint and preferred language, a live consultation and a dashboard of past visits. Doctors get their own portal, where they pick up waiting cases from a shared pool, review the AI's draft and publish a care plan.
The consultation is real time. The browser streams 16 kHz mic audio and camera frames to Gemini's native-audio Live model and plays its 24 kHz voice back through Web Audio. A 3D doctor built with React Three Fiber speaks with lip sync: wLipSync runs MFCC phoneme detection on the outgoing audio and drives five vowel mouth shapes on the model's morph targets, with blinking on top. Getting the transcript to appear in step with the voice, not ahead of it, took a reveal queue that holds each text chunk until its audio actually plays.
When the call ends, the transcript and a few camera snapshots go to Gemini 3 Flash, which writes a structured clinical summary with an urgency score. It assigns diagnosis codes from a Gemini File Search store built on a merged 2026 ICD-10-CM dataset, so the codes come from reference data rather than the model's memory. The doctor edits the draft in a review screen, and the approved plan can be translated into the patient's language and exported as a PDF. Every model call logs token usage and estimated cost.
Firestore rules separate patients, doctors and offices and let a doctor claim a case from the pool only once, and each access to a record writes an audit-log entry. I worked on it from late December 2025 to early February 2026 and stopped at a working prototype. It isn't deployed.
02 What it does
- 01
Live voice and video consultation with a 3D doctor avatar whose mouth moves with its speech
- 02
Real-time transcript of both sides, kept in sync with the avatar's voice
- 03
Intake form with symptoms, history and a choice of 24 languages, with speech-to-text input
- 04
AI-drafted clinical summary with ICD-10-CM codes and an urgency score, from the conversation and camera snapshots
- 05
Doctor portal: claim cases from a shared pool, review and edit the AI draft, approve the final plan
- 06
Final care plan translated into the patient's language and exportable as a PDF
- 07
Patient dashboard of past consultations, plus account and doctor settings
- 08
Demo mode with seeded patients, doctors and offices for walkthroughs
03 How it's built
A browser-only prototype: React talks directly to Gemini and Firebase with no custom server. The doctor portal opens a case's records once the doctor claims it from the pool.
All 8 connections, as text
- Patient web app → Gemini Live (voice + video): mic audio, camera frames
- Gemini Live (voice + video) → 3D avatar + lip sync: voice, transcript
- Patient web app → Gemini 3 Flash (summary, translation): transcript + snapshots
- Gemini 3 Flash (summary, translation) → File Search store: ICD-10-CM: code lookup
- Gemini 3 Flash (summary, translation) → Firestore (session records, audit log): draft summary
- Doctor portal → Firestore (session records, audit log): claim, review, approve
- Patient web app → Firebase Auth: sign in
- Doctor portal → Firebase Auth: sign in
04 Engineering notes
Gemini Live streaming over WebSockets with AudioWorklet PCM capture (16 kHz in, 24 kHz out) and gapless scheduled playback
Audio-driven lip sync: wLipSync MFCC phoneme analysis mapped to A/I/U/E/O visemes on a glTF avatar's morph targets in React Three Fiber
Transcript and audio alignment with a reveal queue timed to the audio playback clock instead of text arrival
ICD-10-CM codes retrieved from a Gemini File Search store, so they come from the reference data rather than the model's memory
Firestore rules and indexes for role separation and a claim-once case pool, an access audit log, and per-session token and cost tracking
05 By the numbers
- 24
- Intake languages
- 72
- Commits over six weeks
06 Stack
App
- React 19
- TypeScript
- Vite 6
- Tailwind CSS 3
- React Router 7
- React Three Fiber / drei / three.js
- Framer Motion
- wLipSync
- jsPDF + html2canvas
Backend
- Firebase Auth
- Gemini Live (gemini-2.5-flash-native-audio)
- Gemini 3 Flash
- Gemini File Search
Data
- Cloud Firestore (session records, audit logs, offices)
- ICD-10-CM 2026 dataset (JSON)
Infrastructure
- Firebase Hosting (configured)
- Firestore security rules and indexes
Tooling
- tsx seed and test scripts
- Google AI Studio
Lines of code (approx.)
- TypeScript (TSX)7,348
- TypeScript3,582
- CSS549
- JavaScript148
- Firestore rules131
≈ 11,758 lines
Repositories
- DigiDocprivate
Full history on GitHub across main and four side branches (newui, stableish and two merged feature branches). It started from a Google AI Studio template; the final commit message is 'done for now'.