PROJECT SCOPE
How we built this.
A signing avatar, a conversation interface, and a few focused packages that connect the two.
THE BUILD
From input to conversation.
- 01
One place for a conversation
We built the interface in React and TypeScript with Vite. Text, voice, and camera input share a conversation history. The avatar stays alongside it.
- 02
Words become a playback plan
The API turns a message into an ordered list of signs and captions. The avatar loads the available motion clips and plays them in order. Words without a matching sign stay visible as text.
- 03
A camera suggests; you confirm
MediaPipe tracks hand landmarks. A geometric classifier suggests a demonstration sign after a stable hold. You review it before adding it to the conversation or speaking it aloud.
- 04
Separate parts, shared contracts
Schemas define the data passed between planning, playback, device controls, and the interface. This keeps camera access and rendering out of the core conversation logic.
WHAT WE USED
The tools behind it.
- Page scrolling
- Lenis
- Interface
- React · TypeScript · Vite
- Design
- Geist Sans · Geist Mono · shared CSS tokens · original SVG icons
- Avatar
- Three.js · React Three Fiber · VRM
- Camera & voice
- MediaPipe · browser speech recognition and synthesis
- API & contracts
- Node.js · Fastify · Zod
- Checks
- Vitest · Testing Library · TypeScript · ESLint · asset checksums
CURRENT BUILD
A working prototype, with a limited vocabulary.
ISL playback uses a small demonstration catalog. Camera suggestions are geometric demonstrations, not validated translation. ASL has its own language path and stays text-only until ASL assets are available.
The default planner runs locally. A server-side Meta planning adapter is available, with a local fallback. Automated checks cover planning, playback, resource cleanup, and preserved assets; they do not establish real-world signing accuracy.