What problem does J.A.R.V.I.S — NIX solve?

Commercial voice assistants are subscription-locked, cannot act on your operating system, and forget everything between sessions. NIX is a local-first assistant that can control the machine and remembers context across sessions.

Which model powers the voice interaction?

The Gemini Live API handles streaming audio in both directions, which allows the assistant to be interrupted mid-response instead of waiting for a complete turn.

How does the assistant control the computer?

A typed tool registry wraps desktop control, file handling, code execution and Playwright browser automation. The model selects tools through structured calls rather than free-form shell strings.

How does it see the screen?

Screen captures and webcam frames are attached to a turn on demand, so questions about what is currently displayed are answered with visual context.

How is memory implemented?

A persistent local store keeps facts, preferences and session logs. Relevant entries are replayed into context at session start so the assistant continues where it left off.

Is the source code available?

Yes, the project is open source at github.com/Nithin69-k/JARVIS.

Solo builder · AI Engineer · 2026

J.A.R.V.I.S — NIX

A local-first real-time voice AI that can hear, see and control your machine.

PythonGemini Live APIPlaywrightDesktop AutomationVector Memory

View code

J.A.R.V.I.S — NIX — A local-first real-time voice AI that can hear, see and control your machine.

Goals

Remove the subscription and cloud lock-in of commercial voice assistants.
Give the assistant real capability on the machine: files, apps, browser and code execution.
Keep context between sessions instead of restarting from zero every conversation.
Hold conversational latency low enough to feel like speech, not request/response.

Approach & architecture

1
Realtime speech layer
Gemini Live Audio streams both directions, so the assistant can be interrupted mid-sentence instead of waiting for a full turn.
2
Tool layer
A registry of typed tools wraps OS control, file handling, code execution and Playwright-driven browser automation; the model picks tools by structured call.
3
Vision
Screen capture and webcam frames are attached on demand, so the assistant can answer questions about what is currently on screen.
4
Memory
A persistent store keeps facts, preferences and session logs, replayed as grounded context at the start of each session.
5
Wake control
Clap-to-wake and hotword handling keep the flow hands-free during long working sessions.