Skip to content
Keyush Patel

Voice agent for macOS

Siri Mac Agent

Watch, speak to, type to, and control a MacBook from an iPhone, from anywhere, including writing and running software by voice.

Role
Sole builder. Architecture, routing, transcription, safety model, and the macOS tool layer.
Status
In development
Year
2026
Start a project
How it works, end to end. The screens inside are real captures; every command and stage is the project's own.

The problem

Siri cannot be made smarter. Apple only lets a third party do three things: launch a Shortcut by name, hand over dictated text, and speak a reply. That is the entire surface area, so every attempt to make Siri genuinely useful runs out of room immediately.

What I did

Treated Siri as a voice pipe rather than a brain, and put all of the intelligence on the Mac. Built the daemon, the router that decides where each request goes, the tiered macOS tool layer, on-device transcription, the phone app, and the safety model that makes remote control of a whole machine defensible.

How it works

Two ways in, one brain. An iPhone Shortcut sends a recording over Tailscale to a daemon on the Mac, and an installed web app carries live video, voice, text, keyboard, and trackpad to the same place. A router sends each request one of four ways: a fast path that answers questions like battery level from a single command with no model at all, an Operator agent that drives macOS through tools, a live Claude Code session inside tmux, or the Builder, which writes and runs real software in a project directory.

siri-mac-agentreplay

Say

Mac

matchbattery|charge|power
runpmset -g batt
Scripted replay. The routes, tiers and commands are the ones the agent really runs; this panel is not connected to a Mac.

What it answers

Real output from the running daemon
Terminal output: three voice requests answered by the daemon, showing which destination each was routed to.
Three requests against the running daemon. The first two matched a pattern and answered from a single CLI call, so no model was involved at all; the third needed the Operator, which read the real window list back.
Terminal output comparing the raw pmset battery line with the sentence the agent speaks instead.
The fast path is fast because nothing rewrites the output, which is a problem when the output is pmset. Read aloud, the raw line is useless, so each fast answer has a formatter that degrades back to the raw text if parsing fails.
Terminal output: a delete request declined, above the safe, confirm, and blocked command tiers.
Deletions are checked before they run. The blocked set has no override path at all, precisely because any override would itself be reachable by voice, the one thing an attacker with a speaker would go looking for.
The Siri Mac Agent phone app showing the Mac's screen with voice and keyboard controls.
The phone app, live.
The phone app in landscape, showing the full Mac screen.
Landscape, so a 16:10 screen uses the glass.
The phone app with its keyboard and modifier keys open.
Keyboard with sticky modifiers.
iPhone Action Button bound to the Shortcut named Mac.
The trigger: an ordinary Action Button.

The hard parts

  1. 01

    Apple's dictation runs on the phone and has never heard of anything on this Mac, so the audit log filled with “cloth terminal” for “Claude terminal”. Transcription moved onto the Mac: whisper.cpp runs locally on the GPU and is told the vocabulary before it decides. Audio never leaves the machine.

  2. 02

    Remote voice control of an entire machine has to be safe. Commands are classified safe, sensitive, or blocked, and the blocked set has no override path, because any override would itself be reachable by voice.

  3. 03

    A finger dragging across a phone fires dozens of pointer events a second, so the app posts input events in-process instead of shelling out once per action.

What changed

  • Remote Mac control works end to end, with common system questions answered in under a second without calling a model.
  • The Builder writes software by voice, with Claude Code's real toolset pointed at a project directory.
  • A permanent phone app replaced the expiring share link: screen, microphone, keyboard with sticky modifiers, a real trackpad, native device pairing, and a lock-screen notice when the Mac is locked.
  • Two offline test suites cover routing, safety classification, transcript repair, session memory, and speech shaping.

Built with

  • Python
  • FastAPI
  • Claude Agent SDK
  • whisper.cpp
  • tmux
  • AppleScript
  • Tailscale
  • launchd
  • iOS Shortcuts