Getting started

Jack is a privacy-first, on-device voice & chat assistant for macOS. Everything runs locally — no audio, text, or memory leaves your machine, except a few opt-in, clearly-disclosed features (web search, an optional cloud LLM, and any remote MCP servers you connect).

Install

Grab the latest .dmg from GitHub Releases — a single, self-contained app (the orb + chat drawer plus the embedded engine). Voice models download on demand the first time you enable voice, so the app stays small. Drag Jack to Applications.

First-run setup

On first run Jack walks you through setup — the LLM provider, models, and permissions. You’ll need, locally, one of:

• Ollama running with a model (local mode, the default), or
• an Anthropic API key (cloud mode) — set it in the orb’s Settings.

You’ll also grant a few macOS permissions (Microphone, Accessibility, Automation). Settings → Permissions shows each one and links straight to the right System Settings pane.

Talking to Jack

Chat is the default. Type in the right-docked chat drawer; replies show as text and the mic stays off. Voice is off until you enable it (Settings → Listening), which downloads the speech models the first time.

Things to try:

• “what time is it” — read-only, runs straight through.
• “open YouTube” / “close youtube.com” — opens a site / closes that tab.
• “open Spotify”, “quit Mail”, “minimize Safari” — app control.
• “find my resume” — searches your files; pick a result to open or reveal it.
• “summarize what I copied” / “translate this and copy it back” — the copy → ask → paste loop: Jack reads the clipboard, transforms it, and (if asked) puts the result back.
• “how’s my battery”, “what’s my Wi-Fi” — system info.
• “empty the trash” — destructive, so Jack confirms first, then does it.

Voice & hands-free

Once voice is enabled, switch to it with the Voice toggle or ⌘⌃V. Start a command with “jack” — “jack, what’s the time” — said naturally. Jack transcribes each phrase and, if the wake word appears, strips it and runs the rest.

• Follow-ups: after a reply Jack keeps listening without the wake word for a window (default 30s); speak again and it just answers.
• Barge-in: talk over Jack to interrupt it (full-duplex when echo cancellation is available; otherwise it finishes, then listens).
• Dismiss: say “go away” and the orb hides; it also auto-hides when idle and returns on the wake word.
• Summon / hide: ⌘⌃J toggles Jack; ⌘⌃C switches to chat, ⌘⌃V to voice.

Configuration

All tunables live in one JSON file, ~/.autobot/settings.json — there are no environment variables. A missing file or key falls back to the built-in default. The orb’s Settings view edits the common ones live (no restart).

{ "llm_provider": "ollama", "llm_model": "qwen3:8b",
  "stt_engine": "faster_whisper", "stt_model": "small.en",
  "follow_up_window_s": 30, "barge_in": true, "aec": true }

Secrets (API keys) are never in this file — they live in the macOS Keychain. Local vs. cloud: ollama (default, fully on-device) or anthropic (Claude — sends the conversation + tool schemas to Anthropic, never audio, never the actions; opt-in and disclosed).

Permissions

Jack manages macOS permissions in one place — Settings → Permissions shows each with its status and a button that opens the exact System Settings pane:

• Microphone — to hear you.
• Accessibility — show/hide/minimize/list app windows.
• Automation — control other apps via Apple Events (close browser tabs, quit apps, empty the Trash).

If a tool needs a permission that isn’t granted, Jack refuses cleanly, opens the right pane, and you grant it once.

Web search (optional, off-device)

Off by default. When enabled, it sends only your search query to a provider; the local LLM summarizes the results. Enable it with "allow_web": true in settings (or the Settings → Capabilities toggle).

Connect tools (MCP)

Jack speaks the Model Context Protocol, so you can extend it with external tool servers — GitHub, Notion, Slack, a local filesystem server, or your own. Their tools show up alongside Jack’s built-ins and run through the same permission gate. MCP is off until you turn it on.

Add one in Settings → Connections: pick a server from the catalog (or add a custom one), choose how it connects — local (a stdio process on your Mac) or remote (an HTTP server) — and sign in with OAuth or paste an access token. Tokens go in the macOS Keychain, never on disk.

• On-device vs. off-device — local (stdio) servers run entirely on your Mac. Remote servers send their requests off-device, so they’re marked with a ↗ badge, listed in Settings → Privacy, and a write to one asks before it sends.
• Per-tool control — the connection detail lists every tool with its risk; switch off any you don’t want Jack to use. If a connected tool’s definition later changes in a way that raises its risk, Jack holds it for your re-consent.
• Stays fast — Jack advertises only the tools relevant to your request, so connecting several servers doesn’t bloat the prompt or slow a simple ask.

Server definitions live in ~/.autobot/mcp/servers.json (config only — secrets stay in the Keychain). Enable MCP and manage connections from Settings → Connections.

Build from source

For developers on Apple Silicon (tested on MacBook Air M2, macOS 15): install uv and Ollama, then:

brew install uv ollama
ollama serve            # leave running
ollama pull qwen3:8b    # ~5 GB, one time
cd /path/to/autobot
make setup              # dev env + git hooks
make run                # the engine/daemon

In dev the engine runs on its own; build the orb separately from ui/orb-shell (cargo tauri dev). Full details are in the README.

Logs & debugging

The easiest report: in the orb, Settings → Raise an issue (or the tray menu) builds a compact, redacted snapshot — recent events, errors, the state sequence, config (no audio, no API keys) — copies it to your clipboard, and can open a prefilled GitHub issue. Session transcripts are written to ~/.autobot/sessions/; the full rotating log is at ~/.autobot/logs/autobot.log.

Troubleshooting

• Engine won’t connect / model not found → ensure ollama serve is running and you’ve pulled the model, or switch to cloud mode and set the Anthropic key in Settings.
• Jack can’t control an app / close a tab → grant Automation (and Accessibility) in Settings → Permissions.
• Barge-in mishears its own voice → that’s AEC not cancelling; it falls back to half-duplex, or use headphones. Toggle barge_in.
• No audio → confirm Microphone permission.