Skip to content
← All work

Voice-first AI assistant on Claude with tool use

Halden

A Jarvis-like personal assistant: talk or type, Claude answers aloud and calls tools for reminders, to-dos, notes, memories, weather, web lookups and Google Calendar, with a live dashboard on Neon Postgres.

My role: Full-stack and AI developer: Claude integration, tool design, voice interface, OAuth and database

▶ Watch the walkthrough (0:33)
  • Claude API
  • Tool use
  • Voice AI
  • Next.js
  • Neon Postgres

Overview

Halden is a voice-first personal AI assistant built with Next.js and the Claude API. Users speak or type, and Claude replies in natural conversation, spoken aloud, while calling tools that act on real data: reminders, to-dos, notes, long-term memories, weather, web lookups, calculations and Google Calendar scheduling through OAuth. A web dashboard shows everything the assistant is tracking, and all data is stored in Neon PostgreSQL. It uses live services throughout, with no mock responses.

The challenge

Most chat assistants only talk. A useful personal assistant has to act: set a reminder, remember a preference, check the weather or put a meeting in the calendar, and do it by voice without the user reaching for a keyboard.

Halden had to feel like a natural conversation while every action touched real data and real services, and it had to be honest when a service was not available instead of pretending.

How it works

  1. 1Speak or type
  2. 2Speech to text
  3. 3Claude + tool calls
  4. 4Neon Postgres / APIs
  5. 5Streamed reply
  6. 6Spoken aloud

Architecture

A Next.js App Router app streams the conversation with the Claude API. Each capability (reminders, to-dos, notes, memories, weather, Wikipedia search, calculations, calendar) is a tool definition; Claude decides when to call one, the server runs it against Neon Postgres or the external API, and Claude explains the result. A system prompt keeps the assistant's personality consistent across turns.

Voice uses browser speech recognition for input and speech synthesis for output, with Deepgram, OpenAI and ElevenLabs available as configured providers. A full-screen voice mode shows listening, thinking and speaking states with captions.

Google Calendar is connected through a real OAuth 2.0 flow from Settings, and events are created through the Calendar API after Claude confirms the details. Conversations, reminders, to-dos, notes and memories are stored in Neon PostgreSQL and shown on the dashboard.

What I built

Streaming conversation with Claude

Multi-turn chat with streaming replies, saved conversation history and a consistent assistant personality from the system prompt; Claude decides when to call a tool and explains the result in plain language.

Voice in and out

Speak or type; replies are read aloud. A full-screen voice mode shows listening, thinking and speaking states with live captions, so the assistant works hands-free.

Tools for everyday tasks

Weather from Open-Meteo, reminders, to-dos, notes and long-term memories in Neon Postgres, Wikipedia search, calculations and unit conversion, each exposed to Claude as a defined tool.

Google Calendar via OAuth

A real OAuth 2.0 connect flow from Settings; once connected, Claude schedules events in the user's calendar through the Calendar API after confirming the details.

Dashboard

Today's reminders and to-dos, recent conversations, saved notes, memories and settings, responsive on desktop and phone.

Honest failure handling

No simulated results: if a service such as Google Calendar or a voice provider is not configured, the app says so clearly instead of inventing an answer.

Engineering decisions

  • Let Claude choose tools from clear definitions, so new abilities are added as tools rather than prompt rules.
  • Confirm details before actions with side effects, such as creating a calendar event.
  • Store memories, notes and tasks in Postgres, so the assistant remembers across conversations.
  • Never fake a result: an unconfigured service is reported, not simulated.

Results

  • Live demo on Vercel with a walkthrough video.
  • Real tool calls against Postgres and external APIs, with no mock responses.
  • Hands-free voice mode with live captions.
  • Google Calendar scheduling through OAuth 2.0.

Stack

App
Next.js (App Router), React, TypeScript, Tailwind CSS
AI
Anthropic Claude API (Claude Sonnet), streaming responses and tool use
Voice
Browser speech recognition and synthesis; Deepgram, OpenAI and ElevenLabs providers via configuration
Integrations
Google Calendar API (OAuth 2.0), Open-Meteo weather, Wikipedia search
Database
Neon PostgreSQL (conversations, reminders, to-dos, notes, memories)
Hosting
Vercel

Need something like this?

I can build a version of this for your product, your data and your stack.

Start a project