Defined in: realtime-client.ts

Client for managing realtime voice conversations.

Handles connection lifecycle, audio I/O, message state, and tool execution for realtime voice-to-voice AI interactions. This is the framework-agnostic core that powers useRealtimeChat in React.

import { RealtimeClient } from '@tanstack/ai-client'
import { openaiRealtime } from '@tanstack/ai-openai'

const client = new RealtimeClient({
  getToken: () => fetch('/api/realtime-token').then(r => r.json()),
  adapter: openaiRealtime(),
  tools: [myTool.client(handler)],
  onMessage: (msg) => console.log('Message:', msg),
})

await client.connect()

new RealtimeClient(options): RealtimeClient;

RealtimeClientOptions

Configuration options for the client.

RealtimeClient

readonly status: RealtimeStatus;

Current connection status ('idle', 'connecting', 'connected', 'reconnecting', 'error').

readonly mode: RealtimeMode;

Current session mode ('idle', 'listening', 'thinking', 'speaking').

readonly messages: Array<RealtimeMessage>;

Array of conversation messages. Updated as transcripts are finalized and messages complete.

readonly error: Error | null;

Current error, if any.

readonly pendingUserTranscript: string | null;

Partial transcript of what the user is currently saying (before finalization).

readonly pendingAssistantTranscript: string | null;

Partial transcript of the assistant's current response (while speaking).

readonly audio: AudioVisualization | null;

Audio visualization data for the current connection. Returns null when not connected.

connect(): Promise<void>;

Connect to the realtime session. Fetches a token via getToken() and establishes the connection through the adapter.

Promise<void>

If token fetch or connection fails.

disconnect(): Promise<void>;

Disconnect from the realtime session. Cleans up audio resources, event subscriptions, and token refresh timers.

Promise<void>

startListening(): void;

Start listening for voice input. Only needed when vadMode is 'manual'.

void

stopListening(): void;

Stop listening for voice input. Only needed when vadMode is 'manual'.

void

interrupt(): void;

Interrupt the current assistant response.

void

sendText(text): void;

Send a text message instead of voice.

string

void

sendImage(imageData, mimeType): void;

Send an image to the conversation.

string

Base64-encoded image data or a URL.

string

MIME type of the image (e.g., 'image/png', 'image/jpeg').

void

onStateChange(callback): () => void;

Subscribe to state changes. The callback is invoked whenever the internal state updates (status, mode, messages, transcripts, errors).

RealtimeStateChangeCallback

() => void

Unsubscribe function.

destroy(): void;

Clean up all resources. Disconnects, clears subscriptions, and releases audio resources. Call this when disposing of the client.

void