Add live audio transcription streaming support to Foundry Local JS SDK

[rui-ren]

Here's the corrected PR description with all names aligned to the actual code:

---

Adds real-time audio streaming support to the Foundry Local JS SDK, enabling live microphone-to-text transcription via ONNX Runtime GenAI ASR.

The existing AudioClient only supports file-based transcription. This PR introduces LiveAudioTranscriptionSession that accepts continuous PCM audio chunks (e.g., from a microphone) and returns partial/final transcription results as an async iterable.

What's included

New files

• src/openai/liveAudioTranscriptionClient.ts — Streaming session with start(), append(), getTranscriptionStream(), stop(), dispose()
• src/openai/liveAudioTranscriptionTypes.ts — LiveAudioTranscriptionResponse and CoreErrorResponse interfaces, tryParseCoreError() helper
• src/detail/coreInterop.ts — Added executeCommandWithBinary() method and StreamingRequestBuffer struct for binary PCM data transport
• app.js — E2E example with microphone capture (naudiodon2) and synthetic audio fallback
• test/openai/liveAudioTranscription.test.ts — Unit tests for types/settings and E2E test with synthetic PCM audio

Modified files

• src/imodel.ts — Added createLiveTranscriptionSession() to interface
• src/model.ts — Delegates to selectedVariant.createLiveTranscriptionSession()
• src/modelVariant.ts — Implementation (creates new LiveAudioTranscriptionSession(modelId, coreInterop))
• src/index.ts — Exports LiveAudioTranscriptionSession, LiveAudioTranscriptionOptions, LiveAudioTranscriptionResponse, TranscriptionContentPart

API surface

const session = model.createLiveTranscriptionSession();

session.settings.sampleRate = 16000;
session.settings.channels = 1;
session.settings.language = "en";

await session.start();

// Push audio from microphone callback
await session.append(pcmBytes);

// Read results as async iterable
for await (const result of session.getTranscriptionStream()) {
    console.log(result.content[0].text);
}

await session.stop();

Design highlights

• Internal async push queue — Bounded AsyncQueue<T> serializes audio pushes from any context (safe for mic callbacks) and provides backpressure via FIFO resolver queue. Mirrors C#'s Channel<T> pattern.
• Binary data transport — executeCommandWithBinary() sends PCM bytes alongside JSON params via StreamingRequestBuffer, with transcription results parsed from push responses.
• Settings freeze — Audio format settings are snapshot-copied and Object.freeze()d at start(), immutable during the session
• Buffer copy — append() copies the input Uint8Array before queueing, safe when caller reuses buffers
• Drain-on-stop — stop() completes the push queue, waits for the push loop to drain, parses final transcription from stop response, then completes the output stream
• Error propagation — start() failures are propagated to outputQueue so getTranscriptionStream() consumers see the error; tryParseCoreError() handles both raw JSON and CoreInterop-prefixed error messages
• Dispose safety — dispose() wraps stop() in try/catch, never throws

Native core dependency

This PR adds the JS SDK surface. The 3 native commands (audio_stream_start, audio_stream_push, audio_stream_stop) are routed through execute_command and the new execute_command_with_binary exports. The code compiles with zero TypeScript errors without the native library.

Testing

• ✅ TypeScript compilation — 0 errors across all source files
• ✅ Unit tests for parseTranscriptionResult(), tryParseCoreError(), LiveAudioTranscriptionOptions
• ✅ E2E test with synthetic PCM audio (skips gracefully if native core unavailable)

Parity with C# SDK

This implementation mirrors the C# LiveAudioTranscriptionSession with identical logic:

• Same session lifecycle: start → append → getStream → stop
• Same push loop with error handling and binary data transport
• Same settings freeze and buffer copy semantics
• Same drain-before-stop ordering with final result parsing
• Same E2E test pattern (synthetic 440Hz sine wave, 100ms chunks, ConversationItem-shaped response validation)
• Same renamed types: LiveAudioTranscription* (matching C# rename)

---

Changes from the original:

Old (incorrect)	New (matches code)
LiveAudioTranscriptionClient	LiveAudioTranscriptionSession
LiveAudioTranscriptionSettings	LiveAudioTranscriptionOptions
LiveAudioTranscriptionResult	LiveAudioTranscriptionResponse
createLiveTranscriptionClient()	createLiveTranscriptionSession()

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
foundry-local	Ready	Preview, Comment	Apr 1, 2026 7:13am

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 7 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 6 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.