ENGINEERING CASE · JULY 2026
Your agent is
never deployed.
Pinecall is a real-time voice AI platform — phone, SIP, browser, WhatsApp and chat, one agent. The part that surprises people: your agent answers a phone call in the cloud, but when the model calls one of your tools, that function runs on your machine. Against your database. Behind your firewall. With credentials that never leave your process.
No webhooks. No tunnel. No deploy step. Just a WebSocket you already have open.
an engineering case by Bernardo Castro — co-founded and built by two people over a year, with clients live on it the whole time
20000+
calls answered in production
5
channels · one agent runtime
26
provider adapters · 9 STT · 6 TTS · 11 LLM
4
commands from nothing to a talking agent
1
runtime dependency in the SDK
31
published SDK releases
17
languages · the recognizer is picked for you
12
agent skills so your AI writes it right
01 · THE MENTAL MODEL
Three pieces, one socket.
Most voice platforms make you publish an endpoint. They call you; you'd better be reachable from the public internet, and your secrets had better be wherever that endpoint lives. Pinecall inverts it: your process dials out once and keeps the line open. Everything afterwards — a call arriving, a model deciding to use a tool, you changing the prompt — travels on that one connection, in both directions.
YOUR PROCESS
Your business logic.
The tools, their implementations, your database client, your API keys. Runs on your laptop while you build, on your server in production. Never uploaded.
ONE WEBSOCKET
The whole API surface.
Not a transport detail — it is the product boundary. Registration, config changes, call events, tool calls and their results all share it. There is no second channel to keep in sync.
THE MEDIA SERVER
The hard real-time part.
Carrier audio, speech recognition, the model, speech synthesis, and deciding when a human stopped talking. The part you don't want to build. Section 04.
WHAT THIS BUYS YOU
The agent can reach things that are not on the internet. An internal orders API, a Postgres box inside a VPC, a legacy system behind a VPN, a payment credential that compliance will never let you paste into a third-party dashboard. The voice pipeline is a managed service; the part that touches your data isn't hosted anywhere — because it's just your own program, and it never moved.
THE SHAPE OF AN AGENT
An agent is a function of state and events, not a server. You declare what it knows, which model speaks for it, and what it's allowed to do. The SDK is 82 files and one runtime dependency — the envelope that carries all of this is 15 lines of type definitions.
import { Pinecall, tool } from "@pinecall/sdk";
const pc = new Pinecall();
const getOrderStatus = tool({
name: "getOrderStatus",
description: "Look up the shipping status of an order.",
schema: z.object({ orderId: z.string() }),
execute: async ({ orderId }) => {
// ← this line runs in YOUR process, mid-call
return db.orders.findOne({ orderId });
},
});
pc.agent("support", { prompt, llm: "openai/gpt-4.1-mini", tools: [getOrderStatus] });
there is no route to register, no signature to verify, no public URL — the process that defines the tool is the process that runs it
02 · WHY IT SURVIVES A BAD NETWORK
A long-lived socket is a promise you have to keep.
Betting the whole API on one connection is only a good idea if the connection is treated as unreliable by default. Laptops sleep. Wi-Fi drops. Deploys restart processes. The interesting engineering isn't opening the socket — it's what happens the moment it dies, because a naive reconnect leaves you with an agent that is technically online and functionally broken.
NOTHING IS DROPPED
Calls made before the server confirms registration are queued, not discarded. You can write straight-line startup code without racing the handshake.
RECONNECT IS NOT REOPEN
On reconnect the SDK re-sends the agent's registered phone numbers and channel credentials before flushing the queue — and skips the ones it just replayed. Skip that and you come back as an agent with no phone.
BACKOFF WITH JITTER
1s, doubling, capped at 30s, plus 0–25% random jitter — so a server restart doesn't get a synchronized stampede from every agent that was connected to it.
THE ONE CASE WHERE RECONNECTING IS WRONG
Two processes claiming the same agent name: the newest wins and the old one is displaced. If the loser reconnected, the two would fight forever, each kicking the other off, and the agent would flap between two versions of your code. So the SDK reads the close reason and deliberately stops trying. Last writer wins, permanently.
AND THAT IS ALSO THE HOT RELOAD
Displacement isn't a failure mode bolted on — it's the update mechanism. Re-running your file replaces the live agent instantly, no deploy and no downtime. The flip side is worth knowing before it bites you: run your file with a production key and you have just taken over production from your laptop. The fix is a one-line environment-driven name, and it's in the docs because it happened.
03 · GETTING SOMETHING TALKING
Four commands. No phone number.
The slowest part of building a voice agent used to be everything around the agent: buying a number, pointing it at a webhook, exposing your laptop to the internet, waiting on a deploy to see whether a prompt change helped. None of that is on the critical path here. You can iterate on behaviour for an afternoon and never touch telephony at all.
~ $ pinecall signup # opens the browser, gives you a key
~ $ export PINECALL_API_KEY=pk_…
~ $ pinecall run agent.ts # registers over the socket
~ $ pinecall chat # talk to it right now, in your terminal
▸ tool calls stream inline as they fire, with their arguments and results
CHANGE THINGS WHILE THE CALL IS RUNNING
Because config travels on the same open socket, an agent's prompt, voice, language, speech recognizer and tools can all change during an active call — applied on the model's next turn. That's a support agent switching to Spanish when it hears Spanish, or a prompt gaining the caller's account context the moment you identify them.
YOUR AI ASSISTANT ALREADY KNOWS THE API
The docs ship as 12 installable agent skills — one command, and Claude Code or Cursor writes Pinecall code against the real API instead of a plausible hallucination of it. They carry the constraints too, the things a model would otherwise get confidently wrong: which recognizer names are real, which knobs are auto-derived and must not be set by hand.
04 · THE PART NOBODY ELSE HAS
A voice agent you can test like software.
Ask how a team validates their voice agent and the honest answer is usually: someone phones it and listens. That doesn't scale, doesn't run in CI, and doesn't catch the regression you shipped at 6pm on a Friday. Pinecall ships a test runner. You write what should happen in plain English, an LLM plays the customer, and the process exits non-zero when your agent misbehaves.
agent: tooltest
workflow: |
You are a customer asking about a shipment.
1. Ask exactly: "Hi, can you check the status of order ABC-123?"
2. The agent must answer with the CONCRETE details of that order.
Call test_passed if the reply states the order is held in customs
AND mentions Rotterdam.
Call test_failed if the agent invents a different status or port,
says it cannot look up orders, or answers vaguely.
no assertion DSL, no mocking framework, no recorded fixtures — the test is a description of the conversation you expect, which is also the only honest specification a conversation has
THE JUDGE PLAYS BOTH PARTS
One neat trick carries the whole design: the judge's text is the customer's next message, and its tool call is the verdict. So the same model that is grading the conversation is also having it — which means it can probe, follow up, and change tack based on what your agent just said, instead of replaying a fixed script.
RUNS WHEREVER YOU NEED IT TO
Five judge providers, including a local model over Ollama — so the suite costs nothing to run and doesn't leave your machine. Exit code 1 on failure, JSON output for CI, and a --grep for the one spec you're iterating on.
~ $
┌ Tester [1]
│ Hi, can you check the status of order ABC-123?
└
┌ Agent
│ Your order ABC-123 is currently held in customs at the port of
│ Rotterdam. It is shipping with Maersk. Estimated arrival: 6 days.
│
│ ⚡ getOrderStatus({"orderId":"ABC-123"})
└
✓ PASS (16.2s, 1 turns)
1 passed, 0 failed
the tool returns a port, a carrier and an ETA — strings no amount of plausible-sounding hallucination arrives at. if the agent says Rotterdam, the function ran on the developer's machine and the result made the round trip
AND THE SAME SPEC, AS AN ACTUAL PHONE CALL
Text tests can't catch the failures that only exist in speech — interrupting, talking over each other, a caller who trails off mid-sentence. So the runner has a voice mode, and it does the obvious brutal thing: it spins up a second, throwaway Pinecall agent and has it call yours. Both sides run the full stack — recognizer, model, synthesis, turn-taking. It records both voices to a WAV, writes the transcript next to it, and opens a live player in your browser so you can listen while it happens. To your agent it is an ordinary inbound call, because it is one.
05 · WHAT REAL CALLERS DO
People interrupt. That's the whole problem.
A demo works because the demo takes turns politely. Real callers don't. They cut you off mid-sentence, they say "no wait—", they pause to think and get talked over for it. Handling that is the difference between a voice agent people use and one they hang up on, and it's most of the engineering underneath this platform. 20,000+ production calls is where these rules came from.
WHEN YOU INTERRUPT, IT STOPS. ACTUALLY STOPS.
Telling the synthesizer to stop isn't enough — audio already in flight keeps arriving and refills the phone network's buffer, so the caller hears a sentence they already cut off. There's a separate output gate that closes first and refuses to reopen for the aborted reply's leftovers. Two independent guards, because there were two independent bugs.
IT ONLY REMEMBERS WHAT YOU HEARD
Cut the agent off halfway and the model believes it said the whole paragraph — so it answers your next question as if you'd heard the part you didn't. On interruption the synthesizer reports back how far it actually got, and the conversation history is truncated to exactly that. The agent's memory matches the caller's.
"NO WAIT—" ISN'T A NEW QUESTION
There's a real fork here. Interrupt in the first two seconds and you were probably correcting yourself — the turn continues. After that, you meant it, and it becomes a new turn with the half-spoken reply recorded as interrupted. One threshold, two completely different conversations.
THE TURN IS A STATE MACHINE, NOT A PILE OF IFS
Five states, and a table that declares only the legal moves — roughly two thirds of the possible transitions are impossible by construction and raise rather than corrupt the call. It's 159 lines with no I/O in it, which is why it can be tested exhaustively: one of its tests walks the table itself, so a new rule arrives with coverage attached.
WHY A STATE MACHINE WAS NECESSARY · FOUR VENDORS, FOUR DEFINITIONS OF "DONE TALKING"
| Deepgram Flux | announces the turn ended, then sometimes takes it back — the caller was only pausing |
| ElevenLabs Scribe | says the turn ended before delivering the last words of it |
| AssemblyAI | signals it through a formatting flag on the transcript |
| Soniox | has no end-of-turn message at all — it emits a sentinel token in the stream |
every one of those is reasonable on its own terms, and no two agree. the state machine is where that disagreement is turned into one consistent answer, so the model above it never has to care which recognizer it's listening to
06 · NOT BETTING ON ONE VENDOR
Swap the model with a string.
A voice agent is three vendors deep — one to hear, one to think, one to speak — and all three are repricing and shipping new models constantly. Being locked to any of them is a commercial risk, not just a technical one. So all 26 adapters sit behind the same shape, and choosing one is a configuration string, not a migration.
26 ADAPTERS, THREE CATEGORIES
THE HONEST PART
Five of the eleven model adapters are about fifty lines — they speak a compatible dialect, so they cost almost nothing to add. The other two are 474 and 363 lines, because Anthropic and Google stream tool calls in genuinely different shapes and one of them rejects conversation histories the other accepts. An abstraction that claims all vendors are the same is lying; this one absorbs the difference so your agent code doesn't have to.
BRING YOUR OWN KEYS
Use your own vendor accounts and your usage is billed to you, not resold. One function decides key-and-billing together, so there's no path where the platform spends its own key while marking the call free.
SENSIBLE BY DEFAULT
Say the language and the recognizer is chosen for you — lowest-latency option where it's supported, a different vendor for the languages it isn't. Turn detection is derived from that choice rather than being another knob to get wrong.
FAILS AT STARTUP, NOT MID-CALL
Configure a provider that needs your key without supplying one and registration is rejected immediately, by name. The alternative is discovering it on a real call, in front of a real customer.
07 · FIVE CHANNELS, ONE AGENT
Write it once. Answer everywhere.
The same agent — same prompt, same tools, same conversation history — answers a phone call, a SIP intercom, a widget on your website, a WhatsApp thread and a text chat. Not five integrations that happen to share a name: one runtime, with the physical differences absorbed underneath it.
AUDIO ARRIVES IN THREE INCOMPATIBLE FORMATS
A phone line is 8kHz μ-law, a browser is 48kHz Opus, a raw socket is 16kHz PCM. Every transport normalizes before the pipeline sees it, so the recognizer, the turn logic and the model are written once. Where a vendor accepts phone audio natively, the fast path skips the conversion entirely rather than transcoding for the sake of uniformity.
NATIVE MOBILE, NOT A WEBVIEW
The React Native and Ionic SDKs put an AI agent behind a real system call — CallKit on iOS, with the real incoming-call screen, proper audio-session routing and Bluetooth handling. It's the WhatsApp-call experience, with an agent on the other end. Doing this properly requires handing audio control to the OS and waiting for it to hand it back — the exact thing a browser-based stack inside an app cannot do.
08 · HOW IT WAS ACTUALLY BUILT
Two people. A year. Clients already on it.
For its first year Pinecall was two people: my co-founder Gabriel Cardama and me. No team to hand the unglamorous parts to, no separate group doing integrations while we did architecture. We took client work, shipped it, and learned what actually breaks on a phone call from the calls themselves — every design decision on this page was made with someone's production traffic already running through it.
That constraint shaped everything. Two people cannot afford a system only one of them understands, or a bug class that needs a heroic debugging session every time it recurs. So the rules became tables, the event handling became a registry, and the reason behind every strange decision got written down next to it. Not process for its own sake — it was the only way two people could keep several clients live at once.
IT OUTGREW US
Today Gabriel and two more engineers run the platform for those clients. I'm not in that loop, and it keeps answering calls whether or not I'm reachable — which is the only real proof that a handoff worked. What I'd point at in an interview isn't the architecture. It's that.
A CONSULTING TOOL BECOMING A PRODUCT
Pinecall started as our own framework — the thing that let two people deliver voice projects at a pace two people shouldn't manage. The SDK, the CLI and the test runner exist because we needed them on real deadlines. That's now being opened up: the toolkit we built to do the work is becoming the product you can buy.
WHAT I'D DO NEXT
The test runner should refuse to report a pass on a conversation that never happened, and retry a judge that's rate-limited instead of reporting it as your agent's failure. Both are known, both are small, both are the kind of thing that quietly erodes trust in a green build. That's the next thing I'd fix.