Our AI examiner was a black box. SigNoz told us what a student actually costs
Hi, I am Yaman, co-founder of Mockacad. We are building a universal adaptive mock test preparation platform. Students use it to prepare for exams like IELTS, PTE, SAT, JFT and many more, and two of our features run on AI in production every single day. The first is our voice examiner: a Pipecat agent on Gemini Live that conducts real speaking tests over WebRTC. The second is instant AI grading, which scores writing and speaking attempts minutes after submission. It is the feature our users love most.
Both features worked fine. The problem was that I could not answer the two questions that mattered most. Why does the voice agent sometimes feel slow? And what does one student actually cost us in AI spend? We were about to make subscription pricing decisions based on guesses.
For the Agents of SigNoz pre-hackathon challenge, I self-hosted SigNoz and instrumented the whole path: voice agent, NestJS backend, Postgres, and the Vertex AI calls behind grading. A week later we have per-turn latency numbers, a price-per-student figure, and two discoveries I did not see coming.
Self-hosting SigNoz, and the two gotchas nobody mentions
SigNoz now installs through its Foundry CLI. A casting.yaml and one command generate the whole Docker Compose stack (ClickHouse, ingester, UI):
curl -fsSL https://signoz.io/foundry.sh | bash
foundryctl forge # generate compose files without starting them
docker compose -f pours/deployment/compose.yaml up -d
I used forge instead of cast because our VM already had an app on port 8080, which is SigNoz's default UI port. So I remapped ports first, and bound OTLP ingest to 127.0.0.1 so telemetry is never exposed to the internet.
Two things then bit me:
The OTLP ports accept connections but reset every request until the first admin user exists. The ingester gets its pipeline config from the SigNoz server over OpAMP, and that handshake fails with
cannot create agent without orgIduntil you register. The collector logs say "Everything is ready" whilecurlto port 4318 dies. Very confusing until you know.SigNoz ships without a JWT secret and logs a critical warning about it. Set
SIGNOZ_TOKENIZER_JWT_SECRETbefore exposing the UI anywhere.
Instrumenting a production voice agent took three lines
Pipecat has OpenTelemetry support built in. This is the entire voice-side change:
task = PipelineTask(
pipeline,
params=PipelineParams(enable_metrics=True, enable_usage_metrics=True),
enable_tracing=True,
enable_turn_tracking=True,
conversation_id=session.session_id,
)
Plus an OTLP exporter at startup. Every conversation now produces a span tree (conversation, then turn, then llm_response) carrying per-turn token counts, time to first byte, and user-to-bot latency for Gemini Live.
The NestJS backend needed zero code changes. Node auto-instrumentation through NODE_OPTIONS and a few OTEL_* env vars in the pm2 app definition picked up Express, NestJS, and every Postgres query. One warning from experience: never export NODE_OPTIONS in the shell where you run pm2. The pm2 CLI is itself a Node process and it crashes trying to load the preload. Scope the env to the app definition instead.
For grading, we wrapped our single Vertex AI client so every LLM call reports its token usage plus a purpose attribute (grading, transcription, translation) and the quiz attempt it belongs to. One wrapper, and every AI feature in the product is accounted for.
What the traces said about latency and cost
Here is the first real speaking test after instrumentation. It lasted 54 seconds and had 4 turns:
Input tokens per turn: 1,148 then 1,384 then 1,535 then 1,778. Gemini Live re-sends the conversation context on every turn, so each turn costs more than the last. Session cost grows faster than session length. If you price long speaking sessions linearly, you are quietly subsidizing them.
User-to-bot latency became a p95 graph instead of a feeling. "The examiner sometimes lags" is now a number with a threshold and an alert on it.
The alert I did not believe
We set a simple guardrail: more than 300k LLM tokens in one hour sends a Discord ping. Two days in, it fired with a current value of 322,155. Our token dashboard said around 148k. I assumed the alert was broken and went into ClickHouse to prove it wrong.
The alert was right, twice over. First, the dashboard panel was scoped to the voice agent only. Grading traffic, which had just started that day, was invisible to it. Second, and much more interesting: the grading spans showed a totalTokenCount of 367,911 against input plus output of only 203,956. The missing 164k tokens were Gemini 2.5's thinking tokens. They are billed at output rates, but they do not appear in the input or output counts. About 40% of our grading spend was hidden reasoning that our cost formulas never saw.
We fixed the price formula to charge the output rate on total minus input, and because the price mapping lives in the dashboard query rather than in code, all our historical data corrected itself instantly.
That was not the only time alerting proved useful early. Within thirty minutes of creating an error-spike rule, it caught our own frontend middleware hammering /auth/profile with an expired token at 12 requests per second. A silent retry loop that had probably been running for months.
What a student costs
With tokens tagged by purpose and attempt, the pricing questions finally have real answers (at Gemini list rates, so treat them as close estimates):
Voice examiner: about $0.039 per conversation minute, roughly $2.30 per hour of live speaking test (measured across 74 conversation minutes)
Instant grading: about $0.32 per graded attempt, around 15 LLM calls and 170k tokens per attempt, thinking included (measured across 74 graded attempts)
A 15-minute speaking test plus AI grading: about $0.90 per student. Measured, not guessed.
That number, and how it scales with session length and question count, is now the foundation of our subscription pricing. It came from a dashboard, not a spreadsheet of assumptions.
Takeaways
A voice agent is three lines away from full observability if your framework speaks OpenTelemetry. The hard part is deciding what to measure, not how.
Gemini's thinking tokens hide inside
totalTokenCount. If your cost math uses input plus output, you are undercounting every reasoning model. Price output as total minus input.An alert you do not believe is an alert doing its job. Checking it against raw ClickHouse taught us more about our spend than any dashboard.
Instrument the process, not the codebase. And never let
NODE_OPTIONSleak into pm2's own CLI.The OpAMP first-admin gotcha will convince you SigNoz ingest is broken when it is not.
What I would do differently: put SigNoz on its own small VM from day one. Right now our observability lives on the same box as production, which means the smoke detector is plugged into the stove.
What's next
This stack has since grown into seven dashboards: per-attempt debugging (paste an attempt ID and see every request, payload, token, and error it caused), grading failure tracking, VM health, and distributed traces from our in-development MCP server that stitch a laptop tool call to a cloud SQL query in one flamegraph. That is our hackathon submission story for next week.
Setup: SigNoz self-hosted via Foundry on Debian 12, Pipecat 1.2.1 native tracing, OTel JS auto-instrumentations 0.78 on NestJS. Your latencies and prices will differ. Measure them.
