# Our AI examiner was a black box. SigNoz told us what a student actually costs

Hi, I am Yaman, co-founder of [Mockacad](https://mockacad.com). We are building a universal adaptive mock test preparation platform. Students use it to prepare for exams like IELTS, PTE, SAT, JFT and many more, and two of our features run on AI in production every single day. The first is our voice examiner: a Pipecat agent on Gemini Live that conducts real speaking tests over WebRTC. The second is instant AI grading, which scores writing and speaking attempts minutes after submission. It is the feature our users love most.

Both features worked fine. The problem was that I could not answer the two questions that mattered most. Why does the voice agent sometimes feel slow? And what does one student actually cost us in AI spend? We were about to make subscription pricing decisions based on guesses.

For the Agents of SigNoz pre-hackathon challenge, I self-hosted SigNoz and instrumented the whole path: voice agent, NestJS backend, Postgres, and the Vertex AI calls behind grading. A week later we have per-turn latency numbers, a price-per-student figure, and two discoveries I did not see coming.

## Self-hosting SigNoz, and the two gotchas nobody mentions

SigNoz now installs through its Foundry CLI. A `casting.yaml` and one command generate the whole Docker Compose stack (ClickHouse, ingester, UI):

```bash
curl -fsSL https://signoz.io/foundry.sh | bash
foundryctl forge   # generate compose files without starting them
docker compose -f pours/deployment/compose.yaml up -d
```

I used `forge` instead of `cast` because our VM already had an app on port 8080, which is SigNoz's default UI port. So I remapped ports first, and bound OTLP ingest to `127.0.0.1` so telemetry is never exposed to the internet.

Two things then bit me:

1.  **The OTLP ports accept connections but reset every request until the first admin user exists.** The ingester gets its pipeline config from the SigNoz server over OpAMP, and that handshake fails with `cannot create agent without orgId` until you register. The collector logs say "Everything is ready" while `curl` to port 4318 dies. Very confusing until you know.
    
2.  **SigNoz ships without a JWT secret** and logs a critical warning about it. Set `SIGNOZ_TOKENIZER_JWT_SECRET` before exposing the UI anywhere.
    

## Instrumenting a production voice agent took three lines

Pipecat has OpenTelemetry support built in. This is the entire voice-side change:

```python
task = PipelineTask(
    pipeline,
    params=PipelineParams(enable_metrics=True, enable_usage_metrics=True),
    enable_tracing=True,
    enable_turn_tracking=True,
    conversation_id=session.session_id,
)
```

Plus an OTLP exporter at startup. Every conversation now produces a span tree (`conversation`, then `turn`, then `llm_response`) carrying per-turn token counts, time to first byte, and user-to-bot latency for Gemini Live.

The NestJS backend needed zero code changes. Node auto-instrumentation through `NODE_OPTIONS` and a few `OTEL_*` env vars in the pm2 app definition picked up Express, NestJS, and every Postgres query. One warning from experience: never `export NODE_OPTIONS` in the shell where you run pm2. The pm2 CLI is itself a Node process and it crashes trying to load the preload. Scope the env to the app definition instead.

For grading, we wrapped our single Vertex AI client so every LLM call reports its token usage plus a `purpose` attribute (`grading`, `transcription`, `translation`) and the quiz attempt it belongs to. One wrapper, and every AI feature in the product is accounted for.

## What the traces said about latency and cost

![Screenshot 2026-07-19 at 00.46.33](https://cdn.hashnode.com/uploads/covers/6277641edc122f5a4f21f37d/00bd73d5-bb93-4172-a18a-6d51eb2448eb.png align="center")

Here is the first real speaking test after instrumentation. It lasted 54 seconds and had 4 turns:

![Screenshot 2026-07-19 at 00.59.30](https://cdn.hashnode.com/uploads/covers/6277641edc122f5a4f21f37d/ad758665-5651-486a-99f7-861187b0f82f.png align="center")

*   **Input tokens per turn: 1,148 then 1,384 then 1,535 then 1,778.** Gemini Live re-sends the conversation context on every turn, so each turn costs more than the last. Session cost grows faster than session length. If you price long speaking sessions linearly, you are quietly subsidizing them.
    
*   User-to-bot latency became a p95 graph instead of a feeling. "The examiner sometimes lags" is now a number with a threshold and an alert on it.
    

## The alert I did not believe

![Screenshot 2026-07-19 at 01.01.23](https://cdn.hashnode.com/uploads/covers/6277641edc122f5a4f21f37d/54e87899-332c-43be-bc02-0715b684054d.png align="center")

We set a simple guardrail: more than 300k LLM tokens in one hour sends a Discord ping. Two days in, it fired with a current value of 322,155. Our token dashboard said around 148k. I assumed the alert was broken and went into ClickHouse to prove it wrong.

![Screenshot 2026-07-19 at 01.08.48](https://cdn.hashnode.com/uploads/covers/6277641edc122f5a4f21f37d/214618b4-d6b2-4b30-b98a-149ae782b97e.png align="center")

The alert was right, twice over. First, the dashboard panel was scoped to the voice agent only. Grading traffic, which had just started that day, was invisible to it. Second, and much more interesting: the grading spans showed a `totalTokenCount` of 367,911 against input plus output of only 203,956. The missing 164k tokens were **Gemini 2.5's thinking tokens**. They are billed at output rates, but they do not appear in the input or output counts. About 40% of our grading spend was hidden reasoning that our cost formulas never saw.

We fixed the price formula to charge the output rate on `total minus input`, and because the price mapping lives in the dashboard query rather than in code, all our historical data corrected itself instantly.

![Screenshot 2026-07-19 at 01.10.10](https://cdn.hashnode.com/uploads/covers/6277641edc122f5a4f21f37d/79a12959-6c4f-48cf-a98d-74690ddb2dab.png align="center")

That was not the only time alerting proved useful early. Within thirty minutes of creating an error-spike rule, it caught our own frontend middleware hammering `/auth/profile` with an expired token at 12 requests per second. A silent retry loop that had probably been running for months.

## What a student costs

![Screenshot 2026-07-19 at 01.13.50](https://cdn.hashnode.com/uploads/covers/6277641edc122f5a4f21f37d/ecc675d3-8309-441c-96ae-5c9e90cbc6f6.png align="center")

With tokens tagged by purpose and attempt, the pricing questions finally have real answers (at Gemini list rates, so treat them as close estimates):

*   Voice examiner: about **$0.039 per conversation minute**, roughly $2.30 per hour of live speaking test (measured across 74 conversation minutes)
    
*   Instant grading: about **$0.32 per graded attempt**, around 15 LLM calls and 170k tokens per attempt, thinking included (measured across 74 graded attempts)
    
*   A 15-minute speaking test plus AI grading: about **$0.90 per student. Measured, not guessed.**
    

That number, and how it scales with session length and question count, is now the foundation of our subscription pricing. It came from a dashboard, not a spreadsheet of assumptions.

## Takeaways

*   **A voice agent is three lines away from full observability** if your framework speaks OpenTelemetry. The hard part is deciding what to measure, not how.
    
*   **Gemini's thinking tokens hide inside** `totalTokenCount`**.** If your cost math uses input plus output, you are undercounting every reasoning model. Price output as total minus input.
    
*   **An alert you do not believe is an alert doing its job.** Checking it against raw ClickHouse taught us more about our spend than any dashboard.
    
*   **Instrument the process, not the codebase.** And never let `NODE_OPTIONS` leak into pm2's own CLI.
    
*   **The OpAMP first-admin gotcha** will convince you SigNoz ingest is broken when it is not.
    
*   What I would do differently: put SigNoz on its own small VM from day one. Right now our observability lives on the same box as production, which means the smoke detector is plugged into the stove.
    

## What's next

![Screenshot 2026-07-19 at 01.21.22](https://cdn.hashnode.com/uploads/covers/6277641edc122f5a4f21f37d/79894928-88a8-488b-8cda-8dfd30e00247.png align="center")

This stack has since grown into seven dashboards: per-attempt debugging (paste an attempt ID and see every request, payload, token, and error it caused), grading failure tracking, VM health, and distributed traces from our in-development MCP server that stitch a laptop tool call to a cloud SQL query in one flamegraph. That is our hackathon submission story for next week.

*Setup: SigNoz self-hosted via Foundry on Debian 12, Pipecat 1.2.1 native tracing, OTel JS auto-instrumentations 0.78 on NestJS. Your latencies and prices will differ. Measure them.*

## References

*   [SigNoz docs: Docker self-host install](https://signoz.io/docs/install/docker/)
    
*   [Pipecat docs: OpenTelemetry tracing](https://docs.pipecat.ai/server/utilities/opentelemetry)
    
*   [OpenTelemetry JS: zero-code instrumentation](https://opentelemetry.io/docs/zero-code/js/)
    
*   [Gemini API docs: thinking](https://ai.google.dev/gemini-api/docs/thinking)
    
*   [Vertex AI generative AI pricing](https://cloud.google.com/vertex-ai/generative-ai/pricing)