Both features worked fine. The problem was that I could not answer the two questions that mattered most. Why does the voice agent sometimes feel slow? And what does one student actually cost us in AI spend? We were about to make subscription pricing decisions based on guesses.

For the Agents of SigNoz pre-hackathon challenge, I self-hosted SigNoz and instrumented the whole path: voice agent, NestJS backend, Postgres, and the Vertex AI calls behind grading. A week later we have per-turn latency numbers, a price-per-student figure, and two discoveries I did not see coming.

Self-hosting SigNoz, and the two gotchas nobody mentions

SigNoz now installs through its Foundry CLI. A casting.yaml and one command generate the whole Docker Compose stack (ClickHouse, ingester, UI):

curl -fsSL https://signoz.io/foundry.sh | bash
foundryctl forge   # generate compose files without starting them
docker compose -f pours/deployment/compose.yaml up -d

I used forge instead of cast because our VM already had an app on port 8080, which is SigNoz's default UI port. So I remapped ports first, and bound OTLP ingest to 127.0.0.1 so telemetry is never exposed to the internet.

Two things then bit me:

1. The OTLP ports accept connections but reset every request until the first admin user exists. The ingester gets its pipeline config from the SigNoz server over OpAMP, and that handshake fails with cannot create agent without orgId until you register. The collector logs say "Everything is ready" while curl to port 4318 dies. Very confusing until you know.

2. SigNoz ships without a JWT secret and logs a critical warning about it. Set SIGNOZ_TOKENIZER_JWT_SECRET before exposing the UI anywhere.

Instrumenting a production voice agent took three lines

Pipecat has OpenTelemetry support built in. This is the entire voice-side change:

task = PipelineTask(
    pipeline,
    params=PipelineParams(enable_metrics=True, enable_usage_metrics=True),
    enable_tracing=True,
    enable_turn_tracking=True,
    conversation_id=session.session_id,
)

Plus an OTLP exporter at startup. Every conversation now produces a span tree (conversation, then turn, then llm_response) carrying per-turn token counts, time to first byte, and user-to-bot latency for Gemini Live.

The NestJS backend needed zero code changes. Node auto-instrumentation through NODE_OPTIONS and a few OTEL_* env vars in the pm2 app definition picked up Express, NestJS, and every Postgres query. One warning from experience: never export NODE_OPTIONS in the shell where you run pm2. The pm2 CLI is itself a Node process and it crashes trying to load the preload. Scope the env to the app definition instead.

For grading, we wrapped our single Vertex AI client so every LLM call reports its token usage plus a purpose attribute (grading, transcription, translation) and the quiz attempt it belongs to. One wrapper, and every AI feature in the product is accounted for.

What the traces said about latency and cost

Here is the first real speaking test after instrumentation. It lasted 54 seconds and had 4 turns:

• Input tokens per turn: 1,148 then 1,384 then 1,535 then 1,778. Gemini Live re-sends the conversation context on every turn, so each turn costs more than the last. Session cost grows faster than session length. If you price long speaking sessions linearly, you are quietly subsidizing them.

• User-to-bot latency became a p95 graph instead of a feeling. "The examiner sometimes lags" is now a number with a threshold and an alert on it.

The alert I did not believe

We set a simple guardrail: more than 300k LLM tokens in one hour sends a Discord ping. Two days in, it fired with a current value of 322,155. Our token dashboard said around 148k. I assumed the alert was broken and went into ClickHouse to prove it wrong.

The alert was right, twice over. First, the dashboard panel was scoped to the voice agent only. Grading traffic, which had just started that day, was invisible to it. Second, and much more interesting: the grading spans showed a totalTokenCount of 367,911 against input plus output of only 203,956. The missing 164k tokens were Gemini 2.5's thinking tokens. They are billed at output rates, but they do not appear in the input or output counts. About 40% of our grading spend was hidden reasoning that our cost formulas never saw.

We fixed the price formula to charge the output rate on total minus input, and because the price mapping lives in the dashboard query rather than in code, all our historical data corrected itself instantly.

That was not the only time alerting proved useful early. Within thirty minutes of creating an error-spike rule, it caught our own frontend middleware hammering /auth/profile with an expired token at 12 requests per second. A silent retry loop that had probably been running for months.

What a student costs

With tokens tagged by purpose and attempt, the pricing questions finally have real answers (at Gemini list rates, so treat them as close estimates):

• Voice examiner: about $0.039 per conversation minute, roughly $2.30 per hour of live speaking test (measured across 74 conversation minutes)

• Instant grading: about $0.32 per graded attempt, around 15 LLM calls and 170k tokens per attempt, thinking included (measured across 74 graded attempts)

• A 15-minute speaking test plus AI grading: about $0.90 per student. Measured, not guessed.

That number, and how it scales with session length and question count, is now the foundation of our subscription pricing. It came from a dashboard, not a spreadsheet of assumptions.

Takeaways

• A voice agent is three lines away from full observability if your framework speaks OpenTelemetry. The hard part is deciding what to measure, not how.

• Gemini's thinking tokens hide inside totalTokenCount. If your cost math uses input plus output, you are undercounting every reasoning model. Price output as total minus input.

• An alert you do not believe is an alert doing its job. Checking it against raw ClickHouse taught us more about our spend than any dashboard.

• Instrument the process, not the codebase. And never let NODE_OPTIONS leak into pm2's own CLI.

• The OpAMP first-admin gotcha will convince you SigNoz ingest is broken when it is not.

• What I would do differently: put SigNoz on its own small VM from day one. Right now our observability lives on the same box as production, which means the smoke detector is plugged into the stove.

What's next

This stack has since grown into seven dashboards: per-attempt debugging (paste an attempt ID and see every request, payload, token, and error it caused), grading failure tracking, VM health, and distributed traces from our in-development MCP server that stitch a laptop tool call to a cloud SQL query in one flamegraph. That is our hackathon submission story for next week.

Setup: SigNoz self-hosted via Foundry on Debian 12, Pipecat 1.2.1 native tracing, OTel JS auto-instrumentations 0.78 on NestJS. Your latencies and prices will differ. Measure them.

References

• SigNoz docs: Docker self-host install

• Pipecat docs: OpenTelemetry tracing

• OpenTelemetry JS: zero-code instrumentation

• Gemini API docs: thinking

• Vertex AI generative AI pricing

Under the Action column, the Buttons would trigger some fetch API calls. I had to implement a click-listener for each row, which would show the detailed information of that row on a modal component. Everything went well. Modal was working fine. But, when I clicked on the Buttons of Action column, it triggered the fetch API call and the modal would open too.

This was an issue because no one would want to see a modal pop-up on their screen every time they clicked another action button. My brain stopped working for a moment as I could not figure out what was happening there. On further investigating the issue, I found out there is something called event propagation on the DOM tree that occurs whenever we attach some event listeners to the DOM node.

How does Event Propagation work? (Bubbling and Capturing)

Let's consider Grand parent , parent and child are three HTML divs. I'll attach three listeners to the three divs.

const grand_parent = document.getElementById("grand_parent"),
parent = document.getElementById("parent"),
child = document.getElementById("child");

grand_parent.addEventListener("click", triggerClick("grand parent clicked"));
parent.addEventListener("click", triggerClick("parent clicked"));
child.addEventListener("click", triggerClick("child clicked"));

function triggerClick(data) {
    console.log(data)
}

On clicking the child, we will see the following things in the console:

On clicking the parent, we will see the parent and grandparent logged-in console. Finally, on clicking grandparent, we will just see the grandparent logged.

This phenomenon is known as the Bubbling phase of event propagation. The event bubbles up from the target element through its ancestors in the DOM tree. This is the more commonly used phase for handling events and the default behavior of any event handlers in JavaScript.

Another phase is known as Capturing phase which does exactly the opposite of the Bubbling. During the capturing phase, the event travels from the root of the DOM tree down to the target element. This phase allows you to intercept and handle the event before it reaches its target. Capturing is less commonly used but can be helpful in certain scenarios.

The code below sets the event handlers to work in Capturing phase.

grand_parent.addEventListener("click", triggerClick("grand parent clicked"), true);
parent.addEventListener("click", triggerClick("parent clicked"), true);
child.addEventListener("click", triggerClick("child clicked"), true);

Passing true property will tell the event listener to work in Capturing mode. Now, whenever we click the child. We will see the exact opposite of the previous output.

Here, when the child is clicked, first of all, the topmost parent's event handler is executed and it travels down to the target through all its ancestors. This is known as Capturing phase of event propagation.

Stopping Event Propagation:

To fix the issue with my data table row click modal and action button click, I had to stop the propagation of the events in my action buttons. JavaScript has a method on the event object that lets us stop the event propagation on that target element.

Let's see how is it done.

grand_parent.addEventListener("click", triggerClick("grand parent clicked"));
parent.addEventListener("click", triggerClick("parent clicked"));

child.addEventListener("click", (event) => {
    triggerClick("child clicked")
    // stop propagation of events (bubbling / capturing)
    event.stopPropagation();
});

stopPropagation() method will tell the handler to stop executing the click handlers on the ancestors of the target element, hence, other listeners are not executed. Here, if you click the child, you will only see child clicked in the console screen.

I set the event.stopPropagation(); on my Buttons under the action columns and now the modal component was not triggered whenever I clicked the action buttons. This was the fix for my issue with the data table row click.

Conclusion

Understanding event propagation in JavaScript is crucial for effective event handling in web development. Whether you're working with the capturing or bubbling phase, being aware of how events propagate through the DOM tree allows you to create more interactive and responsive web applications. Remember to choose the appropriate phase based on your requirements and use the stopPropagation method wisely to control event flow.

In the ever-evolving landscape of the digital world, communication between different applications and services is paramount. Webhooks, often hailed as the unsung heroes of modern development, play a crucial role in facilitating this seamless interaction. In this blog, we will dive into the world of webhooks, exploring what they are, how they work, and the myriad ways in which they contribute to the efficiency and responsiveness of web applications.

Understanding Webhooks

At its core, a webhook is a mechanism that enables one system to notify another about specific events in real time. Unlike traditional APIs, which rely on polling to retrieve information periodically, webhooks initiate the communication process by sending data automatically when a predefined event occurs. This event-driven architecture minimizes latency, enhances efficiency, and fosters more dynamic and responsive applications.

How Webhooks Work

Webhooks operate on a simple principle: event-driven communication. Here's a step-by-step breakdown of how webhooks typically work:

1. Event Occurs:

   • An event, such as a user action or a system event, triggers the need for information to be communicated to another system.

2. HTTP POST Request:

   • The system generating the event sends an HTTP POST request to a predefined URL, which acts as an endpoint for the webhook.

3. Payload Delivery:

   • The HTTP POST request contains a payload – data related to the event. This payload is received by the webhook endpoint, which processes the information.

4. Processing and Response:

   • The receiving system processes the payload and may perform additional actions based on the received data. A response may be sent back to acknowledge the successful reception of the payload.

Use Cases and Benefits

1. Real-time Updates:

   • Webhooks are instrumental in scenarios where real-time updates are crucial, such as chat applications, stock market feeds, or collaborative tools. Users receive immediate notifications, ensuring they stay informed and engaged.

2. Automation and Integration:

   • Webhooks streamline automation by connecting different services and triggering actions based on events. This facilitates seamless integration between applications, reducing manual intervention and enhancing overall workflow efficiency.

3. Enhanced User Experience:

   • Applications utilizing webhooks provide a more responsive and interactive user experience. Users can receive timely updates, notifications, and personalized content, contributing to a more engaging digital environment.

4. Efficient Data Synchronization:

   • In scenarios involving multiple systems or databases, webhooks help maintain data consistency by synchronizing information in real time. This is particularly valuable for e-commerce platforms, content management systems, and collaborative tools.

Conclusion

Webhooks represent a pivotal advancement in the realm of web development, offering a robust solution for real-time communication and event-driven workflows. As we continue to witness the rapid evolution of digital ecosystems, the adoption of webhooks is poised to grow, enabling developers to create more responsive, interconnected, and user-friendly applications. Embrace the power of webhooks, and unlock a new dimension of efficiency and connectivity in the digital era.