AUDIOANALYZER, AI audio analysis with a pluggable evaluation framework

The problem

Plenty of tools transcribe a call. Very few tell you whether the call was any good, and the ones that do decide for you what "good" means.

That last part is the actual problem. A sales team evaluating discovery calls and a teacher evaluating a lesson are both analysing speech, but they are asking completely different questions of it, and a tool with one hardcoded rubric is useless to whichever of them it was not built for.

AudioAnalyzer is an audio intelligence platform where the evaluation methodology is a plugin rather than an assumption. It runs in production at audioanalyzer.saiaryan.in.

Architecture

The pipeline is deliberately boring in its first half and interesting only where it needs to be.

Transcription. AssemblyAI handles speech to text with diarisation. This is a solved problem and not worth solving again.

Analysis. Groq running LLaMA-3 does the reasoning over the transcript. Groq specifically, for latency: the perceived quality of this kind of tool collapses if a user uploads a file and then waits.

Storage. Neon Postgres. Frontend. React with Vite, kept separate from the FastAPI backend. Billing. Razorpay, because the audience is in India and Stripe is not the default there.

Observability. Langfuse, which turned out to matter more than anything else on this list.

The pluggable evaluation framework

The core design decision was to treat the evaluation methodology as configuration rather than code.

A framework in AudioAnalyzer is a named rubric with its own dimensions and scoring logic. Three ship with it:

• Rosenshine's principles of instruction, for teaching and training
• SPIN selling, for discovery and qualification calls
• STAR, for structured behavioural interviews

The same transcript run through Rosenshine and through SPIN produces two genuinely different analyses, because the questions being asked of it are different. Adding a fourth methodology means defining it, not modifying the pipeline. Core logic stays untouched.

Observability, and the bug it caught

Langfuse traces every LLM call in the pipeline: inputs, outputs, latency, cost.

The reason this matters more than it sounds is that an LLM feature does not fail loudly. It degrades. Without traces you find out from a user rather than from a dashboard.

Here is the specific thing it caught.

The analysis was occasionally scoring transcript segments without correctly attributing them to the right speaker. Feedback would come back anchored to the wrong person's words: a critique of something the interviewer said, attributed to the candidate.

This is a nasty class of bug. Nothing errors. The output is well-formed, confidently worded, and plausible. Every score is a real number in a real range. You cannot catch it by looking at the response, because the response looks exactly like a correct one. You can only catch it by reading an individual run against the transcript it came from, which is precisely what per-run tracing gives you.

The fix was in the prompt, not the pipeline. Diarisation was working. The mistake was assuming diarisation output alone gave the model enough context to hold speaker identity across a long transcript while also scoring it. Adding explicit speaker-attribution instructions ahead of the scoring step fixed it.

I gave a lightning talk on Langfuse and LLM observability at GDG Noida, using AudioAnalyzer as the live demo.

Where it breaks

Long-form audio is not fully supported. Anything beyond roughly ten minutes does not get analysed in a single pass.

What happens then is the part worth describing. The transcription is still saved, and the user is prompted to restart the analysis step rather than losing the upload and starting over. That is a deliberate choice about which half of the pipeline to protect: transcription is the expensive, slow, externally-billed step, and analysis is the cheap, fast, retryable one. Failing in a way that preserves the expensive half is the difference between an annoyance and a reason to stop using the product.

It is still a limitation rather than a feature. Chunking with cross-chunk context is the real fix and it is not built yet.

Decisions and trade-offs

Groq over a frontier model. Lower ceiling on reasoning quality, markedly faster. For an interactive tool where a user is waiting on a result, latency won. A slower and slightly better analysis would have been the wrong product.

FastAPI and React as separate services. More deployment surface. In exchange the analysis pipeline is a Python service callable by things other than this frontend, which is what I would want if any of this became an API product.

Razorpay over Stripe. Narrower international support, correct for the actual users.

The decision I regret

Not writing about it.

I built this, shipped it, wired up billing, and then said almost nothing publicly. The first time I talked about it properly was at a live event, and the reaction in the room made it obvious that people cared about the project considerably more than the silence around it suggested. No post has gone out since.

That gap between the response in the room and the silence online is not a marketing oversight, it is a compounding one. Work that nobody knows about generates no inbound, no collaborators, and no second conversation. The engineering decisions on this page were mostly right. This one was not, and it is the one I would go back and change first.

Outcome

Live in production with billing wired up, and 15 users so far. Early, but real usage on something built and shipped solo.

The idea I would carry into the next build is the plugin boundary. Once the rubric became configuration, the product stopped being one opinionated tool and started being something several different users could each point at their own problem.