The core engineering challenge: real-time inference on live speech
Post-call analysis is a solved problem. Record a call, transcribe it, run it through an LLM, generate a summary. There is no latency constraint because the rep is not waiting for guidance. Real-time coaching is fundamentally different. You need to transcribe speech as it happens, understand the conversational context, generate a relevant coaching suggestion, and deliver it to the rep — all within a window of roughly two seconds before the moment passes.
That two-second budget shapes every architectural decision. We use streaming speech-to-text with sub-500ms transcription latency, which feeds into a purpose-built inference pipeline optimised for conversational context. The model does not process the entire call history on every turn — it maintains a rolling context window that prioritises recent dialogue, active deal context from the CRM, and the applicable methodology framework. This is a fundamentally different approach from what we describe as a compounding AI model, though the two concepts work hand-in-hand.
The 2-second latency ceiling is the hardest constraint in real-time coaching — it dictates the entire architecture.
Streaming transcription plus rolling context windows allow fast inference without sacrificing coaching quality.
Private model architecture: why we don't use foundation model APIs
Sending sales call audio to a third-party API creates two problems. First, it introduces network latency that makes real-time delivery nearly impossible for on-prem deployments. Second, it means customer conversation data leaves the organisation's infrastructure — a non-starter for enterprise buyers and regulated industries.
We run private models that deploy directly into the customer's environment. These are not fine-tuned versions of a foundation model accessed via API. They are self-contained models trained to excel at a narrow task: understanding sales conversations and generating actionable coaching suggestions. Smaller, specialised models outperform general-purpose LLMs on this specific task while running on modest hardware. This is the same principle that makes on-premises AI practical for enterprise sales — purpose-built beats general-purpose when the problem domain is well-defined.
- Self-contained models that run entirely within the customer's infrastructure
- Specialised for sales conversation understanding — not general-purpose chatbots
- Optimised to run on standard enterprise GPU hardware without data centre scale compute
- No external API calls — zero data leaves the network
The compounding intelligence layer
The most defensible part of the architecture is not the real-time inference engine — it is the learning loop. Every call generates training signal. When a rep follows a coaching suggestion and the deal progresses, that is a positive signal. When suggestions are ignored or deals stall after a particular pattern, the model learns from that too.
Weekly retraining cycles on the customer's own data mean the model improves continuously. After three months, the coaching is meaningfully different from day one — more specific, better timed, and calibrated to the patterns that actually drive outcomes for that particular team. This compounding effect is extremely difficult for competitors to replicate because it requires the combination of on-prem deployment, continuous learning infrastructure, and enough call volume to train on. Our complete guide to real-time coaching covers the user-facing side of how this learning loop manifests for reps and managers.
Deployment: making enterprise-grade AI operationally simple
The biggest risk to on-prem AI is operational complexity. If it requires a dedicated ML engineering team to maintain, most sales organisations will never adopt it. We invested heavily in making deployment and maintenance as close to zero-ops as possible — containerised deployment, automated model updates within the customer's network, self-healing monitoring, and admin dashboards that sales ops teams can manage without engineering support.
The deployment pipeline supports air-gapped environments, which matters for financial services and government-adjacent customers. Model updates are packaged and transferred through approved channels, validated automatically, and rolled out without downtime. The goal is to make the AI coaching system feel like a SaaS product in terms of operational simplicity while delivering the security posture of on-prem infrastructure.
Key Takeaways
- 1.Real-time coaching requires sub-2-second inference latency, which shapes every architectural decision from transcription to model design.
- 2.Private, specialised models outperform general-purpose LLMs on sales coaching while running on modest hardware inside the customer's infrastructure.
- 3.The compounding learning loop — where the model improves weekly from each customer's own data — is the most defensible part of the technical architecture.
Action Checklist
Frequently Asked Questions
What hardware is required for on-prem AI coaching?
A single enterprise-grade GPU (such as an NVIDIA A100 or equivalent) is sufficient for most deployments up to 100 concurrent users. The models are optimised for inference efficiency, not training scale. Most enterprise IT teams already have compatible hardware.
How does the model improve without sending data to the cloud?
The retraining pipeline runs entirely within the customer's infrastructure. New call data is processed locally, the model is retrained on-prem on a weekly cycle, and the updated model is deployed automatically. No data ever leaves the network.
Why not use a foundation model API with a fine-tuned prompt?
Three reasons: latency (API round-trips add 1 to 3 seconds, blowing the real-time budget), privacy (call data leaves your network), and specialisation (purpose-built models outperform prompted general models on narrow tasks like sales coaching).
How does this approach compare to other AI sales coaching tools?
Most competing tools use cloud-based foundation model APIs for post-call analysis. Real-time, on-prem coaching with compounding learning is architecturally distinct. It trades the simplicity of a SaaS integration for dramatically better latency, privacy, and coaching specificity.
Ready to coach your team in real time?
Parallax learns how your best reps win, then coaches the whole team during live calls.
Book a demo