ABOVE AI
Real-time audio intelligence & AI conversation insights
A distributed .NET + Python backend that turns live conversations into speaker-attributed transcripts, LLM insights and long-lived session memory.
Stream
Transcribe
Diarize
Reason
Persist
Listening…
LLM insights · local inference
Waiting for the transcript to settle…
✓ Session memory persisted → SurrealDB
The system
A distributed, AI-powered platform that captures live conversations, transcribes and separates speakers in real time, and uses a locally-hosted LLM to generate structured insights and long-lived session memories.
Backend & cloud
- ASP.NET Core 10 orchestration: sessions, ingestion, persistence, LiveKit tokens
- Redis Streams consumer groups & job queues for async, decoupled processing
- FastAPI service boundary between .NET and Python ML workloads
- SurrealDB persistence; Docker Compose + Traefik load-balanced STT replicas
AI / ML inside
- Faster-Whisper automatic speech recognition
- NVIDIA NeMo speaker diarization
- PyTorch + Torchaudio audio processing services
- Local Mistral GGUF LLM for summaries, sentiment, categories & action items
Architecture
Hover or tab through the nodes to inspect each service · ● AI/ML ● core ● data ● edge
Client · mic · browser
Participants join a live session; audio is published from the browser.
LiveKit · WebRTC streaming
Low-latency WebRTC audio transport. Access is gated by LiveKit tokens issued by the .NET backend.
Audio middleware · capture · decode · chunk
Captures room audio, decodes frames and cuts them into chunks ready for inference.
ASP.NET Core 10 · sessions · tokens · state
Session orchestration, audio ingestion, transcript persistence, async memory management and LiveKit token generation.
Redis Streams · consumer groups · jobs
Event backbone. Consumer groups and job queues decouple ingestion from ML inference and insight generation.
Faster-Whisper ×N · ASR · Traefik LB
Containerised speech-to-text replicas behind Traefik internal load balancing.
NVIDIA NeMo · speaker diarization
Segments audio by speaker so every transcript line is attributed.
Mistral GGUF · local LLM insights
Locally-served quantised LLM: summaries, discussion analysis, sentiment signals, categories and action items.
SurrealDB · transcripts · memories
Persistent store for transcripts, discussion records, audio metadata and generated insights.
FastAPI · PyTorch · ML service layer
Python services wrapping the models with PyTorch and Torchaudio for audio processing and speaker segmentation.
What I built
- 01Built real-time audio streaming pipelines on LiveKit and WebRTC, with middleware for audio capture, decoding and chunk processing.
- 02Integrated Faster-Whisper ASR and NVIDIA NeMo diarization to produce speaker-attributed transcripts.
- 03Developed Python AI/ML services with PyTorch, Torchaudio and FastAPI for transcription and speaker segmentation.
- 04Integrated local Mistral GGUF inference to generate summaries, discussion analysis, sentiment-related insights, categories and actionable items.
- 05Implemented asynchronous, event-driven processing with Redis Streams, consumer groups and job queues.
- 06Built the ASP.NET Core backend for session orchestration, audio ingestion, transcript persistence, async memory management and LiveKit token generation.
- 07Persisted transcripts, discussion records, audio metadata and insights in SurrealDB.
- 08Containerised services with Docker Compose and added Traefik-based internal load balancing for STT services.
Engineering decisions
GPU-heavy inference must never stall live audio.
→ Redis Streams with consumer groups and job queues decouple ingestion, ML inference and insight generation — each stage scales and fails independently.
Two runtimes, one product.
→ ASP.NET Core owns sessions, tokens and persistence; Python FastAPI services own the models. A clean service boundary lets each ecosystem do what it is best at.
Speech-to-text is the throughput bottleneck.
→ Containerised STT replicas behind Traefik internal load balancing scale transcription horizontally.
Conversation data is sensitive.
→ A quantised Mistral GGUF model served locally keeps LLM reasoning inside the platform's own infrastructure.
- C#
- ASP.NET Core 10
- Python
- FastAPI
- Faster-Whisper
- NVIDIA NeMo
- PyTorch
- Torchaudio
- Mistral GGUF
- LiveKit
- WebRTC
- Redis Streams
- SurrealDB
- Docker
- Traefik
- Serilog
