Behind My AI Avatar: The Architecture of an End-to-End AI Chatbot
But getting a language model to produce an answer on some machine is not the same as turning it into a reliable service. Between the question and the answer sits an interface, input validation, a network call, a model, handling of concurrent visitors, a security layer, logs, and error-recovery mechanisms.
This article is the technical making-of of that demo: what actually runs behind the avatar today, what would still be needed to call it a complete production system, and what remains a possible extension rather than an active feature. The question that structures everything else: how do you go from a model that works to an AI system you can actually operate?
What you see
The avatar sits on the demo's screen; the text field (or the mic) is right next to it. You type a question, it goes to the server, an answer comes back and appears — the avatar can also read it aloud, its mouth animated in real time from the amplitude of the sound it produces (an approximation the code itself documents as such, not phoneme-accurate lip-sync).
That interface is the visible part. Everything that makes an answer correct, fast, and safe to obtain happens behind it, out of sight.
From the question to the API
The input field sends the question to the server as a single request. Server-side: the question is validated (200 characters max, never silently truncated), an anonymous visitor is identified by a signed cookie — no account, no password — and a rate limit applies, a cooldown between questions plus a sliding-window quota. Any network error or timeout returns a precise error code rather than a silent failure.
Going further: Building a production API around an LLM →
The model and its provider
The demo doesn't run any model itself. The server calls a third-party provider's API that hosts the model, with the model name set by configuration rather than hardcoded — two reasoning-effort levels exist depending on the chosen mode, one single model behind both.
Self-hosting a model would mean choosing a serving engine (vLLM, TGI, llama.cpp, Ollama...) and owning its operation — a different trade-off, not made here yet.
Going further: Choosing the right LLM for your use case → · Ollama, vLLM, TGI, or llama.cpp → · Quantizing an LLM →
Cost and capacity of a hosted model
With no GPU server to size yourself, the constraint just moves elsewhere: every call has a per-token cost, a time limit (120 seconds here), and depends entirely on the provider's own availability. The main lever available today is the reasoning-effort level requested from the model — a quality/latency/cost trade-off, not a hardware setting.
The day self-hosting becomes worth it again, it's the hardware (memory, precision, context window) that will set those same limits differently.
Going further: CPU, GPU, and VRAM: sizing an LLM server →
Latency and streaming
Three things matter for perceived latency: time to first word, generation speed, and total latency. Today, the demo does neither as a continuous stream: the server waits for the model's complete answer before sending it back as one single JSON block. The "typing" effect you see on screen is a browser-side animation, not a real stream of tokens.
That's a known limitation, not a final choice: real streaming is the logical next step.
Going further: TTFT, tokens/second, and streaming: understanding LLM latency →
Multiple visitors at once
If several people question the demo at the same time, each one is still bound by the same rules: a minimum delay between two questions, a sliding-window quota, and optionally a daily cap per mode. There is, however, no dedicated queue or concurrency control for inference itself: the server runs as a single process, a choice made for database write-safety, not to absorb heavy load.
Handling many concurrent requests cleanly remains a project of its own.
Going further: Queues, concurrency, and backpressure for serving an LLM →
Context, memory, and RAG
By default, the model knows neither your history nor your documents. Three mechanisms already exist: recent messages are sent back verbatim on every call, older history is automatically summarized once it crosses a certain size, and durable, opt-in memory can be kept across sessions if the visitor explicitly consents.
A RAG layer exists too, but only over documents a visitor uploads into their own conversation — it is not a persistent company knowledge base. Extending it into one remains a possible extension, not an active feature today.
Going further: Managing context, memory, and RAG without overloading the model →
Security and public exposure
A publicly reachable service is exposed to abuse: malicious requests, prompt-injection attempts via an uploaded document, deliberate flooding. The demo answers with several independent layers — a bot honeypot, a signed session cookie rather than a guessable id, rate limiting, and explicit framing of any document's content before it ever reaches the model. The provider's API key never leaves the server, including for real-time voice, where the browser only ever receives a short-lived token.
Going further: Securing an LLM exposed on the Internet →
Observability and resilience
A simple health check answers continuously to signal the service is running. Every exchange keeps the model used, its latency, tokens consumed, and status, so it can be replayed afterward instead of getting lost in a raw log. A real incident — a secret shown in cleartext in an access log — already led to adding an automatic redaction filter.
What's still missing: metrics exposed for a dashboard, automatic retries after a failure, circuit breakers. A foundation exists; the full tooling is still to be built.
Going further: Logs, metrics, and traces: observing an LLM in production → · Timeouts, retries, and fallbacks →
Testing before calling it production-ready
Before calling an AI system production-ready, you need precise answers: what share of answers is correct on a real test set, how the system behaves under load, what hallucination rate is tolerable, what criteria would block a release. That evaluation work doesn't exist yet for this demo — it's part of what separates a convincing prototype from a system you can actually rely on.
Going further: Evaluating, load-testing, and red-teaming an LLM before production →
Running it over time
The model name is a configuration parameter, not a fixed value: changing it doesn't require rewriting code. Everything else — tracking real costs, deciding whether to migrate part or all of it to a self-hosted model, scaling up — depends on usage observed over time, not on a choice made upfront.
Going further: Costs, versions, and scaling an LLM in production →
Conclusion
Go back to the avatar on the homepage: a question, an answer. What this article wanted to show is that the quality of such a system isn't just about the model — it's about everything around it: the API that exposes it, the security that protects it, the observability that lets you watch it, and the operational choices that keep it running.
Three ways to continue:
Try the avatar → · Explore upcoming articles → · Discuss your project →