Important Notes

The quality of your assistant’s responses depends heavily on two factors: the models you choose and the prompts you write. A large, powerful model paired with vague or incomplete instructions will produce inconsistent results, while a smaller model with carefully crafted, domain-specific prompts can deliver surprisingly accurate and useful answers. Invest time in tuning your system prompts — they define your assistant’s persona, response style, and the boundaries of its knowledge. Experiment with different prompt structures, and remember that what works for one understanding module may not work for another.

When running everything locally, memory consumption is the single biggest constraint. Each model you load — whether it is an LLM, an embedding model, or a reranker — occupies its own slice of RAM (or VRAM if you have a GPU). Running all three simultaneously can easily exhaust 16 GB or more. Choose the smallest model that meets your quality bar, and consider using the ollama or openai_compatible embedding and reranker providers to offload computation to a shared Ollama server instead of loading separate copies into your application process. On macOS, Apple Silicon’s unified memory architecture gives you more headroom than the numbers suggest, but the principle remains: every gigabyte counts.

The recommendations in this document reflect our experience running Avrana in production and during development across Linux and macOS environments. They are starting points, not hard requirements. Depending on your use case — whether you need a general-purpose assistant or a narrow, document-focused tool — you may find that different model combinations work better. Start with the suggested setup, measure your memory usage and response quality, then adjust accordingly. If you discover a configuration that outperforms our recommendations, we would love to hear about it.

Minimal Configuration

Linux

Hardware

• 24 GB NVIDIA GPU (RTX 3090, RTX 4090, or A5000)
• 64 GB system RAM

A 16 GB NVIDIA GPU (such as an RTX 4060 Ti, RTX 3080, or an older Tesla P100) is the practical entry point for a fully local Avrana deployment. This gives you enough VRAM to run an LLM quantized to 4-bit or 8-bit precision alongside a lightweight embedding model and, if needed, a reranker — though you will want to load the reranker on CPU or use the ollama provider to share memory with the LLM. CPUs with AVX2 support (Intel Haswell or newer, AMD Excavator or newer) matter more than core count for token generation speed; 8 physical cores is plenty. System RAM should be at least 32 GB so that model weights can be moved between disk, RAM, and VRAM without swapping. Fast NVMe storage is strongly recommended for initial model loading.

Note: It is technically possible to run Avrana locally on a GPU with as little as 8 GB of VRAM — for example, an RTX 3070 or a laptop RTX 4060. You would need to use the smallest available LLM quantization (Q2_K or Q3_K_S), drop the reranker entirely, and run the embedding model on CPU. The pipeline will start and respond, but the LLM’s output quality will be severely degraded: responses will be shorter, less coherent, and prone to hallucination. For any practical use, 16 GB is the realistic minimum.

Recommended Models

Model runs through Ollama with two exception: ASR uses a dedicated Docker image that bundles the model, and TTS runs via a separate piper-wyoming container. Keep in mind that model quality varies significantly by language — English queries will produce noticeably better results than Hungarian or German, as all of these models are trained predominantly on English and Chinese data.

Role	Model	Runtime	VRAM (approx.)
LLM	gemma4:latest	Docker (ollama/ollama)	~10 GB
Embedding	bge-large	Docker (ollama/ollama)	~1 GB
Reranker	bge-large	Docker (ollama/ollama)	shared above
TTS	choose one	Docker (rhasspy/wyoming-piper)	— (CPU)
ASR	whisper	Docker (hwdsl2/whisper-live-server)	— (CPU/GPU)

All three core models — LLM, embedding, and reranker — run through a single Ollama Docker container. This keeps provisioning simple: start the container, pull the two models, and Avrana connects to http://avrana_ollama:11434 using the ollama provider for both embedding and reranker. The bge-large model is pulled once and shared between the embedding and reranker providers inside that same Ollama instance, saving roughly 1 GB of VRAM compared to loading a separate cross-encoder reranker.

ASR and TTS each have their own Docker containers. The ASR image bundles the whisper model, so no additional download is needed. Piper TTS runs entirely on CPU, so it does not compete for GPU memory.

Mac OS

Hardware

On macOS everything runs natively — no Docker required for the model runtimes. Apple Silicon’s unified memory architecture means the CPU and GPU share a single pool, so the 24 GB minimum covers the operating system (~5 GB), the LLM (~10 GB), the embedding model (~1 GB), and the whisper.cpp speech recognition model (~3 GB) with a few gigabytes of headroom. Disk speed matters less than on Linux because model weights load once at startup and stay resident, so a fast external SSD on Thunderbolt is perfectly adequate.

Recommended Models

The model lineup is identical to the Linux minimal configuration. The difference is how they run: instead of Docker containers, Ollama runs natively via ollama serve, and ASR uses whisper.cpp (whisper-server) with a GGML model file stored on disk. Piper TTS is the one exception — it still runs in a Docker container, but since it operates entirely on CPU and uses a lightweight ONNX voice model, it adds negligible overhead.

Role	Model	Runtime	Memory (approx.)
LLM	gemma4:latest	Ollama (native)	~10 GB
Embedding	bge-large	Ollama (native)	~1 GB
Reranker	bge-large	Ollama (native)	shared above
TTS	choose one	Docker (rhasspy/wyoming-piper)	— (CPU)
ASR	whisper	whisper.cpp (native)	~3 GB

Recommended Configuration

Linux

Hardware

• 24 GB NVIDIA GPU (RTX 3090, RTX 4090, or A5000)
• 64 GB system RAM

A 24 GB GPU opens the door to larger, more capable models that would not fit in a minimal setup. With this headroom you can run the LLM at higher precision or step up to a bigger model, load a dedicated cross-encoder reranker on the GPU instead of reusing the embedding model, and still have room for the ASR container. 64 GB of system RAM ensures that model weight files, document indexes, and Qdrant vector storage all stay in the page cache — dramatically reducing disk I/O during search and indexing. A fast NVMe drive remains important for initial model loading and for Qdrant’s on-disk segments.

Recommended Models

The recommended lineup upgrades the LLM to a larger model and adds a proper cross-encoder reranker for noticeably better routing and search relevance. Embedding and reranker still run through Ollama for simplicity, but they use separate models this time — bge-large for embeddings, bge-reranker-v2-m3 for cross-encoder scoring.

Role	Model	Runtime	VRAM (approx.)
LLM	gpt-oss:20b	Docker (ollama/ollama)	~13 GB
Embedding	bge-large	Docker (ollama/ollama)	~1 GB
Reranker	BAAI/bge-reranker-v2-m3	Docker (ollama/ollama)	~1.5 GB
TTS	choose one	Docker (rhasspy/wyoming-piper)	— (CPU)
ASR	whisper	Docker (hwdsl2/whisper-live-server)	— (CPU/GPU)

The total VRAM footprint is around 21–22 GB, comfortably within a 24 GB GPU. The step up from gemma4:latest to gpt-oss:20b provides noticeably better reasoning, longer context handling, and improved multilingual quality — still not native-level for Hungarian or German, but a clear improvement over the minimal variant. The dedicated bge-reranker-v2-m3 cross-encoder produces more accurate relevance scores than cosine similarity on embedding vectors, which translates directly to better understanding module selection and more precise document search results.

Mac OS

Hardware

• Apple Silicon (M4 Max / M5 Pro) 48 GB

With 48 GB of unified memory you can comfortably run the full recommended model stack natively. The operating system consumes roughly 5 GB, the 31B LLM takes about 13 GB, the two embedding/reranker models add another ~3 GB combined, and whisper.cpp uses ~3 GB — leaving around 34 GB for Qdrant, Redis, the Python application, and system overhead. Unlike Linux where VRAM and system RAM are separate pools, Apple Silicon shares everything, so you are not constrained by a GPU memory ceiling — the only limit is total installed RAM.

Recommended Models

The models are the same as Linux, but everything runs natively. Ollama loads the three models through ollama serve, whisper.cpp handles ASR, and only Piper TTS stays in a Docker container. The 31B LLM is the primary upgrade over the minimal configuration: it handles complex, multi-turn conversations more reliably and produces noticeably better Hungarian and German output.

Role	Model	Runtime	Memory (approx.)
LLM	gpt-oss:20b	Ollama (native)	~13 GB
Embedding	bge-large	Ollama (native)	~1 GB
Reranker	BAAI/bge-reranker-v2-m3	Ollama (native)	~1.5 GB
TTS	choose one	Docker (rhasspy/wyoming-piper)	— (CPU)
ASR	whisper	whisper.cpp (native)	~3 GB