Posts

Showing posts with the label GGUF

LLM Systems: Operational Reference for Self-Hosted Inference

llmsystems.dev is live: curated model and GGUF artifact records, llama.cpp, hardware memory classes, and weights-only VRAM bands with explicit evidence labels (SOURCE, POLICY, OBSERVED). Built for people who self-host — not a blog, not fake fit claims. LLM Systems: Operational Reference for Self-Hosted Inference
Post a Comment
Read more

Llama-Server Router Mode - Dynamic Model Switching Without Restarts

How to configure llama-server router mode for dynamic model loading and switching. Covers models.ini setup, systemd service, API usage, and honest comparison to Ollama and llama-swap. Llama-Server Router Mode - Dynamic Model Switching Without Restarts
Post a Comment
Read more

llama.cpp Quickstart with CLI and Server

Install llama.cpp, run GGUF models with llama-cli, and serve OpenAI-compatible APIs using llama-server. Key flags, examples, and tuning tips with a short commands cheatsheet llama.cpp Quickstart with CLI and Server
Post a Comment
Read more

GGUF Quantization: Quality vs Speed on Consumer GPUs

Compare GGUF, GPTQ, and AWQ quantization formats for LLMs on consumer GPUs. Learn how to balance model quality, speed, and memory usage with Q4_K_M, IQ4_XS, and Q3_K_S variants for optimal inference performance. GGUF Quantization: Quality vs Speed on Consumer GPUs
Post a Comment
Read more