GGUF Quantization: Quality vs Speed on Consumer GPUs

Compare GGUF, GPTQ, and AWQ quantization formats for LLMs on consumer GPUs. Learn how to balance model quality, speed, and memory usage with Q4_K_M, IQ4_XS, and Q3_K_S variants for optimal inference performance.

GGUF Quantization: Quality vs Speed on Consumer GPUs

Comments

Post a Comment

Popular posts from this blog

Move Ollama Models to different location

After installing ollama better to reconfigure ollama to store them in new place right away. So after we pull a new model, it doesn’t get downloaded to the old location. Ollama is a text-based frontend to LLM AI Models and an API that can host those too. Install Ollama Goto https://ollama.com/download To install Ollama on linux: curl -fsSL https://ollama.com/install.sh | sh Ollama on Windows is on page: https://ollama.com/download/windows Ollama for Mac is there too: https://ollama.com/download/macOllamaSetup.exe Download, List and Remove Ollama models To download some Ollama models: Go to Ollama Library (https://ollama.com/library) and find the model you need, there you can also find model tags and sizes. Then run: ollama pull gemma2:latest # Or get slightly smarter one still nicely fitting into 16GB VRAM: ollama pull gemma2:27b-instruct-q3_K_S # Or: ollama pull llama3.1:latest ollama pull llama3.1:8b-instruct-q8_0 ollama pull mistral-nemo:12b-instruct-2407-q6_K ollama ...
Read more

Gitflow Workflow overview

Another Gitflow Workflow overview: Steps, Alternatives, Pros & Cons https://dasroot.net/posts/2025/06/gitflow-overview-and-alternatives #Gitflow #git #devops Gitflow is a structured Git branching model designed to manage software development workflows with dedicated branches for features, releases, and hotfixes. It emphasizes **stable code in production**, **parallel development**, and **controlled release cycles**. While it offers robust version control and collaboration, its complexity and rigidity make it less suitable for agile or continuous deployment environments. Alternatives like **GitHub Flow** and **Trunk-Based Development** provide simpler, more flexible workflows.
Read more