Posts

Showing posts with the label gpu

TGI - Text Generation Inference - Install, Config, Troubleshoot

A practical guide to installing Hugging Face TGI, launching your first LLM endpoint, tuning key flags, and fixing the failures you will meet. TGI - Text Generation Inference - Install, Config, Troubleshoot
Post a Comment
Read more

Ollama in Docker Compose with GPU and Persistent Model Storage

Run Ollama as a reproducible single-node LLM server using Docker Compose. Configure OLLAMA_HOST and OLLAMA_MODELS, keep models on persistent volumes, enable NVIDIA GPUs, and upgrade safely with rollbacks. Ollama in Docker Compose with GPU and Persistent Model Storage
Post a Comment
Read more