You can now run 70B model on a single 4GB GPU and it even scales up to the colossal Llama 3.1 405B on just 8GB of VRAM. #AirLLM uses "Layer-wise Inference." Instead of loading the whole model, it loads, computes, and flushes one layer at a time - No quantization needed by default - Supports Llama, Qwen, and Mistral - Works on Linux, Windows, and macOS 100% Open Source.