Llamafile 0.8.1 GPU LLM Offloading Works Now With More AMD GPUs

It was just a few days ago that Llamafile 0.8 released with LLaMA 3 and Grok support along with faster F16 performance. Now this project out of Mozilla for self-contained, easily re-distributable large language model (LLM) deployments is out with a new release.

Most significant with Friday's Llamafile 0.8.1 release is getting GPU support working for more AMD graphics processors / accelerators. Due to some of the AMD offload code within Llamafile only assuming numeric "GFX" graphics IP version identifiers and not alpha-numeric, GPU offload was mistakenly broken for a number of AMD Instinct / Radeon parts. For hardware like the Instinct MI250 with the GFX90A IP, the "A" was not being correctly parsed and not passed to the HIP compiler. In turn this would error out and break Llamafile GPU acceleration on AMD GPUs having non-numeric characters as part of their GFX identifier. That's now fixed up with Llamafile 0.8.1 and thus AMD GPU acceleration working on more hardware for Llamafile-based large language model deployments.

Additionally, Llamafile 0.8.1 now ships pre-built NVIDIA and AMD ROCk modules for both Windows and Linux users for further easing the deployment of Llamafile single-file LLMs that support both CPU and GPU execution.

Llamafile 0.8.1 also adds support for the Phi-3 Mini 4k model, fixed a bug causing GPU model crashes. support for Command-R Plus has proper 64-bit indexing, and other fixes.

Downloads and more details on the new Llamafile 0.8.1 release via Mozilla-Ocho on GitHub.