Tabby Community - Does Tabby support GPU offloading?

general

Does Tabby support GPU offloading?

I want to achieve decent performance/low delay while using DeepSeekCoder 6.7B with 8GB VRAM. The model mentions that using GPU offloading will lower RAM consumption. However, I'm using an AMD GPU. Does Tabby support GPU offloading?

Ri

RiQuY

Asked on Apr 11, 2024

GPU offloading in Tabby is currently only possible for CUDA-based backends.
Tabby does not support GPU offloading for AMD GPUs at the moment.

Apr 11, 2024Edited by