Tabby Community - How to optimize CPU usage when launching Tabby server?

Tabby Community

tabbyml.slack.com

general

How to optimize CPU usage when launching Tabby server?

I'm wondering if it's possible to apply a setup to optimize CPU usage when launching the Tabby server. Clement mentioned about exploiting M3 macOS CPU with LM Studio, and shared the command used to start Tabby server along with the output of /v1/health. Meng Zhang suggested trying to use 1.3B deepseekcoder for --model.

clement-igonet

Asked on Feb 27, 2024

To optimize CPU usage when launching the Tabby server, you can try using a specific model for the --model parameter.
In this case, using the 1.3B deepseekcoder model may help improve CPU performance.
Additionally, ensure that the GPU layers are already maximized for efficient processing.

Example command with optimized --model parameter:

$ TABBY_DISABLE_USAGE_COLLECTION=1 tabby serve --device metal --model TabbyML/DeepseekCoder-1.3B --chat-model TabbyML/Mistral-7B --webserver

Feb 28, 2024Edited by