help-self-hosting

What is the default value of LLAMA_CPP_PARALLELISM environment variable in TabbyML image version 0.5.2?

Wy

Wyatt Gill

Asked on Nov 08, 2023

The default value of LLAMA_CPP_PARALLELISM environment variable in TabbyML image version 0.5.2 is unset. Setting this environment variable to 2 fixes the out of memory issue caused by continuous batching support.

Nov 08, 2023Edited by