Tabby Community
tabbyml.slack.com
Feedback
Wyatt Gill
Asked on Nov 08, 2023
The default value of LLAMA_CPP_PARALLELISM environment variable in TabbyML image version 0.5.2 is unset. Setting this environment variable to 2 fixes the out of memory issue caused by continuous batching support.