general

Can I use more than 4 threads for CPU device with LLAMA_CPP_N_THREADS?

I want to know if it's possible to use more than 4 threads for CPU device with the LLAMA_CPP_N_THREADS environment variable.

cl

clement-igonet

Asked on Nov 24, 2023

Yes, it is possible to use more than 4 threads for CPU device with the LLAMA_CPP_N_THREADS environment variable. By setting the LLAMA_CPP_N_THREADS environment variable to the desired number of threads, you can specify the number of threads to be used for CPU device. For example, if you want to use 8 threads, you can set LLAMA_CPP_N_THREADS to 8. Here's an example of how to set the environment variable in C++:

if (const char* n_thread_str = std::getenv("LLAMA_CPP_N_THREADS")) {
  int n_threads = std::stoi(n_thread_str);
  ctx_params.n_threads = n_threads;
  ctx_params.n_threads_batch = n_threads;
}
Nov 25, 2023Edited by