Ethan Wilkes
Asked on Nov 29, 2023
You need to make sure you have built tabby with the CUDA feature enabled and run the serve command with the --device cuda
flag. Here are the commands you can use:
cargo run --features cuda serve --model TabbyML/StarCoder-1B --device cuda