general

How to use tabby on Jetson device (with CUDA enabled)

Et

Ethan Wilkes

Asked on Nov 29, 2023

You need to make sure you have built tabby with the CUDA feature enabled and run the serve command with the --device cuda flag. Here are the commands you can use:

cargo run --features cuda serve --model TabbyML/StarCoder-1B --device cuda

Dec 06, 2023Edited by