general
How can I load the 4 bit quantization model instead of the 8 bit quantization model in DeepseekCoder-6.7B?
My GPU is too small to support the 8 bit quantization model in ~/.tabby/DeepseekCoder-6.7B/tabby.json
. Is there a way to load the 4 bit quantization model instead?
Pr
Pranav Kumar
Asked on Dec 27, 2023
You can follow the 'how can I convert my own model for use with Tabby' guide in the Tabby documentation to create your own registry.
This will allow you to load the 4 bit quantization model instead of the default 8 bit quantization model. Here is the link to the guide: how can I convert my own model for use with Tabby.
Dec 31, 2023Edited by