general

How can I load the 4 bit quantization model instead of the 8 bit quantization model in DeepseekCoder-6.7B?

My GPU is too small to support the 8 bit quantization model in ~/.tabby/DeepseekCoder-6.7B/tabby.json. Is there a way to load the 4 bit quantization model instead?

Pr

Pranav Kumar

Asked on Dec 27, 2023

You can follow the 'how can I convert my own model for use with Tabby' guide in the Tabby documentation to create your own registry.

This will allow you to load the 4 bit quantization model instead of the default 8 bit quantization model. Here is the link to the guide: how can I convert my own model for use with Tabby.

Dec 31, 2023Edited by