I've started llama.cpp as a web service like explained here: https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md, so I can request with a simple curl command:
--url <http://localhost:8080/completion> \
--header "Content-Type: application/json" \
--data '{"prompt": "Building a website can be done in 10 simple steps:","n_predict": 128}'```
clement-igonet
Asked on Nov 22, 2023
No, it is not possible to plug llama.cpp web service directly to tabby-client or tabby-engine. Tabby heavily customizes the decoding steps and is not suitable to rely on an openai-like API. There is a discussion about this topic on the Tabby GitHub repository: https://github.com/TabbyML/tabby/issues/795