How to self-host a model
You can deploy a model in your AWS, GCP, Azure, Lambda, or other clouds using:
- HuggingFace TGI
- vLLM
- SkyPilot
- Anyscale Private Endpoints (OpenAI compatible API)
- Lambda
Self-hosting an open-source model
For many cases, either Continue will have a built-in provider or the API you use will be OpenAI-compatible, in which case you can use the "openai" provider and change the "baseUrl" to point to the server.
However, if neither of these are the case, you will need to wire up a new LLM object.
Authentication
Basic authentication can be done with any provider using the apiKey
field:
{
"models": [
{
"title": "Ollama",
"provider": "ollama",
"model": "llama2-7b",
"apiKey": "xxx"
}
]
}
This translates to the header "Authorization": "Bearer xxx"
.
If you need to send custom headers for authentication, you may use the requestOptions.headers
property like in this example with Ollama:
{
"models": [
{
"title": "Ollama",
"provider": "ollama",
"model": "llama2-7b",
"requestOptions": {
"headers": {
"X-Auth-Token": "xxx"
}
}
}
]
}
Similarly if your model requires a Certificate for authentication, you may use the requestOptions.clientCertificate
property like in the example below:
{
"models": [
{
"title": "Ollama",
"provider": "ollama",
"model": "llama2-7b",
"requestOptions": {
"clientCertificate": {
"cert": "C:\tempollama.pem",
"key": "C:\tempollama.key",
"passphrase": "c0nt!nu3"
}
}
}
]
}