HuggingFace Inference Endpoints
Hugging Face Inference Endpoints are an easy way to setup instances of open-source language models on any cloud. Sign up for an account and add billing here, access the Inference Endpoints here, click on “New endpoint”, and fill out the form (e.g. select a model like WizardCoder-Python-34B-V1.0), and then deploy your model by clicking “Create Endpoint”. Change ~/.continue/config.json
to look like this:
- YAML
- JSON
config.yaml
models:
- name: Hugging Face Inference API
provider: huggingface-inference-api
model: MODEL_NAME
apiKey: <YOUR_HF_TOKEN>
apiBase: <YOUR_HF_INFERENCE_API_ENDPOINT_URL>
config.json
{
"models": [
{
"title": "Hugging Face Inference API",
"provider": "huggingface-inference-api",
"model": "MODEL_NAME",
"apiKey": "<YOUR_HF_TOKEN>",
"apiBase": "<YOUR_HF_INFERENCE_API_ENDPOINT_URL>"
}
]
}