HuggingFace Inference Endpoints

Hugging Face Inference Endpoints are an easy way to setup instances of open-source language models on any cloud. Sign up for an account and add billing here, access the Inference Endpoints here, click on “New endpoint”, and fill out the form (e.g. select a model like WizardCoder-Python-34B-V1.0), and then deploy your model by clicking “Create Endpoint”. Change ~/.continue/config.json to look like this:

YAML

config.yaml

models:
  - name: Hugging Face Inference API
    provider: huggingface-inference-api
    model: MODEL_NAME
    apiKey: <YOUR_HF_TOKEN>
    apiBase: <YOUR_HF_INFERENCE_API_ENDPOINT_URL>

JSON

config.json

{
  "models": [
    {
      "title": "Hugging Face Inference API",
      "provider": "huggingface-inference-api",
      "model": "MODEL_NAME",
      "apiKey": "<YOUR_HF_TOKEN>",
      "apiBase": "<YOUR_HF_INFERENCE_API_ENDPOINT_URL>"
    }
  ]
}

View the source