Reranking model

A "reranking model" is trained to take two pieces of text (often a user question and a document) and return a relevancy score between 0 and 1, estimating how useful the document will be in answering the question. Rerankers are typically much smaller than LLMs, and will be extremely fast and cheap in comparison.

In Continue, reranking is used by @Codebase in order to select the most relevant code snippets after vector search.

Recommended reranking models

If you have the ability to use any model, we recommend rerank-2 by Voyage AI, which is listed below along with the rest of the options for rerankers.

Voyage AI

Voyage AI offers the best reranking model for code with their rerank-2 model. After obtaining an API key from here, you can configure like this:

YAML

config.yaml

models:
  - name: My Voyage Reranker
    provider: voyage
    apiKey: <YOUR_VOYAGE_API_KEY>
    model: rerank-2
    roles:
      - rerank

JSON

config.json

{
  "reranker": {
      "name": "voyage",
      "params": {
          "model": "rerank-2",
          "apiKey": "<YOUR_VOYAGE_API_KEY>"
      }
  }
}

Cohere

See Cohere's documentation for rerankers here.

YAML

config.yaml

models:
  - name: Cohere Reranker
    provider: cohere
    model: rerank-english-v3.0
    apiKey: <YOUR_COHERE_API_KEY>
    roles:
      - rerank

JSON

config.json

{
  "reranker": {
      "name": "cohere",
      "params": {
        "model": "rerank-english-v3.0",
        "apiKey": "<YOUR_COHERE_API_KEY>"
      }
  }
}

LLM

If you only have access to a single LLM, then you can use it as a reranker. This is discouraged unless truly necessary, because it will be much more expensive and still less accurate than any of the above models trained specifically for the task. Note that this will not work if you are using a local model, for example with Ollama, because too many parallel requests need to be made.

YAML

config.yaml

models:
  - name: LLM Reranker
    provider: openai
    model: gpt-4o
    roles:
      - rerank

JSON

config.json

{
  "reranker": {
      "name": "llm",
      "params": {
          "modelTitle": "My Model Title"
      }
  }
}

The "modelTitle" field must match one of the models in your "models" array in config.json.

Text Embeddings Inference

Hugging Face Text Embeddings Inference enables you to host your own reranker endpoint. You can configure your reranker as follows:

YAML

Package or config.yaml

models:
  - name: Huggingface-tei Reranker
    provider: huggingface-tei
    apiBase: http://localhost:8080
    apiKey: <YOUR_TEI_API_KEY>
    roles:
      - rerank

JSON

config.json

{
  "reranker": {
    "name": "huggingface-tei",
    "params": {
      "apiBase": "http://localhost:8080",
      "apiKey": "<YOUR_TEI_API_KEY>"
    }
  }
}

Free Trial (Voyage AI)

Continue offers a free trial of Voyage AI's reranking model.

YAML

config.yaml

models:
  - name: Free Trial Reranker
    provider: free-trial
    roles:
      - rerank

JSON

config.json

{
  "reranker": {
    "name": "free-trial"
  }
}