LM Studio

LM Studio is an application for Mac, Windows, and Linux that makes it easy to locally run open-source models and comes with a great UI. To get started with LM Studio, download from the website, use the UI to download a model, and then start the local inference server. Continue can then be configured to use the LMStudio LLM class:

YAML

Package or config.yaml

models:
  - name: LM Studio
    provider: lmstudio
    model: llama2-7b

JSON

config.json

{
  "models": [
    {
      "title": "LM Studio",
      "provider": "lmstudio",
      "model": "llama2-7b"
    }
  ]
}

Embeddings model

LMStudio supports embeddings endpoints, and comes with the nomic-ai/nomic-embed-text-v1.5-GGUF model (as of Nov 2024, check your models)

YAML

Package or config.yaml

models:
  - name: Nomic Embed Text
    provider: lmstudio
    model: nomic-ai/nomic-embed-text-v1.5-GGUF
    roles:
      - embed

JSON

config.json

{
  "embeddingsProvider": {
    "provider": "lmstudio",
    "model": "nomic-ai/nomic-embed-text-v1.5-GGUF"
  }
}

Setting up a remote instance

To configure a remote instance of LM Studio, add the "apiBase" property to your model in config.json:

YAML

Package or config.yaml

models:
  - name: Nomic Embed Text
    provider: lmstudio
    model: nomic-ai/nomic-embed-text-v1.5-GGUF
    roles:
      - embed

JSON

config.json

{
  "embeddingsProvider": {
    "provider": "lmstudio",
    "model": "nomic-ai/nomic-embed-text-v1.5-GGUF"
  }
}

This apiBase will now be used instead of the default http://localhost:1234/v1.

View the source