Skip to main content

Amazon Bedrock

Amazon Bedrock is a fully managed service on AWS that provides access to foundation models from various AI companies through a single API.

Chat model

We recommend configuring Claude 3.5 Sonnet as your chat model.

config.yaml
models:
- name: Claude 3.5 Sonnet
provider: bedrock
model: anthropic.claude-3-5-sonnet-20240620-v1:0
region: us-east-1
profile: bedrock
roles:
- chat

If you run into the following error when connecting to the new Claude 3.5 Sonnet 2 models from AWS - 400 Invocation of model ID anthropic.claude-3-5-sonnet-20241022-v2:0 with on-demand throughput isn't supported. Retry your request with the ID or ARN of an inference profile that contains this model.

You can fix this using the following config:

config.yaml
models:
- name: Claude 3.5 Sonnet
provider: bedrock
model: us.anthropic.claude-3-5-sonnet-20241022-v2:0
region: us-east-1
profile: bedrock
roles:
- chat

Autocomplete model

Bedrock currently does not offer any autocomplete models. However, Codestral from Mistral and Point from Poolisde will be supported in the near future.

In the meantime, you can view a list of autocomplete model providers here.

Embeddings model

We recommend configuring amazon.titan-embed-text-v2:0 as your embeddings model.

config.yaml
models:
- name: Embeddings Model
provider: bedrock
model: amazon.titan-embed-text-v2:0
region: us-west-2
roles:
- embed

Reranking model

We recommend configuring cohere.rerank-v3-5:0 as your reranking model, you may also use amazon.rerank-v1:0.

config.yaml
models:
- name: Bedrock Reranker
provider: bedrock
model: cohere.rerank-v3-5:0
region: us-west-2
roles:
- rerank

Authentication

Authentication will be through temporary or long-term credentials in ~/.aws/credentials under a configured profile (e.g. "bedrock").

[bedrock]
aws_access_key_id = abcdefg
aws_secret_access_key = hijklmno
aws_session_token = pqrstuvwxyz # Optional: means short term creds.

Custom Imported Models

To setup Bedrock using custom imported models, add the following to your config file:

config.yaml
models:
- name: AWS Bedrock deepseek-coder-6.7b-instruct
provider: bedrockimport
model: deepseek-coder-6.7b-instruct
modelArn: arn:aws:bedrock:us-west-2:XXXXX:imported-model/XXXXXX
region: us-west-2
profile: bedrock

Authentication will be through temporary or long-term credentials in ~/.aws/credentials under a configured profile (e.g. "bedrock").

[bedrock]
aws_access_key_id = abcdefg
aws_secret_access_key = hijklmno
aws_session_token = pqrstuvwxyz # Optional: means short term creds.