Skip to content

Vendor setup

RAI supports multiple vendors for AI models and tracing tools, both open-source and commercial APIs. To setu[ it is recommended to use the RAI Configurator.

Alternatively vendors can be configured manually in config.toml file.

Vendors Overview

The table summarizes vendor alternative for core AI service and optional RAI modules:

Module Open source Alternative Why to consider alternative? More information
LLM service Ollama OpenAI, Bedrock Overall performance of the LLM models, supported modalities and features LangChain models
Optional: Tracing tool Langfuse LangSmith Better integration with LangChain Comparison
Optional: Text to speech OpenTTS ElevenLabs Arguably, significantly better voice synthesis
  • OpenTTS GitHub
  • RAI voice interface
    • Optional: Speech to text Whisper OpenAI Whisper (hosted) When suitable local GPU is not an option
  • Whisper GitHub
  • RAI voice interface

    • Best-performing AI models

    Our recommendation, if your environment allows it, is to go with OpenAI GPT4o model, ElevenLabs for TTS, locally-hosted Whisper, and Langsmith.

    LLM Model Configuration in RAI

    In RAI you can configure 2 models: simple model and complex model:

    • complex model should be used for sophisticated tasks like multi-step reasoning.
    • simple model is more suitable for simpler tasks for example image description.
    from rai import get_llm_model

complex_llm = get_llm_model(model_type="complex")
simple_llm = get_llm_model(model_type="simple")

    Vendors Installation

    Ollama

    Ollama can be used to host models locally.

    1. Install Ollama see: https://ollama.com/download
    2. Start Ollama server: ollama serve
    3. Choose LLM model and endpoint type. Ollama server deliveres 2 endpoints:
      • Ollama endpoint: RAI Configurator -> Model Selection -> ollama vendor
      • OpenAI endpoint: RAI Configurator -> Model Selection -> openai vendor -> Use OpenAI compatible API Both endpoints should work interchangeably and decision is only dedicated by user's convenience.

    OpenAI

    1. Setup your OpenAI account, generate and set the API key: bash export OPENAI_API_KEY="sk-..."
    2. Use RAI Configurator -> Model Selection -> ollama vendor

    AWS Bedrock

    1. Set AWS Access Keys keys to your AWS account.

      export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
export AWS_SESSION_TOKEN="..."

    2. Use RAI Configurator -> Model Selection -> bedrock vendor

    Complex LLM Model Configuration

    For custom setups please use LangChain API.

    from langchain_openai.chat_models import ChatOpenAI
from langchain_aws.chat_models import ChatBedrock
from langchain_community.chat_models import ChatOllama

llm1 = ChatOpenAI(model="gpt-4o")
llm2 = ChatOllama(model='llava')
llm = ChatBedrock(model="anthropic.claude-3-opus-20240229-v1:0")

    Text To Speech

    For configuration use Text To Speech tab in RAI Configurator.

    Usage examples can be found in Voice Interface Tutorial

    Speech To Text

    For configuration use Speech Recognition tab in RAI Configurator.

    Usage examples can be found in Voice Interface Tutorial