Adding Knowledge (RAG)

Give you agent access to product documents, KBs, and more.

LLMs have tremendous knowledge about the world, but they don’t have all the up-to-date specifics about your organization, your products, or other relevant information you might want to provide to your AI voice agents.

RAG is a common technique for grounding agents in the relevant information for your use case.

Examples of Knowledge Sources

Let’s consider some of the content that might be useful to serve some popular use cases:

Customer Success & Support

• Product Documentation → user guides, FAQs, troubleshooting steps.
• Onboarding Materials → getting started guides, best practices, transcripts from training videos.

Customer Acquisition

• Product Information → features, pricing tiers, competitive comparisons.
• Sales Scripts → qualification questions, objection handling, industry-specific use cases.

Operations

• Internal Processes → call routing rules, department directories, business hours & holiday schedules.
• Survey Materials → question banks, follow-up questions, rating scales.

Adding RAG to Ultravox

As we saw in the Using Tools guide, tools provide power-ups for your agents. To use RAG with an Ultravox agent, it’s as simple as creating a tool and instructing the agent on how to use the tool.

The Easy Way

Ultravox provides the corpus service for RAG.

1. Create a Corpus

   Use the Create Corpus endpoint. Give your new corpus a name and (optional) description. This returns a corpusId.

2. Create a Source

   Add a website to crawl using Create Corpus Source. Each source is given a unique sourceId. We will crawl the URL(s) and ingest all the content.

3. Query the Corpus

   After everything is loaded, try some queries using the Query Corpus endpoint.

4. Use the queryCorpus Tool

   Give the built-in queryCorpus tool. to your agents and provide the corpusId. For example, if we wanted to create a voice agent to answer questions about Seattle, we could provide the tool like this:

   {
     "systemPrompt": "Use the queryCorpus tool to answer questions about Seattle.",
     "selectedTools": [
       {
         "toolName": "queryCorpus", 
         "parameterOverrides": {
           "corpus_id": "<your_corpus_id_here>",
           "max_results": 5
         }
       }
     ]
   }
   

The Other Way

Let’s assume we have already stored our product documentation in a vector database and can search that content at https://foo.bar/lookupProductInfo.

Here’s how we might create a tool for our Ultravox agent to use:

Example: Adding a RAG tool

{
  "systemPrompt": "You are a helpful assistant. You have a tool called 'lookupProductInfo' that you must use to find answers.",
  "model": "fixie-ai/ultravox",
  "selectedTools": [
    {
      "temporaryTool": {
        "modelToolName": "lookupProductInfo",
        "description": "Searches official product documentation using semantic similarity to find relevant information. Use this tool to look up specific product features, specifications, limitations, pricing, or support information. The tool returns the most relevant text chunks from the documentation.",
        "dynamicParameters": [
          {
            "name": "query",
            "location": "PARAMETER_LOCATION_BODY",
            "schema": {
              "description": "A specific, focused search query to find relevant product information",
              "type": "string"
            },
            "required": true
          }
        ],
        "http": {
          "baseUrlPattern": "https://foo.bar/lookupProductInfo",
          "httpMethod": "POST"
        }
      }
    }
  ]
}

Add Static Files as Corpus Sources

You can use files as sources for any of your corpora. The process requires using the Create Corpus File Upload API to generate an upload URL.

Follow these steps:

1. Step 1: Request Upload URL

   • Use the Create Corpus File Upload API
   • Include the MIME type string in the request body
   • This returns the URL to use for upload and the unique ID for the document
   • URLs expire after 5 minutes. Request a new one if it expires before using it

   The URL that is returned is tied to the provided MIME type. The same MIME type must be used during upload.
2. Step 2: Upload File

   • Use the presignedUrl from Step 1 to upload the document
   • Ensure the MIME type in the upload matches what was specified in Step 1

   For example, if we requested an upload URL for a text file (MIME type text/plain):

   FILE_PATH="/path/to/your/file"
   UPLOAD_URL="https://storage.googleapis.com/fixie-ultravox-prod/..."
   
   curl -X PUT \
     -H "Content-Type: text/plain" \
     --data-binary @"$FILE_PATH" \
     "$UPLOAD_URL"
   
3. Step 3: Create New Source with Uploaded Document

   • Use the Create Corpus Source API
   • Use upload to provide the documentId from Step 1

   You can provide an array of Document IDs to bulk create a source.

Supported File Types

Here’s the information in a markdown table format:

File Extension	Type of File	MIME Type
doc	Microsoft Word Document	application/msword
docx	Microsoft Word Open XML Document	application/vnd.openxmlformats-officedocument.wordprocessingml.document
txt	Plain Text Document	text/plain
md	Markdown Document	text/markdown
ppt	Microsoft PowerPoint Presentation	application/vnd.ms-powerpoint
pptx	Microsoft PowerPoint Open XML Presentation	application/vnd.openxmlformats-officedocument.presentationml.presentation
pdf	Portable Document Format	application/pdf