Objectives

In this tutorial, you complete the following tasks:

• Authenticate to the Red Hat AI Inference API.
• Embed a knowledge base using the granite-embedding-278m-multilingual model.
• Store embeddings in an in-memory vector index and search it with cosine similarity.
• Query the gpt-oss-120b chat model both without and with retrieved context to see the improvement in accuracy firsthand.

The application you build has the following architecture:

1. Each fact in your knowledge base is converted to an embedding vector and added to a MemoryStore.
2. A user question is embedded using the same model.
3. Cosine similarity ranks the stored vectors against the question vector, and the top 3 facts are retrieved.
4. Those facts are injected into the system prompt sent to the chat model.
5. The model returns a grounded answer.

Before you begin

Make sure you have the following:

• A Pay-As-You-Go or Subscription IBM Cloud account. Trial accounts are not supported. For more information or to upgrade your account, see Account types.

• A Red Hat AI Inference project.

• The Writer role or greater on the Red Hat AI Inference service. For more information, see Managing IAM access.

• Python 3.9 or later installed on your workstation.

• The openai Python package. Install it by running the following command:

  pip install openai
  

Set up API authentication

Before you can call the Red Hat AI Inference API, you need to authenticate your requests using a service ID and API key. If you already completed the getting started tutorial, you can reuse the service ID and API key you created there and skip to Get your project ID and endpoint.

Get your project ID and API endpoint

Your project ID is required for all API requests.

1. Go to Red Hat AI Inference projects.

2. Open your project.

3. Copy your project ID and save it for the next steps.

The base URL for all API requests in this tutorial takes the following form:

https://us-east.rhai.ibm.com/v1/projects/<project_id>

Replace <project_id> with the project ID you copied.

Set environment variables

Set your endpoint and store your API key in a variable:

export ENDPOINT="https://us-east.rhai.ibm.com/v1/projects/<project_id>"
export API_KEY="<your-api-key>"

Replace <project_id> with your project ID and <your-api-key> with the API key you created earlier.

Never hardcode credentials in source files. Use environment variables or a secrets manager such as IBM Cloud Secrets Manager.

Generate a bearer token

The API requires an IAM bearer token. Exchange your API key for an IAM bearer token by running the following command:

export AUTH=$(curl -s -X POST "https://iam.cloud.ibm.com/identity/token" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "grant_type=urn:ibm:params:oauth:grant-type:apikey" \
  --data-urlencode "apikey=$API_KEY" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])")

Verify the token was issued successfully. A valid token starts with eyJ:

echo $AUTH | cut -c1-4

IAM tokens expire after 1 hour. Re-run the token exchange command if you receive a 401 error when running the script.

Build the vector store

The vector store is an in-memory index that holds text alongside its embedding vector. The following Python implementation uses cosine similarity to find the entries most semantically similar to a query vector.

Create a file named store.py and add the following code:

import math


def cosine_similarity(a: list[float], b: list[float]) -> float:
    """Return the cosine similarity between two vectors."""
    dot = sum(x * y for x, y in zip(a, b))
    mag_a = math.sqrt(sum(x * x for x in a))
    mag_b = math.sqrt(sum(x * x for x in b))
    if mag_a == 0 or mag_b == 0:
        return 0.0
    return dot / (mag_a * mag_b)


class MemoryStore:
    """In-memory vector store with cosine similarity search."""

    def __init__(self) -> None:
        self._entries: list[dict] = []

    def add(self, text: str, vector: list[float]) -> None:
        """Store a text/vector pair."""
        self._entries.append({"text": text, "vector": vector})

    def search(self, query: list[float], top_k: int) -> list[dict]:
        """Return the top_k entries most similar to the query vector."""
        scored = [
            {"text": e["text"], "score": cosine_similarity(query, e["vector"])}
            for e in self._entries
        ]
        scored.sort(key=lambda x: x["score"], reverse=True)
        return scored[:top_k]

Cosine similarity measures the angle between two vectors, not their magnitude. This makes it well-suited for comparing text embeddings because it captures semantic direction regardless of text length.

Create the API client

The API client wraps two Red Hat AI Inference endpoints: the embeddings endpoint and the chat completions endpoint. This implementation uses the openai package, which provides a Python client for any OpenAI-compatible API.

Create a file named client.py and add the following code:

import os

from openai import OpenAI


def _client() -> OpenAI:
    """Return an OpenAI client pointed at the InstructLab endpoint."""
    return OpenAI(
        api_key=os.environ["AUTH"],
        base_url=f"{os.environ['ENDPOINT']}/inference/v1",
    )


def embed_text(text: str) -> list[float]:
    """Call the embeddings endpoint and return the vector for the given text."""
    response = _client().embeddings.create(
        model="granite-embedding-278m-multilingual",
        input=text,
        encoding_format="float",
    )
    return response.data[0].embedding


def chat_complete(system_prompt: str, user_message: str, prefix: str = "") -> None:
    """Call the chat completions endpoint with streaming and print tokens as they arrive."""
    messages = []
    if system_prompt:
        messages.append({"role": "system", "content": system_prompt})
    messages.append({"role": "user", "content": user_message})

    stream = _client().chat.completions.create(
        model="gpt-oss-120b",
        messages=messages,
        stream=True,
    )

    print(prefix, end="", flush=True)
    for chunk in stream:
        content = chunk.choices[0].delta.content if chunk.choices else None
        if content:
            print(content.replace("\n", f"\n{prefix}"), end="", flush=True)
    print()

The _client() helper constructs an OpenAI instance pointed at the Red Hat AI Inference API using your ENDPOINT and AUTH environment variables. The embed_text function calls the embeddings endpoint with encoding_format="float" to ensure the response is returned as a plain float list. The chat_complete function calls the chat completions endpoint with streaming enabled, so tokens are printed to the terminal as they arrive.

Seed the memory store and run the demo

With the vector store and API client in place, you can now write the main script that ties everything together: it seeds the store, answers each question without memory, then answers with RAG and prints both responses side-by-side for comparison.

Create a file named main.py and add the following code:

import os
import sys

from client import chat_complete, embed_text
from store import MemoryStore

# knowledge is the set of facts to embed into the vector store.
# This example uses IBM Cloud support FAQs as the knowledge base.
KNOWLEDGE = [
    "To reset your IBM Cloud account password, go to the IBM Cloud login page and click 'Forgot password' to receive a reset email.",
    "IBM Cloud Pay-As-You-Go accounts are charged monthly based on resource usage, with no upfront commitment required.",
    "IBM Cloud resource groups are used to organize account resources and manage access with IAM policies.",
    "You can monitor IBM Cloud service health and view planned maintenance on the IBM Cloud Status page at cloud.ibm.com/status.",
    "IBM Cloud Lite accounts are free and include a selection of always-free services, but do not support all paid features.",
    "To create an IBM Cloud API key, go to Manage > Access (IAM) > API keys in the IBM Cloud console.",
    "IBM Cloud support plans range from Basic (included with all accounts) to Advanced and Premium tiers with faster response times.",
    "IBM Cloud Object Storage stores data as objects across multiple regions for high availability and durability.",
]

QUESTIONS = [
    "How do I reset my IBM Cloud password?",
    "What is the difference between a Lite account and a Pay-As-You-Go account?",
]

TOP_K = 3


def build_system_prompt(entries: list[dict]) -> str:
    facts = "\n".join(f"{i + 1}. {e['text']}" for i, e in enumerate(entries))
    return (
        "You are a helpful assistant with access to the following verified facts.\n"
        "Use these facts to answer the user's question accurately and concisely.\n\n"
        f"Relevant facts:\n{facts}\n"
    )


def main() -> None:
    if not os.environ.get("ENDPOINT") or not os.environ.get("AUTH"):
        print("ERROR: ENDPOINT and AUTH environment variables must be set.", file=sys.stderr)
        sys.exit(1)

    # Seed the vector store.
    print("Embedding knowledge base...")
    store = MemoryStore()
    for i, fact in enumerate(KNOWLEDGE):
        print(f"  [{i + 1}/{len(KNOWLEDGE)}] {fact[:60]}")
        store.add(fact, embed_text(fact))
    print(f"\nMemory store ready with {len(KNOWLEDGE)} entries.\n")

    # Run each question without memory, then with RAG.
    for question in QUESTIONS:
        print(f"Question: {question}\n")

        print("  WITHOUT memory:")
        chat_complete("", question, prefix="    ")

        print("\n  WITH memory (RAG):")
        query_vec = embed_text(question)
        relevant = store.search(query_vec, TOP_K)

        print("  Retrieved facts:")
        for i, entry in enumerate(relevant):
            print(f"    {i + 1}. {entry['text']}")

        print()
        chat_complete(build_system_prompt(relevant), question, prefix="    ")
        print("\n" + "─" * 70)


if __name__ == "__main__":
    main()

Run the Python script:

python main.py

The output shows each question answered twice. The first response comes from the model's general training knowledge alone. The second response is grounded in the facts retrieved from your vector store:

Question: How do I reset my IBM Cloud password?

  WITHOUT memory:
    To reset your password, visit the account login page and look for a
    'Forgot password' or 'Reset password' option ...

  WITH memory (RAG):
  Retrieved facts:
    1. To reset your IBM Cloud account password, go to the IBM Cloud login
       page and click 'Forgot password' to receive a reset email.
    2. To create an IBM Cloud API key, go to Manage > Access (IAM) > API keys
       in the IBM Cloud console.
    3. IBM Cloud support plans range from Basic to Advanced and Premium tiers.

    To reset your IBM Cloud account password, go to the IBM Cloud login page
    and click 'Forgot password'. You will receive an email with instructions
    to set a new password.

The RAG response is more specific and directly cites the retrieved facts because those facts were injected into the system prompt before the chat model was called.

Next steps

Now that you've built a complete RAG pipeline, here are some ways to extend it:

• Bring your own knowledge base — Replace the KNOWLEDGE list in main.py with any list of strings: product documentation, support tickets, internal wikis, or domain-specific data.
• Persist the vector store — Serialize the store's entries to disk (for example, as JSON) so you don't re-embed on every run.
• Tune retrieval — Adjust TOP_K in main.py to retrieve more or fewer facts per query and observe the effect on response quality.