Working with chat completions

Chat completions are the core of inference. They allow you to send messages to a foundation model and receive AI-generated responses. This is how you build conversational experiences, get answers to questions, generate content, or process natural language inputs. You can control the conversation flow by providing system prompts that define the model's behavior and maintain message history for context-aware interactions.

The following APIs are supported for chat completions:

Chat completions /v1/chat/completions: Create - OpenAI documentation; Get - OpenAI documentation; List - OpenAI documentation; Delete - OpenAI documentation

Before you begin

Create a Pay-As-You-Go or Subscription IBM Cloud account. Trial accounts are not supported. For more information or to upgrade your account, see Account types.
Create a Red Hat AI Inference project.
Make sure that you have the Writer role or greater on the Red Hat AI Inference service. For more information, see Managing IAM access.
Authenticate to the API.
Get your API endpoint. All API requests use the following base URL format:
```
https://us-east.rhai.ibm.com/v1/projects/<project_id>/inference
```
Replace <project_id> with your project ID. To find it, go to Red Hat AI Inference projects and open your project.

Before you begin

Create a Pay-As-You-Go or Subscription IBM Cloud account. Trial accounts are not supported. For more information or to upgrade your account, see Account types.
Create a Red Hat AI Inference project.
Make sure that you have the Writer role or greater on the Red Hat AI Inference service. For more information, see Managing IAM access.

Working with chat completions by using the console

The console provides an interactive playground where you can experiment with different models, test prompts, and refine your AI interactions before integrating them into your applications.

The playground does not support embeddings. Use the API to create embeddings.

In the console, open the Red Hat AI Inference service and click the name of your project to open it.
From the project page, click Playground to open the inference playground.
Begin your chat session. You can customize your chat session with the following options:

Model selection: You can choose from a list of foundation models.
System prompt: The system prompt instructs the model on how to conduct the dialog.
Inference settings: Adjust the Randomness, Repetition, and Response limits.
Chat history: You can filter the chat history by Model or Date range.

Generating a chat completion by using the API

The following example shows how to generate a chat completion. For a complete list of the available parameters, see OpenAI Chat Completion.

curl https://us-east.rhai.ibm.com/v1/projects/<project_id>/inference/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer <bearer_token>" -d '{
 "model": "granite-4-0-h-small",
 "messages": [
   {
     "role": "developer",
     "content": "You are a helpful assistant"
   },
   {
     "role": "user",
     "content": "Hello! Tell me about yourself"
   }
 ]
}'

from openai import OpenAI
client = OpenAI(
  api_key="<bearer_token>",
  base_url="https://us-east.rhai.ibm.com/v1/projects/<project_id>/inference",
)

completion = client.chat.completions.create(
  model="granite-4-0-h-small",
  messages=[
    {"role": "developer", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello! Tell me about yourself"}
  ]
)

print(completion.choices[0].message)

Getting a chat completion by ID by using the API

Retrieving a specific chat completion by ID is useful for auditing, debugging, or analyzing past interactions.

The following example shows how to get a chat completion by its ID. For a complete list of the available parameters, see Get Chat Completion.

curl -L 'https://us-east.rhai.ibm.com/v1/projects/<project_id>/inference/chat/completions/<completion_id>' \
-H 'Accept: application/json' -H "Authorization: Bearer <bearer_token>"

from openai import OpenAI
client = OpenAI(
  api_key="<bearer_token>",
  base_url="https://us-east.rhai.ibm.com/v1/projects/<project_id>/inference",
)

completion = client.chat.completions.retrieve(completion_id="<completion_id>")
print(completion)

Listing chat completions by using the API

Listing chat completions provides an overview of all your inference activity, so you can monitor usage patterns, track costs, and analyze how your application is interacting with foundation models. This is particularly valuable for understanding user behavior, identifying popular use cases, and optimizing your AI integration strategy.

The following example shows how to list chat completions. For a complete list of the available parameters, see List Chat Completions.

curl -L 'https://us-east.rhai.ibm.com/v1/projects/<project_id>/inference/chat/completions' \
-H 'Accept: application/json' -H "Authorization: Bearer <bearer_token>"

from openai import OpenAI
client = OpenAI(
  api_key="<bearer_token>",
  base_url="https://us-east.rhai.ibm.com/v1/projects/<project_id>/inference",
)

completions = client.chat.completions.list()
print(completions)

Deleting a chat completion by using the API

Deleting chat completions helps you clean up test data and comply with privacy requirements.

The following example shows how to delete a chat completion. For a complete list of the available parameters, see Delete chat completion.

curl -X DELETE https://us-east.rhai.ibm.com/v1/projects/<project_id>/inference/chat/completions/<completion_id> \
-H "Content-Type: application/json" -H "Authorization: Bearer <bearer_token>"

from openai import OpenAI
client = OpenAI(
  api_key="<bearer_token>",
  base_url="https://us-east.rhai.ibm.com/v1/projects/<project_id>/inference",
)

client.chat.completions.delete("<completion_id>")