Deploying LLM to RedHat OpenShift AI cluster

Objectives

You will learn how to provision and configure a Red Hat OpenShift AI cluster (RHOAI) on IBM Cloud with NVIDIA GPUs and host a large language model (LLM).

Before you begin

Before starting this tutorial ensure you have completed these prerequisites.

• IBM Cloud account with IBMid
• Account must have privileges from these roles. Check Roles to provision Red Hat OpenShift cluster

This tutorial provisions a Red Hat OpenShift cluster with AI add-on with following features:

• Cluster spans 2-zones of a region.
• Worker pool: single worker node per zone, i.e., total 2 worker nodes.
• Worker node profile: bx2.8x32 - 8vCPU/32GB mem.
• GPU worker pool: single GPU worker node per zone, i.e., total 2 GPU worker nodes.
• GPU worker node profile: gx3.16x80.l4 - 16vCPU/80GBmem 1 L4 NVIDIA GPU.
• Model deployed: granite-3.3-8b-instruct.

The deployment provides an LLM inferencing endpoint for use with AI/Gen AI applications.

The figure below shows the architecture of the deployment that will be done as part of this tutorial.

Steps summary

Below is a summary of the steps for completing the tutorial

1. Login to IBM Cloud account
2. Decide the geography and region where the resources will be created
3. Create a Resource group where all resources will be created
4. Create Cloud Object Storage instance for cluster images, model files etc.
5. Download the IBM granite LLM from Hugging Face repository
6. Upload the IBM granite LLM files to COS bucket
7. Create a VPC in a region, and single zone.
8. Create a Red Hat OpenShift cluster
9. Add GPU worker pool to the Red Hat OpenShift cluster
10. Enable the Red Hat OpenShift AI add-on
11. Create a Data Science Project for LLM inferencing
12. Create a connection to the COS bucket
13. Create NVIDIA GPU Accelerator
14. Deploy the LLM model
15. Verify the LLM model by inferencing

Login to IBM Cloud account

Log in to IBM Cloud console

Decide the geography and region where the resources will be created

To help in the decision, refer to the following

• IBM Cloud geographies, regions and locations
• GPU profiles

This tutorial uses the geography and region below.

Create a Resource group where all resources will be created

1. In the IBM Cloud console, go to Manage > Account > Account resources > Resource groups.
2. Click Create.
3. Enter a name for your resource group. (see below)
4. Click Create.

You will need this information:

Create Cloud Object Storage (COS) instance for cluster images, model files etc.

1. In the IBM Cloud console, navigate to the catalog, by clicking Catalog in the top navigation bar.
2. In the left menu, Click the Storage category. Click the Object Storage tile.
3. Give the service instance a name and choose a plan. (see below)
4. Click Create and you are redirected to your new instance.

COS stores images of the cluster internal registry. A COS bucket will automatically be created for your cluster's internal registry.

It is also used to store the LLM/model files. You will create a bucket and upload yourself this yourself.

Download the IBM granite LLM from Hugging Face repository

Read about about the granite-3.3-8b-instruct model

From your local computer, download the model files.

cd <your local work directory>
mkdir lab-content
mkdir lab-content/granite-3.3-8b-instruct
mkdir lab-content/granite-3.3-8b-instruct/model-files
chmod -R 755 lab-content

cd <your local work directory>/lab-content/granite-3.3-8b-instruct/model-files
huggingface-cli download ibm-granite/granite-3.3-8b-instruct --local-dir .

ls -ltr
du -h .

Upload the IBM granite LLM files to COS bucket

Using the IBM Cloud console, create a bucket in Cloud Object Storage (COS) instance that was provisioned earlier and upload the model files

Create a bucket.

Upload using Aspera. Use IBM Aspera for large file uploads.

Upload by selecting the model-files directory. (Do not upload the individual files).

Note: The upload path and file structure must be followed because the RHOAI model deployment requires a path inside the bucket. This path cannot point to a root bucket. Selecting the directory will include the model path.

Check upload status until it is complete.

Here's how the bucket looks after upload. See how the model path ins included in the file name.

You can ignore the extra files model-files/ and model-files/.DS_Store.

From the Configuration tab scroll to Endpoints and capture the "Public" and “Direct” endpoint URL. This will be needed in configuration later.

Create a service credential to access the COS instance and bucket contents and capture the "access_key_id" and “secret_access_key". These will be needed in configuration later

Create a VPC in a region with two zones.

Before you create a RHOAI cluster, you must set up a Virtual Private Cloud (VPC). The VPC provides the necessary networking environment for your cluster.

To create a VPC, complete the following steps:

1. Go to the Navigation menu > Infrastructure > Network > VPCs.
2. Click Create and give the VPC instance a name. (see below).
3. Click Create virtual private cloud.

Create a Red Hat OpenShift cluster

Create your VPC Red Hat OpenShift cluster by using the IBM Cloud console.

Navigate to the cluster console and click Create cluster.

Follow the console instructions and use the information provided below to make the cluster configurations.

Red Hat OpenShift cluster and associated worker nodes will incur costs. Check the estimates before provisioning the cluster.

Cluster must also meet minimum requirements for RHOAI (operating system, worker nodes etc.) listed here

A GPU worker pool will be added later.

Use the information below to create the cluster.

Note:

Make sure to check the version comaptibility of Red Hat OpenShift on IBM Cloud versions with the OpenShift AI add-on you will install in the following steps.

It can take up to 1 hour to provision cluster with the worker nodes.

Once deployed and all status is “green”, launch the “OpenShift web console” to confirm successful deployment.

Check the deployed worker pool.

Add GPU worker pool to the Red Hat OpenShift cluster

On the IBM Cloud console for OpenShift cluster, navigate to Worker pools on the left side menu and click on Add+ to add a new worker pool.

Use the information below to create the GPU worker pool.

GPU worker nodes will incur costs. Check the estimates before provisioning the nodes.

Cluster must meet minimum requirements for RHOAI GPU worker nodes listed here.

From the work pools section, click on "Add".

Enter name and select two zones.

Select the gpu node profile.

Click create.

The gpu node provisioning will take several minutes. (typically ~20 minutes). Check the progress as seen on screen below.

NOTE:

• You may choose only one zone (subnet -01) and deploy only one GPU worker node.
• The profile gx3.16x80.l4 is suitable for deploying relative small LLM e.g., granite-3.3-8b-instruct - 8b parameters. The L4 GPU has 48GB memory on the GPU and 80GB physical server memory.
• To deploy larger sized LLM models e.g., with 20b parameters or more, you will need a GPU worker node with server profile that provides GPUs with more memory. (Roughly > ~2GB per 1b param). The memory may be distributed across multiple GPUs/GPU cards on the same server node. You will also need to attach additional secondary storage to the server node.
• Refere to the available GPU profiles.

Enable the Red Hat OpenShift AI add-on

Read about Red Hat OpenShift AI add=on

Launch the IBM Cloud console for the OpenShift cluster.

Navigate to Overview > Add-ons > Red Hat OpenShift AI

Note:

To install the add-on select version a recent version. Make sure to check the version comaptibility of Red Hat OpenShift on IBM Cloud version that was deployed in the earlier step, with the OpenShift AI add-on version you are deploying.

Install the add-on.

Once completed status on the IBM Cloud console for the OpenShift cluster will say “Normal”.

Even after status says “Normal”, pods continue to initialize and deploy on the cluster. To check the state of related pods and installation status of the operators deployed as part of this add-on, launch the OpenShift web console

From the IBM Cloud console for the OpenShift cluster, click on the OpenShift web console button on the top right to launch the OpenShift web console.

To authenticate access, click on "Login with OpenShift".

Check the status of operators and pods.

Check installed operators. They should show "Successful" status.

Many pods might still be in “Pending” or "Init" status. Wait for them to show "Complete" or "Running" status before proceeding.

Create NVIDIA GPU Accelerator

This accelerator is required as a selection when deploying the model.

From the OpenShift web console, launch Red Hat OpenShift AI console from the grid/matrix icon on the top middle. You may need to login with OpenShift.

On the Red Hat OpenShift AI console, navigate to Settings > Accelerator profiles. Create accelerator profile. Use the information below.

Create a Data Science Project for LLM inferencing

On the Red Hat OpenShift AI console, create a new project. Click Create project.

Create a connection to the COS bucket

In the project, create a new connection. Click Connections tab and create connection. Use the information below.

Deploy the LLM model

In the project, click Models tab and Deploy model as single-model serving. Use the information below.

Use this information to configure the deployment

Click "Deploy" and check the "Status". Once deployed successfully, status changes to green checkmark.

Note the external inferencing endpoint for the deployed model. You will need it in the next step.

NOTE:

• When using GPU worker node with multiple GPUs
  • Change to number of accelerators to match the number of GPUs on the node.
  • Add configuration parameter --tensor-parallel-size and set it to match the number of GPUs.
• Depending on the model, GPU memory and performance requirements adjust other configuration parameters. A higher --max_model_len needs more available GPU memory.

Verify the LLM model by inferencing

Append /docs to the external inference endpoint of the deployed model.. For example:

https://model-granite-33-8b-instruct-aal4lab-v1.l4-lab-paas-rhoai-cluster-XXXXXXXXXX-0000.br-sao.containers.appdomain.cloud/docs

Open the inference endpoint in bowser to get the complete list of OpenAPI endpoints for inferencing the deployed model.

Use the /v1/models endpoint to list the deployed model name. For example:

https://model-granite-33-8b-instruct-aal4lab-v1.l4-lab-paas-rhoai-cluster-XXXXXXXXXX-0000.br-sao.containers.appdomain.cloud/v1/models

Capture the model name/id, e.g. “model-granite-33-8b-instruct” This will be needed in steps later.

Use the /v1/chat/completions endpoint to list the deployed model name.

This is a POST call so use cURL. For example:

curl https://model-granite-33-8b-instruct-aal4lab-v1.l4-lab-paas-rhoai-cluster-XXXXXXXXXX-0000.br-sao.containers.appdomain.cloud/v1/chat/completions \
    -H "Content-Type: application/json" \
  	-d '{
      "model": "model-granite-33-8b-instruct",
      "messages": [
        {
          "role": "developer",
          "content": "You are a helpful assistant."
        },
        {
          "role": "user",
          "content": "capital of usa is"
        }
      ]
  }'

Chech the output.

{
  "id":"chatcmpl-ba28…143b","object":"chat.completion","created":1752174978,
  "model":"model-granite-33-8b-instruct",
  "choices":[{
    "index":0,
    "message":{
      "role":"assistant","reasoning_content":null,
      "content":"The capital of the United States is Washington, D.C. While D.C. isn't a state, it's a federal district, and serves as the seat of the U.S. federal government. It's located on the east coast of the United States, along the Potomac River, bordering Maryland and Virginia.",
      "tool_calls":[]
    },
    "logprobs":null,
    "finish_reason":"stop",
    "stop_reason":null
  }],
  "usage":{
    "prompt_tokens":74,
    "total_tokens":152,
    "completion_tokens":78,
    "prompt_tokens_details":null
  },
  "prompt_logprobs":null
}

Above output confirms successful provisioning, configuration and inferencing of LLM deployed on Red Hat OpenShift AI cluster.

The inferencing endpoint can now be used by other applications.

End of tutorial. Thank you.