Using virtual private endpoints for VPC to privately connect to Red Hat AI Inference

Send inference requests from your Virtual Private Cloud (VPC) to Red Hat AI Inference over the IBM Cloud private network, so that your request traffic does not cross the public internet.

IBM Cloud® Virtual Private Endpoints (VPE) for VPC enable you to connect to Red Hat AI Inference from your VPC network by using a reserved IP address that is allocated from a subnet in your VPC. The IP address is bound to an endpoint gateway. The endpoint gateway is a virtualized function that scales horizontally, is redundant and highly available, and spans all availability zones of your VPC. Virtual server instances and other workloads in your VPC then call Red Hat AI Inference through a private hostname. For more information about endpoint gateways, see About virtual private endpoint gateways.

Before you begin

Before you set up a virtual private endpoint for Red Hat AI Inference, make sure that you have the following items.

A Pay-As-You-Go or Subscription IBM Cloud account. Trial accounts are not supported. To upgrade your account, see Upgrading your account.
A Red Hat AI Inference project.
The Writer role or greater on the Red Hat AI Inference service.
A VPC in the Washington DC (us-east) region. For more information, see Getting started with VPC.
A plan for your virtual private endpoints, including the subnets and reserved IP addresses that you want to use.
A way to authenticate your requests. Authentication is the same as for the public endpoint.
Ensure that correct access controls are set for your virtual private endpoint.
Understand the limitations of having a virtual private endpoint.
Understand how to view details about a virtual private endpoint.

Private endpoint hostname and CRN

When you use the private endpoint, send your API requests to the private hostname instead of the public hostname. The private hostname is not reachable from the public internet. It resolves to the reserved IP address of your endpoint gateway only from within a VPC that has an endpoint gateway for Red Hat AI Inference.

If you create the endpoint gateway by using the CLI or API, you must specify the Cloud Resource Name (CRN) of the Red Hat AI Inference endpoint. The following table lists the private hostname and CRN for the supported region.

Region, private hostname, and CRN for connecting to Red Hat AI Inference over the private network
Region	Private hostname	CRN
`us-east`	`private.us-east.rhai.ibm.com`	`crn:v1:bluemix:public:instructlab:us-east:::endpoint:private.us-east.rhai.ibm.com`

Configuring an endpoint gateway

Create an endpoint gateway for Red Hat AI Inference in the VPC that runs the workloads that call the service.

You must create the endpoint gateway before you call the private hostname from your VPC. Without an endpoint gateway, requests from your VPC do not use the virtual private endpoint path.

Create an endpoint gateway in your us-east VPC. For the target service, select Red Hat AI Inference. If you create the endpoint gateway by using the CLI or API, specify the CRN from the previous table.
Bind a reserved IP address to the endpoint gateway.
View the details of the endpoint gateway to confirm that it is available. For more information, see Viewing details of an endpoint gateway.

After the endpoint gateway is available, workloads in the VPC can call Red Hat AI Inference through the private hostname.

Verifying the private connection

To confirm that your VPC reaches Red Hat AI Inference over the private network, complete the following steps from a virtual server instance in the VPC.

Resolve the private hostname.
```
dig private.us-east.rhai.ibm.com
```
The hostname must resolve to the reserved IP address of your endpoint gateway. If it resolves to a different address, see Why doesn't the private hostname resolve to my endpoint gateway?.
Send a request to the private endpoint. For example, list the models in your project. Replace <project_id> with your project ID and <bearer_token> with a valid bearer token. For more information, see Authenticating to the API.
```
curl -L 'https://private.us-east.rhai.ibm.com/v1/projects/<project_id>/inference/models' \
-H 'Accept: application/json' -H "Authorization: Bearer <bearer_token>"
```
A successful request returns the list of models that are available in your project.

Using the private endpoint

The private endpoint serves the same API as the public endpoint. The request paths, request bodies, and authentication are the same. Only the hostname is different. To use the private endpoint with your applications, replace the public hostname with the private hostname in your base URL.

https://private.us-east.rhai.ibm.com/v1/projects/<project_id>/inference

The following example shows how to set the private base URL when you use the OpenAI Python client.

from openai import OpenAI
client = OpenAI(
  api_key="<bearer_token>",
  base_url="https://private.us-east.rhai.ibm.com/v1/projects/<project_id>/inference",
)

models = client.models.list()
print(models)

For more examples of working with the API, see Working with chat completions and Creating embeddings.

Troubleshooting the private connection

If your requests to the private endpoint fail, see the following troubleshooting topics.