Managing virtual network interfaces for OpenShift Virtualization

Key features

Static VNIs per worker node

When you create a bare metal-based Red Hat OpenShift on IBM Cloud cluster or a new worker pool in IBM Cloud VPC, two VNIs are automatically created and attached statically to every bare metal worker node. One VNI handles regular worker traffic (pod network, overlay UDNs, and master communication). The second VNI acts as a carrier for dynamic VNI attachments that you manage.

Dynamic VNI attachments

You can create and manage VNIs on-demand after cluster creation. Dynamic VNIs can be attached to specific workers or configured to float between workers in the same zone, following VM workloads during live migration.

Live migration support

With VNIs attached to bare metal worker nodes, live migrations of OpenShift Virtualization-based VMs can preserve network connections. The VNI implicitly floats and follows the workload between bare metal worker instances within the same zone.

For important limitations and considerations about VNIs, see Limitations and considerations.

Cross-account attachment

In Red Hat OpenShift on IBM Cloud, worker nodes are not provisioned in your account, which means the lifecycle management of VNIs is slightly different from standalone VPC bare metal instances. The Red Hat OpenShift on IBM Cloud cluster administrator has different visibility to the attachments of the VNIs because they are attached to workloads that are not visible within the account. The differences in VNI management are covered in this documentation.

For more information about VNIs on standalone VPC bare metal instances, see About virtual network interfaces.

Creating a Primary UDN

To use a Primary User Defined Network for pods or VM workload, first create the UDN itself.

Example for a Cluster User Defined Network that can be used across namespaces:

apiVersion: k8s.ovn.org/v1
kind: ClusterUserDefinedNetwork
metadata:
  name: primary
spec:
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: green
  network:
    layer2:
      ipam:
        lifecycle: Persistent
      role: Primary
      subnets:
        - 10.0.0.0/24
    topology: Layer2

Then, create the namespace(s) that will use it. The namespace must contain a specific label (called k8s.ovn.org/primary-user-defined-network) on creation for Primary UDN usage.

Example:

kind: Namespace
apiVersion: v1
metadata:
  name: green
  labels:
    k8s.ovn.org/primary-user-defined-network: ''

Once the (C)UDN and the namespaces are in place, any workload created in this namespace will access the UDN with the default route.

Creating a Secondary UDN

Secondary UDN configuration is similar to Primary UDN, but with the role set to Secondary. For detailed information, see the Red Hat OpenShift multiple networks documentation.

Installing the NMState operator

OpenShift Virtualization Service clusters have the NMState operator pre-installed and managed by the openshift-virtualization add-on. Skip this step if you're using Virtualization Service.

1. Deploy the NMState Operator from the OperatorHub in the Red Hat OpenShift on IBM Cloud console or by using the CLI.

2. Create an NMState instance with the default configuration.

   apiVersion: nmstate.io/v1
   kind: NMState
   metadata:
     name: nmstate
   spec:
     probeConfiguration:
       dns:
         host: root-servers.net
   

Creating the OVS bridge

OpenShift Virtualization Service clusters have the required NNCP resources pre-configured. You only need to create these resources manually for standard OpenShift clusters with manual OpenShift Virtualization installation.

1. Create a dedicated OVS bridge with eth1 attached to it by deploying the following NMState custom resource.

   apiVersion: nmstate.io/v1
   kind: NodeNetworkConfigurationPolicy
   metadata:
     name: "br-eth1"
   spec:
     desiredState:
       interfaces:
       - name: "br-eth1"
         description: A dedicated OVS bridge with a NIC as a port
         type: ovs-bridge
         state: up
         bridge:
           allow-extra-patch-ports: true
           options:
             stp: false
           port:
           - name: "eth1"
   

2. Check the custom resource status and wait until all applicable cluster nodes reconcile the bridge creation request.

3. Patch the new OVS bridge to the default one by creating the following NMState custom resource. Replace vpc-vlans with your desired network name.

   apiVersion: nmstate.io/v1
   kind: NodeNetworkConfigurationPolicy
   metadata:
     name: "vpc-vlans"
   spec:
     desiredState:
       ovn:
         bridge-mappings:
         - localnet: "vpc-vlans"
           bridge: "br-eth1"
           state: present
   

4. Check the custom resource status and wait until all applicable cluster nodes reconcile the bridge mapping request.

Creating a localnet user defined network

Create a UDN or ClusterUserDefinedNetwork (CUDN) that makes available the traffic from VNIs by using the selected VLAN.

apiVersion: k8s.ovn.org/v1
kind: ClusterUserDefinedNetwork
metadata:
  name: "vlan250"
spec:
  namespaceSelector:
    matchExpressions:
    - key: kubernetes.io/metadata.name
      operator: In
      values:
      - "default"
  network:
    topology: Localnet
    localnet:
      role: "Secondary"
      physicalNetworkName: "vpc-vlans"
      ipam:
        mode: Disabled
      vlan:
        mode: Access
        access:
          id: 250

Replace the following values:

• vlan250: The name of your CUDN
• default: The namespace where you want to use the CUDN
• vpc-vlans: The physical network name you defined in the bridge mapping
• 250: Your desired VLAN ID (range: 1-500)

This concludes the network plumbing of a specific VLAN. You can define multiple CUDNs by using different VLAN IDs while reusing the bridge mappings.

After completing the localnet UDN setup, you can attach VNIs to the cluster by using the IBM Cloud console or the IBM Cloud CLI ks vni command. See the examples in the following sections.

Before you begin

1. Create VNIs in your VPC with the appropriate subnet and IP address configuration.

2. Make sure that you have the required permissions to manage both the cluster and VNIs.

Example

ibmcloud ks vni attach baremetal --vni 0716-aac49630-f3b4-4ef6-9a3a-5145481697ba --worker kube-c9pqfcmw0tq431jdnrrg-mycluster-default-00000123 --vlan 251

Example output

OK
Successfully attached VNI 0716-aac49630-f3b4-4ef6-9a3a-5145481697ba to worker node kube-c9pqfcmw0tq431jdnrrg-mycluster-default-00000123.
Worker Node ID                                         VNI ID                                      VLAN ID
kube-c9pqfcmw0tq431jdnrrg-mycluster-default-00000123   0716-aac49630-f3b4-4ef6-9a3a-5145481697ba   251

Example output

ibmcloud ks vni ls --cluster-id c9pqfcmw0tq431jdnrrg

OK
VNI ID                                      Worker Node                                            IP Address   MAC Address         VLAN   Floating   Auto-delete
0716-d97d3626-acb2-476b-8b12-bcafa1dec5bb   kube-c9pqfcmw0tq431jdnrrg-mycluster-default-00000456   10.240.1.5   02:00:02:00:73:A5   250    -          false
0716-aac49630-f3b4-4ef6-9a3a-5145481697ba   kube-c9pqfcmw0tq431jdnrrg-mycluster-default-00000123   10.240.1.4   02:00:01:00:73:A5   251    -          false

Example

ibmcloud ks vni detach --vni 0716-aac49630-f3b4-4ef6-9a3a-5145481697ba --worker kube-c9pqfcmw0tq431jdnrrg-mycluster-default-00000123

Detach VNI 0716-aac49630-f3b4-4ef6-9a3a-5145481697ba from worker node kube-c9pqfcmw0tq431jdnrrg-mycluster-default-00000123? This action cannot be undone. [y/N]> y
OK
Successfully detached VNI 0716-aac49630-f3b4-4ef6-9a3a-5145481697ba from worker node kube-c9pqfcmw0tq431jdnrrg-mycluster-default-00000123.

Why can't I attach regular pods to localnet UDNs?

Localnet UDNs have IP Address Management (IPAM) disabled on the OVN side, which means static IP address assignment to pods does not happen from OVN. This configuration is designed for VM workloads that use DHCP or set up static IPs inside the guest operating system. These options are not available for pod workloads.

Why is live migration pending across worker pools?

Different worker pools might have different worker flavors with CPUs from different generations and capabilities. If the full CPU feature set is transparently visible to the VM workload, you cannot live migrate the VM to another worker that does not support all the CPU features. You can limit the guest operating system visible CPU features at the beginning to gain more compatibility across worker flavors.

Why are dynamic attached VNIs not working?

Check the following items:

• Verify that the VLAN ID in the CUDN matches the VLAN ID in the VNI attachment. VPC DHCP might still work without further traffic if the VLAN is mismatched.
• If floating is not enabled, make sure the workload is scheduled on the worker where the VNI is attached.
• Verify that the dedicated OVS bridge and the mapping are present on the workers. Check the NodeNetworkConfigurationPolicy resource status.