Backing up virtual machines on Red Hat OpenShift Virtualization with Veeam Kasten

Veeam Kasten prerequisite checker

Veeam Kasten provides a prerequisite checker, the Veeam Kasten Primer, that you use to verify that the Kubernetes cluster and StorageClasses meet the installation requirements. If a Container Storage Interface (CSI) provisioner exists, it conducts a basic validation check.

The download location includes a path to the version that you want to install. Version 8.5.1 is used in this example. Change these values to a specific version or use the value latest. The primer tool also expects the helm package to be installed on your system with the Veeam Kasten Helm charts repository. For more information, see the Veeam Kasten documentation.

To start the Pre-Flight checker, run the following command after you create a command line session to the Red Hat OpenShift cluster:

curl https://docs.kasten.io/downloads/8.5.1/tools/k10_primer.sh | bash

For sample output for the checker, see Veeam Kasten Primer for Pre-Flight Checks link. In general, you can expect a combination of Success and OK messages in the output. If errors occur, correct them before you install Veeam Kasten.

Installing Veeam Kasten through OperatorHub

Use the following information to install Veeam Kasten through OperatorHub.

1. From the Red Hat OpenShift Container Platform (OCP) web console, go to Operators > OperatorHub.
2. Search for Kasten in the filter and click the resulting Veeam Kasten entry. Installation packages are available with different licensing options. Click Tile for the appropriate licensing.
3. Click Install.
4. In the Install operator, you can use the default values. You can also enable the Console plug-in, which enables the Veeam Kasten tab of the Red Hat OpenShift web console navigation window.
5. Click Install.

Configuring the environment for Veeam Kasten

Configuring the environment to use Veeam Kasten involves the use of the Red Hat OpenShift command-line interface (CLI). Verify that the command-line session to the Red Hat OpenShift cluster is active. If not, create a new command-line session before you proceed.

If your storage class name differs from the example, then you must adjust the commands.

Set the Red Hat OpenShift container storage class as the default for Kasten deployment. You can change the default storage class after you install Kasten.

1. From the Red Hat OpenShift Container Platform (Red Hat OpenShift Container Platform) web console go to Storage > StorageClasses
2. Search for the ocs-storagecluster-ceph-rbd entry. In the entry row, click Vertical ellipsis > Set as default.

Veeam Kasten uses annotations on StorageClass and VolumeSnapshotClass resources in Kubernetes to configure data protection behaviors and integrate with underlying storage providers. Use the Kubernetes CLI and the Red Hat OpenShift CLI to create the required annotations.

Run the following commands from the command line session to the Red Hat OpenShift cluster.

• oc annotate volumesnapshotclass ocs-storagecluster-rbdplugin-snapclass k10.kasten.io/is-snapshot-class=true
• oc annotate storageclass ocs-storagecluster-ceph-rbd k10.kasten.io/sc-supports-block-mode-exports=true
• oc annotate storageclass ocs-storagecluster-ceph-rbd-virtualization k10.kasten.io/sc-supports-block-mode-exports=true
• oc annotate storageclass ocs-storagecluster-ceph-rbd k10.kasten.io/block-mode-export=true
• oc annotate storageclass ocs-storagecluster-ceph-rbd-virtualization k10.kasten.io/block-mode-export=true

Deploying Veeam Kasten

After you install Veeam Kasten and annotate StorageClasses, complete the configuration by deploying and configuring more Veeam Kasten pods into the environment.

1. From the Red Hat OpenShift Container Platform web console, go to Operators > Installed Operators > Kasten and verify that All projects or kasten-io is selected.
2. Click Veeam Kasten.
3. Click the K10 tab in the Veeam Kasten Operator.
4. Click Create K10.
5. In the Create K10 page, enable the following settings:
   1. Enable basic authentication.
   2. Enable the K10 dashboard to display through the route.
   3. Create secured edge route for the displayed K10.
6. In the Specify StorageClassName to be used for PVCs field, enter ocs-storagecluster-ceph-rbd.
7. Click Create.
8. Go to Workload > Pods. View the Project: kasten-io and wait for the pods to show running.

Accessing the Veeam Kasten web interface

1. Verify that Project: kasten-io is selected.
2. From the web console, go to Networking > Routes.
3. Locate the K10-route entry and click the link in the Location value. The path for Kasten UI is /K10/#/dashboard. For example, https://<openshift-console-fqdn>/k10/#/dashboard.
4. Log in to Kasten by using the Red Hat OpenShift authentication token. You can use the web console or CLI. For more information about authentication, see the Authentication.
5. From the Red Hat OpenShift Container Platform web console, click the dropdown menu in the page header that is next to the IAM username.
6. Click Copy login command > Display Token.
7. Copy the Your API token is value to the Kasten login prompt.

Validating the Red Hat OpenShift Storage Cluster configuration

You also use the same Veeam Kasten prerequisite checker tool to verify that Veeam Kasten is configured correctly. For more information, see Veeam Kasten Primer Block Mount Check.

Run the Veeam Kasten primer tool by using the following command.

% curl https://docs.kasten.io/downloads/8.5.1/tools/k10_primer.sh | bash /dev/stdin blockmount -s ${STORAGE_CLASS_NAME}

The Veeam Kasten primer tool can also be used to verify that the CSI StorageClass has the required configuration to support block VolumeMode exports in Veeam Kasten by running the following command. For more information, see Veeam Kasten Primer for storage integration checks.

% curl https://docs.kasten.io/downloads/8.5.1/tools/k10_primer.sh | bash /dev/stdin csi -s ${STORAGE_CLASS_NAME}

The two checks that are described in the previous section can be run against both storage classes for Kasten use, in this example, ocs-storagecluster-ceph-rbd and ocs-storagecluster-ceph-rbd-virtualization. If errors occur, you need to correct them before Veeam Kasten can protect your data.

As of writing the solution guide, a bug exists in the validation tool where it fails if the StorageClass name is too long. Until it is fixed, you can create a temporary copy of the StorageClass for validation purpose. Take the YAML of the StorageClass and create a new one with a smaller name. Make sure to adjust the name and remove extra fields like uid, resourceVersion, creationTimestamp, and labels in the YAML content.

Creating a S3-compatible location profile

Before you create an IBM Cloud Object Storage bucket location profile, you have the following prerequisites:

• IBM Cloud Object Storage bucket that is already defined with a networking path for access from Veeam Kasten on Red Hat OpenShift
• S3 access key
• S3 secret
• IBM Cloud Object Storage bucket endpoint URL
• Bucket region

Enable the object-locking option on the IBM Cloud Object Storage bucket. This option is required for immutable backups.

1. From the Veeam Kasten web interface, go to Profiles > Location and click Create profile.
2. In the Name and Provider page, enter the Location Profile Name and in the Storage Provider dropdown menu, select S3 Compatible
3. Click Next and in the Configure S3 Generic Profile page, enter the following information:
   • S3 access key
   • S3 secret
   • Endpoint
   • Choose whether to Skip certificate chain and hostname verification
   • Region
   • Bucket name
   • Enable immutable backups, if applicable. Click Validate bucket, if enabled. If immutable backup is enabled, set the Protection period. The minimum protection period is 1 day and the maximum is 90 days.
4. Click Next
5. Review the Summary and click Submit.

Creating an NFS location profile

Creating an NFS location profile requires the following prerequisites:

• An existing NFS share with a networking path for access from Veeam Kasten on Red Hat OpenShift.
• The name of the PersistentVolumeClaim (PVC) mounting the NFS export location. For setup instructions, see the Veeam Kasten documentation.
• A path suffix to use inside the NFS export path (optional unless supplemental groups are set).
• A supplemental group that has access to the path (optional).

Use the following steps to create an NFS location profile.

1. From the Veeam Kasten web interface, go to Profiles > Location and click Create profile.
2. In the Name and provider page, enter the Location profile name and in the Storage provider dropdown menu select NFS/SMB.
3. Click Next
4. In the Configure NFS/SMB profile page, enter the following information:
   1. Claim name
   2. Path (optional unless you specify a supplemental group)
   3. Supplemental group (optional)
5. Click Next
6. Review the Summary and click Submit.

Creating a Veeam Scale-Out Backup Repository location profile

Creating a Veeam Scale-Out Backup Repository (SOBR) location profile requires the following prerequisites:

• A Veeam SOBR that is configured with a networking path for access from Veeam Kasten on Red Hat OpenShift
• Veeam Backup & Replication (VBR) server name and location details
• Access to credentials with adequate permissions on the VBR
• In the Veeam VBR console, confirm that access permissions in the repository are configured to allow Veeam Kasten access.
• On the Backup Infrastructure page, select your SOBR under Scale-Out Repositories and select Access permissions.
• To make the setup simple, select Allow for everyone.

For a secure setup, configure access by using only a service user.

• In the VPC that hosts Kasten, confirm that the relevant security group allows traffic between the VBR and Kasten.

Use the following steps to create a Veeam Scale-Out Backup Repository location profile.

1. From the Veeam Kasten web interface, go to Profiles > Location.
2. Click Create New Profile
3. In the Name and Provider page, enter the Location profile name and in the Storage provider menu select Veeam Repository.
4. Click Next
5. In the Configure Veeam Profile page, enter the following information:
   • The Veeam backup server
   • TVeeam Backup Server API port
   • Backup repository
   • Username
   • Password
   • Choose whether to Skip certificate chain and hostname verification.
6. Click Next
7. Review the Summary and click Submit.

Creating a data protection policy in the Veeam Kasten web UI

The policy that is used in a data protection backup action is determined by the following questions:

• What data is backed up?
• How is the data backed up?
• Where is the data is copied?
• When does the policy run?
• How long is the data stored?

Every data protection restore option must be tested frequently to verify that the requirements are being met.

After the requirements are defined, you can create a data protection policy by following these steps:

1. From the Veeam Kasten web interface, go to policies > Policies.
2. Click Create a policy.
3. Provide a Name for the policy. Optionally, you can add Comments to identify, describe, or share helpful information about the policy.
4. In the Action entry, choose Snapshot and define the following options.
   1. Use the preset option (optional).
      • Enabled: select a pre-defined schedule and export settings from the settings menu.
      • Disabled: define custom schedule and export settings in the Backup frequency section.
   2. Set the back up frequency (if the preset option is disabled).
      • Select the back up schedule (hourly, daily, weekly, monthly, yearly, or on demand).
   3. Define the advanced frequency options.
      • Enabled:
        • Select Use local or UTC time.
        • Select the Hours of the day for daily snapshots.
      • Disabled: nothing to set
   4. Select a back up window. If the backup creation is not finished within the selected interval, the back up cancels.
      • Enabled:
        • Use staggering:
          • Enabled: You can stagger when policies run to help reduce peak loads on the system.
          • Disabled: Staggering doesn't run.
        • Define the From and To time for the backup interval.
   5. Snapshot retention: You can customize the snapshot retention schedule. Use the user interface to set daily, weekly, monthly, and yearly snapshots.
   6. Enable Backups through Snapshot exports (optional but recommended)
      • Enabled:
        • Set an export frequency of daily, weekly, monthly, or yearly.
        • Export location profile: You can create or select an existing location profile.
        • Retention of exported snapshots: You define or use an existing one.
        • Storage class exceptions are used to define exceptions for specific storage classes
      • Disabled: nothing to define.
   7. Selection Type: Namespaces or virtual servers
      • Namespaces:
        • Select applications: if namespaces is selected you must select applications.
        • Snapshot cluster-scoped resources (optional)
          • All cluster-scoped resources
          • Filter cluster-scoped resources
            • Include filters - based inclusion rules, define Group, Version, Resource, Name, or Label.
            • Exclude filters - based exclusion rules, define Group, Version, Resource, Name, or Label.
      • Virtual servers:
        • Specify virtual servers: if virtual servers are selected, you must specify the virtual servers by selecting one or more namespaces.
        • Select VM resources to create filters that include and exclude resources that are dependencies for each virtual server.
5. Advanced settings (optional)
   1. Pre and post-snapshot action hooks: These actions hooks are blueprint actions that you can apply before or after a snapshot success, or after a snapshot failure.
   2. Location profile for Kanister actions: Specify a location profile for Kanister Blueprint.
   3. Location profile for container images: If the application snapshot includes ImageStreams specify a container image location profile.
   4. Pre and post-export action hooks: These action hooks are blueprint actions that you can apply before or after an export success, or after an export failure.
   5. Ignore exceptions and continue if possible. This action is useful in environments when applications are in an unhealthy state, but the policy actions can continue.

Continuous log shipping for stateful database virtual servers

Continuous log shipping for stateful database virtual servers is not supported by Veeam Kasten.

If such a feature is required, it needs to be facilitated through agent-based backup by using a VBR server.

See the following documentation that is an example for MS SQL server: Agents-based log shipping by using VBR.

Virtual servers that were migrated by using the Red Hat Migration Toolkit for Virtualization

If you use Kasten to back up and restore virtual servers, some issues can occur. These issues occur for virtual servers that are migrated by using the Red Hat Migration Toolkit for Virtualization, if the migration plan is not archived and deleted after the migration. These issues can also occur because of references to storage resources that are not released after the migration. For more information, see PVCs are stuck in a stopping state {: external}.

Federal Information Processing Standard (FIPS)

You can deploy Kasten in a FIPS-compliant mode through only the helm chart with the correct parameters selected. Because not all components that connect to Kasten are FIPS-compliant, carefully review the FIPS configuration to avoid enabling features that are not FIPS-compliant.

For more information, see Installing Kasten in FIPS mode.

Veeam Backup & Replication (VBR) repository backup

Backups exported to VBR repositories have the following limitation: the VBR repository stores only snapshot data, not the application metadata.

The VBR repository can store only the snapshot data itself and not the metadata that is a part of a backed-up application in Red Hat OpenShift.

When you configure a backup policy that exports to a VBR repository, you also need to select a secondary export location like NFS or an object storage bucket. This means that to successfully restore from a VBR repository, Veeam Kasten needs to pull the snapshot data from the VBR repository and the metadata from the other export location.

If immutability is required, you must configure immutability on both the VBR repository and the secondary immutable object storage bucket. The immutability period that is configured for the object storage bucket location in Veeam Kasten must be less than or equal to the immutability period that is defined in the VBR repository.

For more information, see VBR Considerations.

Recommendations to follow the 3-2-1 backup rule

See the following best practices to follow the 3-2-1 backup rule.

• Retain three copies of the data. These copies can be original data in the persistent volume claim (PVC), the snapshot data, and the snapshot data that is exported to an external location.

• Storing the copies on two different media. Veeam Kasten supports multiple storage infrastructures, including block, file, and object storage. For example, you store your data on PVC and external S3. Having two copies of data, one on PVC and the other as a local snapshot in the Kubernetes cluster, does not count as two different media. In a disaster, the cluster is beyond recoverable and both copies of data are lost.

• Keep one backup copy off-site. Set up backup copy jobs to transfer your backups offsite to another location, such as a public cloud provider or auxiliary storage in a separate site. Exporting backups to a locally hosted profile through NFS file storage is not considered offsite.

You can follow the 3-2-1 rule in several ways. The simplest is to export your backup data to an IBM Cloud Object Storage bucket. You can also use Veeam SOBR for enhanced 3-2-1 rule. You have the same backup copy on performance tier and capacity tier.

Immutability

Enable immutability wherever possible to protect against ransomware attacks. You can configure immutability on S3-compatible buckets and on Veeam SOBR when you use a Linux Hardened Repository.

When you use immutable buckets as an export location, Kasten adds a hidden buffer period to the retention period. This buffer affects the actual immutability duration. According to Kasten documentation, the following points apply when you create an S3 immutable bucket profile:

• The maximum protection period is 90 days.
• A safety buffer is automatically added to the selected protection period to help safeguard ongoing protection for new objects that are written to the bucket before their retention expires.
• The minimum protection period is 1 day.

For more information, see Setting Up Immutable Backup Protection.

Use Kasten Disaster Recovery (DR) to backup configurations

Kasten can help back up and restore Kasten configuration through the Kasten DR feature, which can be configured as part of best practices to recover your Kasten instance from any unintended actions.

Kasten cannot be recovered without a few pieces of information that enable an administrator to restore the Kasten environment.

• Depending on how Kasten DR is configured, you need the passphrase information that was used to set up Kasten DR.
• One or more export locations where the DR backups were exported.
• Cluster ID, an auto-generated ID when Kasten DR is set up.

Make a note of the value shown in the Kasten DR tab on the Kasten UI for successful recovery.

For more information, see the Kasten DR documentation.

Kasten multi-cluster feature

Veeam Kasten can set up a multi-cluster configuration where all your Kasten environments can be managed under one UI. Depending on the number of clusters you have, it might or might not be a beneficial feature to implement.

See Getting Started instructions on enabling the Veeam Kasten Multi-cluster feature.

For the multi-cluster feature, for environments that use a root CA certificate, the CA certificate needs to be applied to Veeam Kasten.

Follow the steps in Kasten certificate management to install root certificates.

Blueprints

Veeam Kasten has a tool inside the product that is known as Kanister. Kanister is a tool that allows users to write blueprints that run based on hooks in the backup or restore process. The available hooks allow blueprint actions to run before and post snapshot and export operations. These blueprints are yaml files with code inside them that define what action needs to be taken when a backup or restore operation runs.

The main use case of these blueprints is to enable Kasten to conduct application-aware backups. These blueprints run automated steps to help ensure that applications are properly backed up when a simple snapshot is insufficient. Typical uses cases revolve around database backup.

Application-aware backups of databases use the available template blueprints. Review your use case and update the blueprint to meet your specific requirements.

Veeam provides several examples in their documentation. See a PostgreSQL example at Application Consistent PostgreSQL Backup.

For more information, see the Kanister documentation to understand the blueprint files.

For more information, see the Blueprints documentation.

Transforms

Veeam Kasten has a feature that is called Transforms that allows users to automatically modify applications during restore operations.

Transforms happen on restore operations only. They provide a way to transform the final state of the restored workload to conform with the landing location, by changing the metadata configuration of the restored workload. Transforms can be defined through the Kasten UI and filling out the form with the wanted configuration changes.

Transforms are arranged into transform sets. A transform set contains one or more transforms. The order of the transforms defined in the transform sets determines the order that they run during execution.

Each transform can have one or more command operations that are tied to the transform. Several different transform operations are available: test, add, remove, copy, move, and replace (see doc). The test command can be used as a pseudo conditional statement. For example, "if a test fails, the remaining commands in the current transform document will be skipped and processing of the next transform document will begin." Otherwise, failure of the other commands results in the failure of the transform.

For more information, see Veeam Kasten Transforms.