Why do I see an etcd database size is approaching the maximum error?

Virtual Private Cloud Classic infrastructure Satellite

You see an error message similar to the following.

etcd database size is approaching the maximum

Red Hat OpenShift on IBM Cloud uses an etcd database as its backing store for all cluster data. This database is where resources like configmaps, secrets, deployments, and all other Kubernetes resources are stored. Red Hat OpenShift on IBM Cloud limits the size of the backing etcd database to a maximum of 4 GiB, which provides sufficient capacity under normal operating conditions. However, if a user or process creates an extremely large number of objects, the in-use database size can reach the maximum.

Identify the resources which are occupying space in the etcd database.

1. Inspect the metrics from the apiserver by running the following command.

   kubectl get --raw /metrics
   

2. Search the output for apiserver_storage_objects metrics, which tells you how many of each kind of object is present in the etcd database. Look for resource types that have tens or hundreds of thousands of associated objects. These resources are the most likely cause of the issue. However, keep in mind that it is the size of these objects rather than their number that is important.

3. For the resources that you identified in the previous step, review and adjust their settings to reduce the number of objects that are getting created. For example, a common issue is misconfigured operators that automatically create a large number of secrets.

4. After adjust the resource settings, clean up the resources to bring the size of the etcd database back down.

5. If the issue persists, contact support. Open a support case. In the case details, be sure to include any relevant log files, error messages, or command outputs.