Tuning performance

Default worker node settings

By default, your worker nodes have the operating system and compute hardware of the worker node flavor that you choose when you create the worker pool.

Modifying worker node settings to optimize performance

Optimizing network keepalive sysctl settings

If a pod has long running TCP connections that are occasionally disconnected when they are idle for a period of time, it might help to change the sysctl keepalive settings for the pod.

These scenarios and suggested settings are also described in the Troubleshooting Outgoing Connection Issues with IBM VPC Public and Service Gateways blog.

There currently isn't a way to set these sysctl keepalive settings on all pods by default in a cluster. The best way to modify the settings on all pods is to use a privileged initContainer. Review the following example of how to set up an initContainer for a deployment in a test-ns namespace.

Allow privileged initContainers in the test-ns namespace:

```sh {: pre}
oc adm policy add-scc-to-groupl privileged system:serviceaccounts:test-ns
```

Deploy the following example initContainer. Remember to change the containers: section to your own application containers. The initContainer then sets the sysctl settings for all the regular containers in the pod because they all share the same network namespace.

```sh {: pre}
kubectl apply -f - << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-sysctl
  namespace: test-ns
  labels:
    run: test-sysctl
spec:
  replicas: 2
  selector:
    matchLabels:
      run: test-sysctl
  template:
    metadata:
      labels:
        run: test-sysctl
    spec:
      initContainers:
      - command:
        - sh
        - -c
        - sysctl -e -w net.ipv4.tcp_keepalive_time=40; sysctl -e -w net.ipv4.tcp_keepalive_intvl=15; sysctl -e -w net.ipv4.tcp_keepalive_probes=6;
        image: us.icr.io/armada-master/alpine:latest
        imagePullPolicy: IfNotPresent
        name: sysctl-init
        resources: {}
        securityContext:
          privileged: true
      containers:
      - name: test-sysctl
        image: us.icr.io/armada-master/alpine:latest
        command: ["sleep", "2592000"]
  EOF
```      

Changing the Calico maximum transmission unit (MTU)

Increase or decrease the Calico plug-in maximum transmission unit (MTU) to meet the network throughput requirements of your environment.

All VPC workers nodes support jumbo frames. However, on classic infrastructure, only bare metal workers support jumbo frames.

By default, the Calico network plug-in in your Red Hat OpenShift on IBM Cloud cluster has an MTU of 1480 bytes. For most cases, this default MTU value provides sufficient throughput for packets that are sent and received in your network workloads. Review the following cases in which you might need to modify the default Calico MTU:

• Jumbo frames have an MTU value in the range of 1500 to 9000. To ensure that your cluster's pod network can use this higher MTU value, you can increase the Calico MTU to 20 bytes less than the jumbo frame MTU. This 20 byte difference allows space for packet header on encapsulated packets. For example, if your worker nodes' jumbo frames are set to 9000, you can set the Calico MTU to 8980. Note that all worker nodes in the cluster must use the same Calico MTU, so to increase the Calico MTU, all worker nodes in the cluster must be bare metal and use jumbo frames.
• If you have a VPN connection set up for your cluster, some VPN connections require a smaller Calico MTU than the default. Check with the VPN service provider to determine whether a smaller Calico MTU is required.
• If your cluster's worker nodes exist on different subnets, increasing the MTU value for the worker nodes and for the Calico MTU can allow pods to use the full bandwidth capability of the worker nodes.

Before you begin

If your worker nodes still run the default MTU value, increase the MTU value for your worker nodes first before you increase the MTU value for the Calico plug-in. For example, you can apply the following daemon set to change the MTU for your worker nodes jumbo frames to 9000 bytes. Note the interface names that are used in the ip link command vary depending on the type of your worker nodes.

• Example command for Bare Metal worker nodes: ip link set dev bond0 mtu 9000;ip link set dev bond1 mtu 9000;
• Example command VPC Gen 2 worker nodes: ip link set dev ens3 mtu 9000;

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: jumbo-apply
  namespace: kube-system
  labels:
    tier: management
    app: jumbo-apply
spec:
  selector:
    matchLabels:
      name: jumbo-apply
  template:
    metadata:
      labels:
        name: jumbo-apply
    spec:
      hostNetwork: true
      hostPID: true
      hostIPC: true
      tolerations:
      - operator: Exists
      initContainers:
        - command:
            - sh
            - -c
            - ip link set dev bond0 mtu 9000;ip link set dev bond1 mtu 9000; # Update this command based on your worker node type.
          image: alpine:3.6
          imagePullPolicy: IfNotPresent
          name: iplink
          resources: {}
          securityContext:
            privileged: true
            capabilities:
              add:
                - NET_ADMIN
          volumeMounts:
            - name: modifysys
              mountPath: /sys
      containers:
        - resources:
            requests:
              cpu: 0.01
          image: alpine:3.6
          name: sleepforever
          command: ["/bin/sh", "-c"]
          args:
            - >
              while true; do
                sleep 100000;
              done
      volumes:
        - name: modifysys
          hostPath:
             path: /sys

Disabling the port map plug-in

The portmap plug-in for the Calico container network interface (CNI) enables you to use a hostPort to expose your app pods on a specific port on the worker node. Prevent iptables performance issues by removing the port map plug-in from your cluster's Calico CNI configuration.

When you have many services in your cluster, such as more than 500 services, or many ports on services, such as more than 50 ports per service for 10 or more services, many iptables rules are generated for the Calico and Kubernetes network policies for these services. Using many iptables rules can lead to performance issues for the port map plug-in, and might prevent future updates of iptables rules or cause the calico-node container to restart when no lock is received to make iptables rules updates within a specified time. To prevent these performance issues, you can disable the port map plug-in by removing it from your cluster's Calico CNI configuration.

If you must use hostPorts, don't disable the port map plug-in.

1. Edit the default Calico installation resource.

   oc edit installation default -n calico-system
   

2. In the spec.calicoNetwork section, change the value of hostPorts to Disabled.

   ...
   spec:
     calicoNetwork:
       hostPorts: Disabled
       ipPools:
       - cidr: 172.30.0.0/16
         encapsulation: IPIPCrossSubnet
         natOutgoing: Enabled
         nodeSelector: all()
       mtu: 1480
       nodeAddressAutodetectionV4:
         interface: (^bond0$|^eth0$|^ens6$|^ens3$)
     kubernetesProvider: OpenShift
     registry: registry.ng.bluemix.net/armada-master/
     variant: Calico
   status:
     variant: Calico
   

3. Save and close the file. Your changes are automatically applied.