調整 Red Hat CoreOS 工作者節點的效能

Satellite

支援的工作者節點作業系統: Red Hat CoreOS (RHCOS)

您可以透過啟用 CPU pinning、非均勻記憶體存取 (NUMA) 和巨頁來調整 Red Hat CoreOS Worker 節點的效能。這些組態可讓有嚴格效能要求的應用獲益。不過，這些客製化可能會造成工作負載排程問題。

您可以使用 daemonset 檔案修改主機，而不是使用 Red Hat OpenShift 中的 MachineConfig 檔案調整 Worker 節點效能。如需詳細資訊，請參閱變更 Calico MTU 或調整 Red Hat CoreOS Worker 節點的效能。

部署 Node Feature Discovery Operator

Satellite

您必須先部署 Node Feature Discovery Operator，然後才能在工作者節點上啟用 NUMA、CPU 綁定及巨型分頁。如需詳細資訊，請參閱 Node Feature Discovery Operator。

在工作者節點上啟用非一致記憶體存取 (NUMA)、CPU 固定及巨型分頁

Satellite

開始之前，請確定您已部署 Node Feature Discovery Operator。

將下列 DaemonSet 儲存至稱為 customize.yaml 的檔案。

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ibm-user-custom-configurator-privileged
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:openshift:scc:privileged
subjects:
  - kind: ServiceAccount
    name: ibm-user-custom-configurator
    namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ibm-user-custom-configurator
  namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: ibm-user-custom-configurator
  namespace: kube-system
data:
  89-hugepages.conf: |
    vm.nr_hugepages=<<NUMBER_OF_HUGEPAGES>>
  configure.sh: |
    #!/usr/bin/env bash
    set -x
    cp -f /scripts/ibm-user-custom-configuration.sh /host-usr-local-bin/ibm-user-custom-configuration.sh
    chmod 0755 /host-usr-local-bin/ibm-user-custom-configuration.sh
    cp -f /scripts/ibm-user-custom-configuration.service /host-etc-systemd-dir/ibm-user-custom-configuration.service
    chmod 0644 /host-etc-systemd-dir/ibm-user-custom-configuration.service
    if [[ -f /scripts/89-hugepages.conf ]]; then
      cp -f /scripts/89-hugepages.conf /host-etc-systctld-dir/89-hugepages.conf
    fi
    nsenter -t 1 -m -u -i -n -p -- systemctl daemon-reload
    nsenter -t 1 -m -u -i -n -p -- systemctl enable ibm-user-custom-configuration.service
    nsenter -t 1 -m -u -i -n -p -- systemctl start ibm-user-custom-configuration.service
  ibm-user-custom-configuration.sh: |
    #!/usr/bin/env bash
    set -x
    GIGABYTES_RESERVED_MEMORY=$(echo $SYSTEM_RESERVED_MEMORY | awk -F 'Gi' '{print $1}')
    GIGABYTES_RESERVED_MEMORY_ROUNDED_UP=$(echo $GIGABYTES_RESERVED_MEMORY | awk '{print int($1+0.999)}')
    sed -i "s/SYSTEM_RESERVED_MEMORY=.*/SYSTEM_RESERVED_MEMORY=${GIGABYTES_RESERVED_MEMORY_ROUNDED_UP}Gi/g" /etc/node-sizing.env
    TOTAL_NUMA_MEMORY_TO_ALLOCATE=$(echo "$GIGABYTES_RESERVED_MEMORY_ROUNDED_UP" "1024" | awk '{print $1 * $2 + 100}')
    if cat /etc/kubernetes/kubelet.conf | jq -r .; then
      cat >/tmp/ibm-user-config.conf.json <<EOF
      {
        "topologyManagerPolicy": "<<TOPOLOGY_MANAGER_POLICY_VALUE>>",
        "memoryManagerPolicy": "Static",
        "cpuManagerPolicy": "static",
        "reservedMemory": [
          {
            "numaNode": 0,
            "limits": {
            "memory": "${TOTAL_NUMA_MEMORY_TO_ALLOCATE}Mi"
              }
          }
        ]
      }
    EOF
      if ! cat /tmp/ibm-user-config.conf.json | jq -r .; then
        exit 1
      fi
      if ! jq -s '.[0] * .[1]' /tmp/ibm-user-config.conf.json /etc/kubernetes/kubelet.conf > /etc/kubernetes/tmp-kubelet.conf; then
        exit 1
      fi
      mv -f /etc/kubernetes/tmp-kubelet.conf /etc/kubernetes/kubelet.conf
      else
        cat >/tmp/ibm-user-config.conf <<EOF
    #START USER CONFIG
    topologyManagerPolicy: <<TOPOLOGY_MANAGER_POLICY_VALUE>>
    memoryManagerPolicy: Static
    cpuManagerPolicy: static
    reservedMemory:
      - numaNode: 0
        limits:
          memory: ${TOTAL_NUMA_MEMORY_TO_ALLOCATE}Mi
    #END USER CONFIG
    EOF
      sed -i '/#START USER CONFIG/,/#END USER CONFIG/d' /etc/kubernetes/kubelet.conf
      cat /tmp/ibm-user-config.conf >>/etc/kubernetes/kubelet.conf
    fi
  ibm-user-custom-configuration.service: |
    [Unit]
    Description=Add custom user config to kubelet
    Before=kubelet.service
    After=kubelet-auto-node-size.service
    [Service]
    Type=oneshot
    RemainAfterExit=yes
    EnvironmentFile=/etc/node-sizing.env
    ExecStart=/usr/local/bin/ibm-user-custom-configuration.sh
    [Install]
    WantedBy=multi-user.target
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: ibm-user-custom-configurator
  name: ibm-user-custom-configurator
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: ibm-user-custom-configurator
  template:
    metadata:
      labels:
        app: ibm-user-custom-configurator
    spec:
      nodeSelector:
        feature.node.kubernetes.io/memory-numa: "true"
        ibm-cloud.kubernetes.io/os: RHCOS
      tolerations:
        - operator: "Exists"
      hostPID: true
      serviceAccount: ibm-user-custom-configurator
      initContainers:
        - name: configure
          image: "registry.access.redhat.com/ubi8/ubi:8.6"
          command: ['/bin/bash', '-c', 'mkdir /cache && cp /scripts/configure.sh /cache && chmod +x /cache/configure.sh && /bin/bash /cache/configure.sh']
          securityContext:
            privileged: true
          volumeMounts:
            - mountPath: /scripts
              name: script-config
            - mountPath: /host-etc-systemd-dir
              name: etc-systemd-dir
            - mountPath: /host-usr-local-bin
              name: usr-local-bin
            - mountPath: /host-etc-systctld-dir
              name: etc-systctld-dir
      containers:
        - name: pause
          image: us.icr.io/armada-master/pause:3.2
      volumes:
        - name: etc-systemd-dir
          hostPath:
            path: /etc/systemd/system
        - name: etc-systctld-dir
          hostPath:
            path: /etc/sysctl.d
        - name: usr-local-bin
          hostPath:
            path: /usr/local/bin
        - name: script-config
          configMap:
            name: ibm-user-custom-configurator

編輯 DaemonSet 值以調整效能。

NUMBER_OF_HUGEPAGES

輸入您要配置的巨型分頁數目。例如： 2048。如果您不想啟用巨型分頁，請輸入 0。您配置的巨型分頁越多，應用程式可用的整體記憶體就越少。

TOPOLOGY_MANAGER_POLICY_VALUE

輸入要使用的拓樸管理員原則。建議使用 best-effort 政策，以確保最大的排程可用性。但是，您可以使用其他政策進行更嚴格的需求驗證，同時降低工作負載排程的可用性。如需詳細資訊，請參閱 Topology Manager。

您可以編輯 nodeSelector 區段，以僅將配置套用至工作者節點的子集。

請執行以下指令，套用 DaemonSet。

kubectl replace --force -f customize.yaml

驗證 Pod 已進入 Running 狀態。

kubectl get pods -n kube-system -l app=ibm-user-custom-configurator -o wide

在 Pod 執行之後，將每一個工作者節點重新開機。

在工作者節點上部署除錯 Pod。

    oc debug node/NODE_NAME
    ```
1. 在除錯階段作業啟動之後，請執行下列指令。
```sh {: pre}
    nsenter -t 1 -m -u -i -n -p -- reboot
    ```
1. 請針對每個您想要重新啟動的工作節點，重複執行這些步驟。

在工作者節點上啟用 CPU 固定及巨型分頁

Satellite

開始之前，請確定您已部署 Node Feature Discovery Operator。

將下列 DaemonSet 儲存至稱為 cpu-pinning.yaml 的檔案。

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ibm-user-custom-configurator-privileged
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:openshift:scc:privileged
subjects:
  - kind: ServiceAccount
    name: ibm-user-custom-configurator
    namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ibm-user-custom-configurator
  namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: ibm-user-custom-configurator
  namespace: kube-system
data:
  89-hugepages.conf: |
    vm.nr_hugepages=<<NUMBER_OF_HUGEPAGES>>
  configure.sh: |
    #!/usr/bin/env bash
    set -x
    cp -f /scripts/ibm-user-custom-configuration.sh /host-usr-local-bin/ibm-user-custom-configuration.sh
    chmod 0755 /host-usr-local-bin/ibm-user-custom-configuration.sh
    cp -f /scripts/ibm-user-custom-configuration.service /host-etc-systemd-dir/ibm-user-custom-configuration.service
    chmod 0644 /host-etc-systemd-dir/ibm-user-custom-configuration.service
    if [[ -f /scripts/89-hugepages.conf ]]; then
      cp -f /scripts/89-hugepages.conf /host-etc-systctld-dir/89-hugepages.conf
    fi
    nsenter -t 1 -m -u -i -n -p -- systemctl daemon-reload
    nsenter -t 1 -m -u -i -n -p -- systemctl enable ibm-user-custom-configuration.service
    nsenter -t 1 -m -u -i -n -p -- systemctl start ibm-user-custom-configuration.service
  ibm-user-custom-configuration.sh: |
    #!/usr/bin/env bash
    set -x
    if cat /etc/kubernetes/kubelet.conf | jq -r .; then
      cat >/tmp/ibm-user-config.conf.json <<EOF
      {
        "cpuManagerPolicy": "static"
      }
    EOF
      if ! cat /tmp/ibm-user-config.conf.json | jq -r .; then
        exit 1
      fi
      if ! jq -s '.[0] * .[1]' /tmp/ibm-user-config.conf.json /etc/kubernetes/kubelet.conf > /etc/kubernetes/tmp-kubelet.conf; then
        exit 1
      fi
      mv -f /etc/kubernetes/tmp-kubelet.conf /etc/kubernetes/kubelet.conf
    else
      cat >/tmp/ibm-user-config.conf <<EOF
    #START USER CONFIG
    cpuManagerPolicy: static
    #END USER CONFIG
    EOF
      sed -i '/#START USER CONFIG/,/#END USER CONFIG/d' /etc/kubernetes/kubelet.conf
      cat /tmp/ibm-user-config.conf >>/etc/kubernetes/kubelet.conf
    fi
  ibm-user-custom-configuration.service: |
    [Unit]
    Description=Add custom user config to kubelet
    Before=kubelet.service
    After=kubelet-auto-node-size.service
    [Service]
    Type=oneshot
    RemainAfterExit=yes
    EnvironmentFile=/etc/node-sizing.env
    ExecStart=/usr/local/bin/ibm-user-custom-configuration.sh    
    [Install]
    WantedBy=multi-user.target
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: ibm-user-custom-configurator
  name: ibm-user-custom-configurator
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: ibm-user-custom-configurator
  template:
    metadata:
      labels:
        app: ibm-user-custom-configurator
    spec:
      nodeSelector:
        ibm-cloud.kubernetes.io/os: RHCOS
      tolerations:
        - operator: "Exists"
      hostPID: true
      serviceAccount: ibm-user-custom-configurator
      initContainers:
        - name: configure
          image: "registry.access.redhat.com/ubi8/ubi:8.6"
          command: ['/bin/bash', '-c', 'mkdir /cache && cp /scripts/configure.sh /cache && chmod +x /cache/configure.sh && /bin/bash /cache/configure.sh']
          securityContext:
            privileged: true
          volumeMounts:
            - mountPath: /scripts
              name: script-config
            - mountPath: /host-etc-systemd-dir
              name: etc-systemd-dir
            - mountPath: /host-usr-local-bin
              name: usr-local-bin
            - mountPath: /host-etc-systctld-dir
              name: etc-systctld-dir
      containers:
        - name: pause
          image: us.icr.io/armada-master/pause:3.2
      volumes:
        - name: etc-systemd-dir
          hostPath:
            path: /etc/systemd/system
        - name: etc-systctld-dir
          hostPath:
            path: /etc/sysctl.d
        - name: usr-local-bin
          hostPath:
            path: /usr/local/bin
        - name: script-config
          configMap:
            name: ibm-user-custom-configurator

編輯 DaemonSet 值以調整效能。

NUMBER_OF_HUGEPAGES

輸入您要配置的巨型分頁數目。例如： 2048。如果您不想啟用巨型分頁，請輸入 0。您配置的巨型分頁越多，應用程式可用的整體記憶體就越少。

您可以編輯 nodeSelector 區段，以僅將配置套用至工作者節點的子集。

請執行以下指令，套用 DaemonSet。

kubectl replace --force -f cpu-pinnning.yaml

驗證 Pod 已進入 Running 狀態。

kubectl get pods -n kube-system -l app=ibm-user-custom-configurator -o wide

在 Pod 執行之後，將每一個工作者節點重新開機。

在工作者節點上部署除錯 Pod。

    oc debug node/NODE_NAME
    ```
1. 在除錯階段作業啟動之後，請執行下列指令。
```sh {: pre}
    nsenter -t 1 -m -u -i -n -p -- reboot
    ```
1. 請針對每個您想要重新啟動的工作節點，重複執行這些步驟。

啟用 `kernel-devel` 套件

Satellite

您可能需要啟用 kernel-devel 套件，才能使用 Satellite 服務或儲存體，例如 Spectrum Scale Fusion。

透過將自訂配置對映及機器配置套用至工作者節點，完成下列步驟以啟用 kernel-devel。

執行以下指令以套用 MachineConfig。

ibmcloud ks cluster config --cluster CLUSTERID
cat >"/tmp/kernel-devel-payload.yaml" <<EOF
apiVersion: v1
kind: List
metadata:
  name: pvg-machine-config-tester
  annotations:
items:
  - apiVersion: v1
    kind: Namespace
    metadata:
      name: ibm-machine-config
  - apiVersion: v1
    data:
      config: |+
        apiVersion: machineconfiguration.openshift.io/v1
        kind: MachineConfig
        metadata:
          name: 97-kerneldevel
          labels:
            machineconfiguration.openshift.io/role: worker
        spec:
          config:
            ignition:
              version: 3.2.0
          extensions:
          - kernel-devel
    kind: ConfigMap
    metadata:
      labels:
        ibm-cloud.kubernetes.io/user-specified-config: "true"
      name: user-ignition-config-97-kerneldevel
      namespace: ibm-machine-config
EOF
kubectl apply -f /tmp/kernel-devel-payload.yaml

請等待資源完成部署。這可能需要 5 分鐘或更長時間。

檢閱配置對映的詳細資料，以確認部署成功。

確認 config-validation="valid" 欄位存在。

    kubectl get cm -n ibm-machine-config user-ignition-config-97-kerneldevel -o yaml | grep config-validation
    ```
1. 確認 `user-ignition-config-97-kerneldevel` 存在於配置對映中。
```sh {: pre}
    kubectl get cm -n ibm-machine-config -l ibm-cloud.kubernetes.io/nodepoolfeedback="true" -o yaml | grep user-ignition-config-97-kerneldevel
    ```

將工作節點新增至您的叢集。您新增的工作者節點已啟用 kernel-devel。

驗證 kernel-devel 已啟用。

在其中一個節點上啟動除錯 Pod。

    oc debug node/NODEIP
    ```
1. 執行以下 `nsenter` 指令。
```sh {: pre}
    nsenter -t 1 -m -u -i -n -p -- rpm -qa | grep kernel-devel
    ```

選用: 如果不再需要 kernel-devel，您可以執行下列指令來移除它。
```
kubectl delete cm -n ibm-machine-config user-ignition-config-97-kerneldevel
```

移除效能自訂作業

Satellite

如果您要從工作者節點移除自訂作業並將它們重設為預設配置，請套用下列 DaemonSet。

將下列 DaemonSet 儲存至稱為 remove-custom.yaml 的檔案。

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ibm-user-custom-configurator-privileged
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:openshift:scc:privileged
subjects:
  - kind: ServiceAccount
    name: ibm-user-custom-configurator
    namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ibm-user-custom-configurator
  namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: ibm-user-custom-configurator
  namespace: kube-system
data:
  89-hugepages.conf: |
    vm.nr_hugepages=0
  configure.sh: |
    #!/usr/bin/env bash
    set -x
    cp -f /scripts/ibm-user-custom-configuration.sh /host-usr-local-bin/ibm-user-custom-configuration.sh
    chmod 0755 /host-usr-local-bin/ibm-user-custom-configuration.sh
    cp -f /scripts/ibm-user-custom-configuration.service /host-etc-systemd-dir/ibm-user-custom-configuration.service
    chmod 0644 /host-etc-systemd-dir/ibm-user-custom-configuration.service
    if [[ -f /scripts/89-hugepages.conf ]]; then
      cp -f /scripts/89-hugepages.conf /host-etc-systctld-dir/89-hugepages.conf
    fi
    nsenter -t 1 -m -u -i -n -p -- systemctl daemon-reload
    nsenter -t 1 -m -u -i -n -p -- systemctl enable ibm-user-custom-configuration.service
    nsenter -t 1 -m -u -i -n -p -- systemctl start ibm-user-custom-configuration.service
  ibm-user-custom-configuration.sh: |
    #!/usr/bin/env bash
    set -x
    if cat /etc/kubernetes/kubelet.conf | jq -r .; then
      if ! jq 'del(.topologyManagerPolicy, .memoryManagerPolicy, .cpuManagerPolicy, .reservedMemory)' /etc/kubernetes/kubelet.conf > /etc/kubernetes/tmp-kubelet.conf; then
        exit 1
      fi
      mv -f /etc/kubernetes/tmp-kubelet.conf /etc/kubernetes/kubelet.conf
    else
      sed -i '/#START USER CONFIG/,/#END USER CONFIG/d' /etc/kubernetes/kubelet.conf
    fi
  ibm-user-custom-configuration.service: |
    [Unit]
    Description=Add custom user config to kubelet
    Before=kubelet.service
    After=kubelet-auto-node-size.service
    [Service]
    Type=oneshot
    RemainAfterExit=yes
    EnvironmentFile=/etc/node-sizing.env
    ExecStart=/usr/local/bin/ibm-user-custom-configuration.sh
    [Install]
    WantedBy=multi-user.target
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: ibm-user-custom-configurator
  name: ibm-user-custom-configurator
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: ibm-user-custom-configurator
  template:
    metadata:
      labels:
        app: ibm-user-custom-configurator
    spec:
      nodeSelector:
        ibm-cloud.kubernetes.io/os: RHCOS
      tolerations:
        - operator: "Exists"
      hostPID: true
      serviceAccount: ibm-user-custom-configurator
      initContainers:
        - name: configure
          image: "registry.access.redhat.com/ubi8/ubi:8.6"
          command: ['/bin/bash', '-c', 'mkdir /cache && cp /scripts/configure.sh /cache && chmod +x /cache/configure.sh && /bin/bash /cache/configure.sh']
          securityContext:
            privileged: true
          volumeMounts:
            - mountPath: /scripts
              name: script-config
            - mountPath: /host-etc-systemd-dir
              name: etc-systemd-dir
            - mountPath: /host-usr-local-bin
              name: usr-local-bin
            - mountPath: /host-etc-systctld-dir
              name: etc-systctld-dir
      containers:
        - name: pause
          image: us.icr.io/armada-master/pause:3.2
      volumes:
        - name: etc-systemd-dir
          hostPath:
            path: /etc/systemd/system
        - name: etc-systctld-dir
          hostPath:
            path: /etc/sysctl.d
        - name: usr-local-bin
          hostPath:
            path: /usr/local/bin
        - name: script-config
          configMap:
            name: ibm-user-custom-configurator

請執行以下指令，將 DaemonSet 套用至您的叢集。
```
kubectl replace --force -f remove-custom.yaml
```

檢查 Pod 是否已進入 Running 狀態。

kubectl get pods -n kube-system -l app=ibm-user-custom-configurator -o wide

在 Pod 執行之後，將每一個工作者節點重新開機。

在工作者節點上部署除錯 Pod。

    oc debug node/NODE_NAME
    ```
1. 在除錯階段作業啟動之後，請執行下列指令。
```sh {: pre}
    nsenter -t 1 -m -u -i -n -p -- reboot
    ```
1. 請針對每個您想要重新啟動的工作節點，重複執行這些步驟。

調整 Red Hat CoreOS 工作者節點的效能

調整 Red Hat CoreOS 工作者節點的效能

部署 Node Feature Discovery Operator

在工作者節點上啟用非一致記憶體存取 (NUMA)、CPU 固定及巨型分頁

在工作者節點上啟用 CPU 固定及巨型分頁

啟用 kernel-devel 套件

移除效能自訂作業

啟用 `kernel-devel` 套件