Setup On-Prem Kubernetes HA-Part 2

Moreover, we are going to create a kubeadm config file that holds all of our cluster definitions in just one place. (typically ordinary installations would use command arguments instead, but that’s not a recommended way unless you’re going for a simple installation)

To point out, my current setup being more packed having included the use of Audit Policies, Encryption and a central Container registry to bypass dockerhub rate limitations (Although covering these bits is out of scope for this guide, It’s still recommend to do on a production environment)

With that in mind, I added here the top most common parameters needed if you decide to go full metal jacket, typically this should be the complete kube-config)

if I missed anything, please do let me know in the comments section below.

Official Kuberentes Kubeadm Config

On Node1 (Primary Control node)

vim /root/kube-config.yaml

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
clusterName: "k8_peter_awesome"
kubernetesVersion: "1.18.13"
etcd:
  local:
    serverCertSANs:
    - 192.168.1.155
    - 192.168.1.156
    - 192.168.1.157
    - k8master1.peter.loc
    - k8master2.peter.loc
    - k8master3.peter.loc
    - 192.168.1.161
    - k8mastervip.peter.loc
    - localhost
    - 127.0.0.1
    peerCertSANs:
    - 192.168.1.155
    - 192.168.1.156
    - 192.168.1.157
    - k8master1.peter.loc
    - k8master2.peter.loc
    - k8master3.peter.loc
    - 192.168.1.161
    - k8mastervip.peter.loc
    - localhost
    - 127.0.0.1
    imageRepository: "dockeregistry.malaty.net/myproject"
    extraArgs:
      election-timeout: "5000"
      heartbeat-interval: "250"
      quota-backend-bytes: "4294967296"
dns:
  imageRepository: "dockeregistry.malaty.net/myproject"
apiServer:
  extraArgs:
    audit-log-path: "/var/log/audit/kube-apiserver-audit.log"
    audit-log-maxage: "30"
    audit-log-maxbackup: "10"
    audit-log-maxsize: "100"
    audit-policy-file: /etc/kubernetes/audit-policy/apiserver-audit-policy.yaml
    cloud-provider: ""
    enable-admission-plugins: "AlwaysPullImages,NodeRestriction"
    disable-admission-plugins: ""
    encryption-provider-config: /etc/kubernetes/pki/encryption-config.yaml
    profiling: "false"
    target-ram-mb: "2048"
    max-requests-inflight: "1200"
    max-mutating-requests-inflight: "400"

  certSANs:
    - 192.168.1.155
    - 192.168.1.156
    - 192.168.1.157
    - k8master1.peter.loc
    - k8master2.peter.loc
    - k8master3.peter.loc
    - 192.168.1.161
    - k8mastervip.peter.loc
    - localhost
    - 127.0.0.1
  extraVolumes:
    - name: audit-policy
      hostPath: /etc/kubernetes/audit-policy
      mountPath: /etc/kubernetes/audit-policy
    - name: audit-logs
      hostPath: /var/log/kubernetes/audit
      mountPath: /var/log/audit
      readOnly: false
controllerManager:
  extraArgs:
    cloud-provider: ""
    profiling: "false"
    terminated-pod-gc-threshold: "1000"
scheduler:
  extraArgs:
    profiling: "false"
    kube-api-burst: "120"
    kube-api-qps: "80"
controlPlaneEndpoint: "192.168.1.161:6443"
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.96.0.0/16"
imageRepository: "dockeregistry.malaty.net/myproject"
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
nodeRegistration:
  kubeletExtraArgs:
    cgroup-root: ""
    cloud-provider: ""
    event-burst: "30"
    event-qps: "0"
    fail-swap-on: "True"
    kube-api-burst: "30"
    kube-api-qps: "15"
    kube-reserved: ""
    max-pods: "110"
    node-labels: "k8master.peter.loc/inventory_hostname=192.168.1.155"
    node-ip: "192.168.1.155"
    pods-per-core: "0"
  criSocket: "unix:///run/containerd/containerd.sock"
  taints:
    - effect: NoSchedule
      key: node-role.kubernetes.io/master
localAPIEndpoint:
  advertiseAddress: "192.168.1.155"
  bindPort: 6443