focaaby's Notes

為什麼 EKS cluster 知道預設 CNI Plugin 為 Amazon VPC CNI plugin

Mao-Lin Wang — Tue, 20 Sep 2022 10:12:07 GMT

預設 EKS cluster 使用 Amazon VPC Container Network Interface（CNI）Plugin 作為 CNI。EKS 集群建置後，我們即可以透過 kubectl 命令查看 DaemonSet aws-node 而無需手動安裝 CNI plugin。

$ kubectl -n kube-system get ds
NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
aws-node     3         3         3       3            3                     5d11h
kube-proxy   3         3         3       3            3                     5d11h

本文將探討「為什麼 EKS cluster 知道預設 CNI Plugin 為 Amazon VPC CNI plugin」，希望理解 EKS CNI plugin 設定過程。

work node

與上一篇查看 kubelet systemd unit 設定檔時，我們其實已經看過 kubelet 啟用 --network-plugin=cni。kubelet 將會讀取 --network-plugin-dir 目錄設定檔，並使用 CNI 設定檔設置每一個 Pod network。

kubelet systemd unit

[ec2-user@ip-192-168-65-212 ~]$ systemctl cat kubelet
# /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service iptables-restore.service
Requires=docker.service

[Service]
ExecStartPre=/sbin/iptables -P FORWARD ACCEPT -w 5
ExecStart=/usr/bin/kubelet --cloud-provider aws \
    --config /etc/kubernetes/kubelet/kubelet-config.json \
    --kubeconfig /var/lib/kubelet/kubeconfig \
    --container-runtime docker \
    --network-plugin cni $KUBELET_ARGS $KUBELET_EXTRA_ARGS
...
... 
...

登入 EKS worker node 檢視 kubelet 確認預設 CNI plugin 參數 [2]：
--cni-bin-dir="/opt/cni/bin"：kubelet 啟用時會查看此目錄
--cni-conf-dir="/etc/cni/net.d"

[ec2-user@ip-192-168-65-212 ~]$ journalctl -u kubelet | grep -e "--cni"
Sep 19 09:46:56 ip-192-168-65-212.eu-west-1.compute.internal kubelet[3347]: I0919 09:46:56.765003    3347 flags.go:59] FLAG: --cni-bin-dir="/opt/cni/bin"
Sep 19 09:46:56 ip-192-168-65-212.eu-west-1.compute.internal kubelet[3347]: I0919 09:46:56.765008    3347 flags.go:59] FLAG: --cni-cache-dir="/var/lib/cni/cache"
Sep 19 09:46:56 ip-192-168-65-212.eu-west-1.compute.internal kubelet[3347]: I0919 09:46:56.765014    3347 flags.go:59] FLAG: --cni-conf-dir="/etc/cni/net.d"

不免俗的來驗證一下，我們可以直接透過 EC2 啟用 Amazon EKS optimized Amazon Linux AMI 來比對：

透過 EC2 啟用 EKS AMI 的 node

使用 AMI：amazon/amazon-eks-node-1.22-v20220824

[ec2-user@ip-172-31-44-117 ~]$ ls /opt/cni/bin
bandwidth  bridge  dhcp  firewall  flannel  host-device  host-local  ipvlan  loopback  macvlan  portmap  ptp  sbr  static  tuning  vlan

[ec2-user@ip-172-31-44-117 ~]$ ls -al /etc/cni/net.d/
ls: cannot access /etc/cni/net.d/: No such file or directory

已經有 VPC CNI plugin Pod 的 node

[ec2-user@ip-192-168-65-212 ~]$ ls /opt/cni/bin
aws-cni  aws-cni-support.sh  bandwidth  bridge  dhcp  egress-v4-cni  firewall  flannel  host-device  host-local  ipvlan  loopback  macvlan  portmap  ptp  sbr  static  tuning  vlan

[ec2-user@ip-192-168-65-212 ~]$ sudo cat /etc/cni/net.d/10-aws.conflist
{
  "cniVersion": "0.3.1",
  "name": "aws-cni",
  "plugins": [
    {
      "name": "aws-cni",
      "type": "aws-cni",
      "vethPrefix": "eni",
      "mtu": "9001",
      "pluginLogFile": "/var/log/aws-routed-eni/plugin.log",
      "pluginLogLevel": "DEBUG"
    },
    {
      "name": "egress-v4-cni",
      "type": "egress-v4-cni",
      "mtu": 9001,
      "enabled": "false",
      "nodeIP": "192.168.65.212",
      "ipam": {
         "type": "host-local",
         "ranges": [[{"subnet": "169.254.172.0/22"}]],
         "routes": [{"dst": "0.0.0.0/0"}],
         "dataDir": "/run/cni/v6pd/egress-v4-ipam"
      },
      "pluginLogFile": "/var/log/aws-routed-eni/egress-v4-plugin.log",
      "pluginLogLevel": "DEBUG"
    },
    {
      "type": "portmap",
      "capabilities": {"portMappings": true},
      "snat": true
    }
  ]
}

/opt/cni/bin 路徑正如同 CNI 參數名稱為 CNI binary 路徑。預設 EKS 在製作 worker node AMI 時，透過 script [3] 安裝常見 CNI plugin binary [4]。上述比對可以了解，在 aws-node Pod 尚未執行之前，aws-cni binary 及設定檔 /etc/cni/net.d/10-aws.conflist 皆尚未被安裝或建立。

接續，我們也可以透過 kubectl 檢視 aws-node Pod logs，可以看到由 Amazon VPC CNI plugin 預設使用的 initContainers amazon-k8s-cni-init 安裝 loopback 、portmap、bandwidth 及 aws-cni-support.sh plugin。init container 執行結束後，aws-node 也有執行複製設定檔及 binary。以下為 logs：

$ kubectl -n kube-system logs aws-node-5c6w5 --all-containers --timestamps
2022-09-19T09:48:57.333672597Z Copying CNI plugin binaries ...
2022-09-19T09:48:57.333810096Z + PLUGIN_BINS='loopback portmap bandwidth aws-cni-support.sh'
2022-09-19T09:48:57.333831182Z + for b in '$PLUGIN_BINS'
2022-09-19T09:48:57.333835571Z + '[' '!' -f loopback ']'
2022-09-19T09:48:57.333838091Z + for b in '$PLUGIN_BINS'
2022-09-19T09:48:57.333840648Z + '[' '!' -f portmap ']'
2022-09-19T09:48:57.333843085Z + for b in '$PLUGIN_BINS'
2022-09-19T09:48:57.333845490Z + '[' '!' -f bandwidth ']'
2022-09-19T09:48:57.333847874Z + for b in '$PLUGIN_BINS'
2022-09-19T09:48:57.333850503Z + '[' '!' -f aws-cni-support.sh ']'
2022-09-19T09:48:57.333908878Z + HOST_CNI_BIN_PATH=/host/opt/cni/bin
2022-09-19T09:48:57.333911360Z + echo 'Copying CNI plugin binaries ... '
2022-09-19T09:48:57.333914044Z + for b in '$PLUGIN_BINS'
2022-09-19T09:48:57.333916695Z + install loopback /host/opt/cni/bin
2022-09-19T09:48:57.358632542Z + for b in '$PLUGIN_BINS'
2022-09-19T09:48:57.358667213Z + install portmap /host/opt/cni/bin
2022-09-19T09:48:57.371180218Z + for b in '$PLUGIN_BINS'
2022-09-19T09:48:57.371397188Z + install bandwidth /host/opt/cni/bin
2022-09-19T09:48:57.386902686Z + for b in '$PLUGIN_BINS'
2022-09-19T09:48:57.387089749Z + install aws-cni-support.sh /host/opt/cni/bin
2022-09-19T09:48:57.389252423Z + echo 'Configure rp_filter loose... '
2022-09-19T09:48:57.389584940Z Configure rp_filter loose...
2022-09-19T09:48:57.389945375Z ++ get_metadata local-ipv4
2022-09-19T09:48:57.391264947Z +++ curl -X PUT http://169.254.169.254/latest/api/token -H 'X-aws-ec2-metadata-token-ttl-seconds: 60'
2022-09-19T09:48:57.473320665Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2022-09-19T09:48:57.473705220Z                                  Dload  Upload   Total   Spent    Left  Speed
100    56  100    56    0     0  28000      0 --:--:-- --:--:-- --:--:-- 56000
2022-09-19T09:48:57.478526895Z ++ TOKEN=AQAEAJKJMlzdfIbRn8BQh4U5I2i0dMhKrW4DxRx64XuuP_g4rPzuOw==
2022-09-19T09:48:57.478907025Z ++ attempts=60
2022-09-19T09:48:57.478914505Z ++ false
2022-09-19T09:48:57.478917627Z ++ '[' 1 -gt 0 ']'
2022-09-19T09:48:57.478920837Z ++ '[' 60 -eq 0 ']'
2022-09-19T09:48:57.479064447Z +++ curl -H 'X-aws-ec2-metadata-token: AQAEAJKJMlzdfIbRn8BQh4U5I2i0dMhKrW4DxRx64XuuP_g4rPzuOw==' http://169.254.169.254/latest/meta-data/local-ipv4
2022-09-19T09:48:57.486678298Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2022-09-19T09:48:57.486692175Z                                  Dload  Upload   Total   Spent    Left  Speed
100    14  100    14    0     0  14000      0 --:--:-- --:--:-- --:--:-- 14000
2022-09-19T09:48:57.488090438Z ++ meta=192.168.65.212
2022-09-19T09:48:57.488210992Z ++ '[' 0 -gt 0 ']'
2022-09-19T09:48:57.488331224Z ++ '[' 0 -gt 0 ']'
2022-09-19T09:48:57.488430469Z ++ echo 192.168.65.212
2022-09-19T09:48:57.488788529Z + HOST_IP=192.168.65.212
2022-09-19T09:48:57.489299361Z ++ get_metadata mac
2022-09-19T09:48:57.489716539Z +++ curl -X PUT http://169.254.169.254/latest/api/token -H 'X-aws-ec2-metadata-token-ttl-seconds: 60'
2022-09-19T09:48:57.497186993Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2022-09-19T09:48:57.497200745Z                                  Dload  Upload   Total   Spent    Left  Speed
100    56  100    56    0     0  56000      0 --:--:-- --:--:-- --:--:-- 56000
2022-09-19T09:48:57.499248303Z ++ TOKEN=AQAEAJKJMlxeFjZlUL_0INYoCXkf7UWmVm4nIKV6nnDeG_VvdZ9-Ig==
2022-09-19T09:48:57.499358341Z ++ attempts=60
2022-09-19T09:48:57.499440488Z ++ false
2022-09-19T09:48:57.499594430Z ++ '[' 1 -gt 0 ']'
2022-09-19T09:48:57.499599969Z ++ '[' 60 -eq 0 ']'
2022-09-19T09:48:57.500015110Z +++ curl -H 'X-aws-ec2-metadata-token: AQAEAJKJMlxeFjZlUL_0INYoCXkf7UWmVm4nIKV6nnDeG_VvdZ9-Ig==' http://169.254.169.254/latest/meta-data/mac
2022-09-19T09:48:57.511676682Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2022-09-19T09:48:57.511709736Z                                  Dload  Upload   Total   Spent    Left  Speed
100    17  100    17    0     0  17000      0 --:--:-- --:--:-- --:--:-- 17000
2022-09-19T09:48:57.515247164Z ++ meta=06:1b:3e:40:af:fd
2022-09-19T09:48:57.515648955Z ++ '[' 0 -gt 0 ']'
2022-09-19T09:48:57.515656468Z ++ '[' 0 -gt 0 ']'
2022-09-19T09:48:57.515659528Z ++ echo 06:1b:3e:40:af:fd
2022-09-19T09:48:57.515827055Z + PRIMARY_MAC=06:1b:3e:40:af:fd
2022-09-19T09:48:57.516696407Z ++ grep -F 'link/ether 06:1b:3e:40:af:fd'
2022-09-19T09:48:57.516908826Z ++ awk '-F[ :]+' '{print $2}'
2022-09-19T09:48:57.517104498Z ++ ip -o link show
2022-09-19T09:48:57.529715651Z + PRIMARY_IF=eth0
2022-09-19T09:48:57.529881853Z + sysctl -w net.ipv4.conf.eth0.rp_filter=2
2022-09-19T09:48:57.549511061Z net.ipv4.conf.eth0.rp_filter = 2
2022-09-19T09:48:57.550035252Z + cat /proc/sys/net/ipv4/conf/eth0/rp_filter
2022-09-19T09:48:57.552013538Z 2
2022-09-19T09:48:57.552296409Z + '[' false == true ']'
2022-09-19T09:48:57.552638323Z + sysctl -e -w net.ipv4.tcp_early_demux=1
2022-09-19T09:48:57.557945154Z net.ipv4.tcp_early_demux = 1
2022-09-19T09:48:57.559523168Z + '[' false == true ']'
2022-09-19T09:48:57.560537081Z + echo 'CNI init container done'
2022-09-19T09:48:57.560663349Z CNI init container done
2022-09-19T09:48:58.240326675Z {"level":"info","ts":"2022-09-19T09:48:58.237Z","caller":"entrypoint.sh","msg":"Validating env variables ..."}
2022-09-19T09:48:58.241282124Z {"level":"info","ts":"2022-09-19T09:48:58.240Z","caller":"entrypoint.sh","msg":"Install CNI binaries.."}
2022-09-19T09:48:58.316896574Z {"level":"info","ts":"2022-09-19T09:48:58.316Z","caller":"entrypoint.sh","msg":"Starting IPAM daemon in the background ... "}
2022-09-19T09:48:58.318121492Z {"level":"info","ts":"2022-09-19T09:48:58.317Z","caller":"entrypoint.sh","msg":"Checking for IPAM connectivity ... "}
2022-09-19T09:49:00.401119692Z {"level":"info","ts":"2022-09-19T09:49:00.397Z","caller":"entrypoint.sh","msg":"Retrying waiting for IPAM-D"}
2022-09-19T09:49:00.441859850Z {"level":"info","ts":"2022-09-19T09:49:00.441Z","caller":"entrypoint.sh","msg":"Copying config file ... "}
2022-09-19T09:49:00.450179414Z {"level":"info","ts":"2022-09-19T09:49:00.449Z","caller":"entrypoint.sh","msg":"Successfully copied CNI plugin binary and config file."}
2022-09-19T09:49:00.451045960Z {"level":"info","ts":"2022-09-19T09:49:00.450Z","caller":"entrypoint.sh","msg":"Foregrounding IPAM daemon ..."}

在 Amazon VPC CNI plugin entrypoint [5]，可以確認其 aws-cni binary 為自身啟用時會複製至本地端，同時也能查看到 volumeMount mount 相應目錄。

$ kubectl -n kube-system get ds aws-node -o yaml
...
...
		image: 602401143452.dkr.ecr.eu-west-1.amazonaws.com/amazon-k8s-cni:v1.10.1-eksbuild.1
...
...
        volumeMounts:
        - mountPath: /host/opt/cni/bin
          name: cni-bin-dir
        - mountPath: /host/etc/cni/net.d
          name: cni-net-dir
        - mountPath: /host/var/log/aws-routed-eni
          name: log-dir
        - mountPath: /var/run/aws-node
          name: run-dir
        - mountPath: /var/run/dockershim.sock
          name: dockershim
        - mountPath: /run/xtables.lock
          name: xtables-lock
...
...
      volumes:
      - hostPath:
          path: /opt/cni/bin
          type: ""
        name: cni-bin-dir
      - hostPath:
          path: /etc/cni/net.d
          type: ""
        name: cni-net-dir
      - hostPath:
          path: /var/run/dockershim.sock
          type: ""
        name: dockershim
      - hostPath:
          path: /run/xtables.lock
          type: ""
        name: xtables-lock
      - hostPath:
          path: /var/log/aws-routed-eni
          type: DirectoryOrCreate
        name: log-dir
      - hostPath:
          path: /var/run/aws-node
          type: DirectoryOrCreate
        name: run-dir

那為什麼需要這樣設定一個 entrypoint script 安裝 CNI Binary 呢？

VPC CNI plugin 主要有兩個元件：

CNI Plugin：主要整合 host 及 Pod 網路設定。
ipamd：長期執行於節點本地端的 IP Address Management (IPAM) daemon 主要負責以下兩件事：
維持一個可用的 IP 地址 warm-pool
指派 IP 地址給 Pod

在 Amazon VPC CNI plugin entrypoint [5] 的註解裡解釋：一般來說，kubelet 在查看 well-known directory（這邊指的是 /opt/cni/bin 及 /etc/cni/net.d）時，就會將 CNI plugin 視為 Ready 狀態。然而因為 VPC CNI plugin 提供了上述兩個元件，希望在成功啟動 IPAM daemon 可以確保連接上 Kubernetes 及本地端 EC2 metadata service，並將 aws-cni binary 複製到 well-known directory。

API server

透過 CloudWatch Logs insight syntax 檢視 kube-apiserver-audit logs，並且過濾 verb 為 create。

filter @logStream like /^kube-apiserver-audit/
 | fields @timestamp, @message
 | sort @timestamp asc
 | filter objectRef.name == 'aws-node' AND objectRef.resource == 'daemonsets' AND verb == 'create'
 | limit 10000

能發現 username 為 eks: cluster-bootstrap 透過 kubectl 方式部署 create 了 aws-node DaemonSet。

userAgent: kubectl/v1.22.12 (linux/amd64) kubernetes/dade57b
user.username: eks: cluster-bootstrap

總結

由上述驗證後，我們可以了解 EKS 在 cluster bootstrap 過程中透過 kubectl 方式部署 VPC CNI plugin。而在 worker node 層級，設定了 kubelet CNI 相關參數，並由 VPC CNI plugin 複製 binary 檔案至主機上使用。

上述資訊透過 EKS 所提供 Logs 來驗證上游 Kubernetes 運作原理，倘若上述內文有所錯誤，隨時可以留言或是私訊我。

參考文件

Pod networking in Amazon EKS using the Amazon VPC CNI plugin for Kubernetes - https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html
Network Plugins | Kubernetes 1.22 - https://v1-22.docs.kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/#cni
https://github.com/awslabs/amazon-eks-ami/blob/master/scripts/install-worker.sh#L240
The Container Network Interface - https://www.cni.dev/plugins/current/
https://github.com/aws/amazon-vpc-cni-k8s/blob/master/scripts/entrypoint.sh

為什麼 EKS worker node 可以自動加入 EKS cluster（二）？

Mao-Lin Wang — Mon, 19 Sep 2022 14:55:28 GMT

接續前一篇，我們了解了 EKS node group 定義，並知道了 EKS worker node 使用了 EKS optimized Amazon Linux AMI 內預先設置好的 bootstrap 設定 container runtime、kubelet 等設定。

Kubernetes 為確保 Control Plane 及 worker node 溝通安全性，於 Kubernetes 1.4 導入了 certificate request 及 signing API 機制 [1]。當然 EKS worker node 也皆是基於原生 Kubernetes 架構實現，EKS worker node 使用 kubelet TLS bootstrapping 流程與 API server 進行溝通。

故本文將繼續 EKS worker node 如何讓 kubelet 自動化完成 TLS bootstrap 流程，自動加入 EKS cluster。

kubelet bootstrap 初始化步驟

根據 Kubernetes TLS bootstrapping [2] 文件，kubelet 在 bootstrap 初始化步驟如下：

kubelet 啟動。
kubelet 查看是否有相應 kubeconfig 設定檔。
kueblet 檢視是否有對應 bootstrap-kubeconfig 設定檔。
基於 bootstrap 設定檔，取得 API server endpoint 及有限制權限 token。
透過上述 token 認證（authenticates）與 API server 建立連線。
kubelet 使用建立有限制權限 credentials 建立並取得 Certificate Signing Requests（CSR） [3]。
kubelet 建立 CSR，並將 singerName 設定為 kubernetes.io/kube-apiserver-client-kubelet。
CSR 被核准。可以透過以下兩種方式：

kube-controller-manager 自動核准 CSR
透過外部流程或是人為核准，透過 kubectl 或是 Kubernetes API 方式核准 CSR

kubelet 所需要的憑證（certificate）被建立。
憑證（certificate）被發送給 kubelet。
kubelet 取得該憑證（certificate）。
kubelet 建立 kubeconfig，其中包含金鑰和已簽名的憑證。
kubelet 開始正常執行
（選擇性設定）kubelet 在憑證接近於過期時間將會自動請求更新憑證
更新憑證會被核准並發證。

驗證 EKS 設定

在設定上，為達到自動化核准發證流程，我們必須設定以下元件，並需要 Kubernetes Certificate Authority (CA)：

kube-apiserver
kube-controller-manager
kubelet

kubelet

首先，透過 systemctl 命令查看 kubelet systemd unit 設定檔：

[ec2-user@ip-192-168-65-212 ~]$ systemctl cat kubelet
# /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service iptables-restore.service
Requires=docker.service

[Service]
ExecStartPre=/sbin/iptables -P FORWARD ACCEPT -w 5
ExecStart=/usr/bin/kubelet --cloud-provider aws \
    --config /etc/kubernetes/kubelet/kubelet-config.json \
    --kubeconfig /var/lib/kubelet/kubeconfig \
    --container-runtime docker \
    --network-plugin cni $KUBELET_ARGS $KUBELET_EXTRA_ARGS

Restart=always
RestartSec=5
KillMode=process

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/kubelet.service.d/10-kubelet-args.conf
[Service]
Environment='KUBELET_ARGS=--node-ip=192.168.65.212 --pod-infra-container-image=602401143452.dkr.ecr.eu-west-1.amazonaws.com/eks/pause:3.5 --v=2'
# /etc/systemd/system/kubelet.service.d/30-kubelet-extra-args.conf
[Service]
Environment='KUBELET_EXTRA_ARGS=--node-labels=eks.amazonaws.com/sourceLaunchTemplateVersion=1,alpha.eksctl.io/nodegroup-name=ng1-public-ssh,alpha.eksctl.io/cluster-name=ironman-2022,eks.amazonaws.com/nodegroup-image=ami-0ec9e1727a24fb788,eks.a

可以觀察到設定了：

kubeconfig：/var/lib/kubelet/kubeconfig 使用了 kubelet username 並透過 aws-iam-authenticator 取得 cluster token。

[ec2-user@ip-192-168-65-212 ~]$ cat /var/lib/kubelet/kubeconfig
apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority: /etc/kubernetes/pki/ca.crt
    server: https://A8E7A39CAEBEF6AA9250DFA9366FDFA2.gr7.eu-west-1.eks.amazonaws.com
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubelet
  name: kubelet
current-context: kubelet
users:
- name: kubelet
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: /usr/bin/aws-iam-authenticator
      args:
        - "token"
        - "-i"
        - "ironman-2022"
        - --region
        - "eu-west-1"

此 kubeconfig 也再次驗證了 EKS optimized Amazon Linux AMI 預設安裝了 aws-iam-authenticator [4] 以提供 EKS worker node 上 kubelet 進行驗證（authenticate）流程。

kubelet config：/etc/kubernetes/kubelet/kubelet-config.json

[ec2-user@ip-192-168-90-19 ~]$ cat /etc/kubernetes/kubelet/kubelet-config.json
{
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  "address": "0.0.0.0",
  "authentication": {
    "anonymous": {
      "enabled": false
    },
    "webhook": {
      "cacheTTL": "2m0s",
      "enabled": true
    },
    "x509": {
      "clientCAFile": "/etc/kubernetes/pki/ca.crt"
    }
  },
  "authorization": {
    "mode": "Webhook",
    "webhook": {
      "cacheAuthorizedTTL": "5m0s",
      "cacheUnauthorizedTTL": "30s"
    }
  },
  "clusterDomain": "cluster.local",
  "hairpinMode": "hairpin-veth",
  "readOnlyPort": 0,
  "cgroupDriver": "cgroupfs",
  "cgroupRoot": "/",
  "featureGates": {
    "RotateKubeletServerCertificate": true
  },
  "protectKernelDefaults": true,
  "serializeImagePulls": false,
  "serverTLSBootstrap": true,
  "tlsCipherSuites": [
    "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
    "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
    "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305",
    "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
    "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305",
    "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
    "TLS_RSA_WITH_AES_256_GCM_SHA384",
    "TLS_RSA_WITH_AES_128_GCM_SHA256"
  ],
  "clusterDNS": [
    "10.100.0.10"
  ],
  "evictionHard": {
    "memory.available": "100Mi",
    "nodefs.available": "10%",
    "nodefs.inodesFree": "5%"
  },
  "kubeReserved": {
    "cpu": "70m",
    "ephemeral-storage": "1Gi",
    "memory": "574Mi"
  }
}

在這些 kubelet flag [5] 其中我們關注與 TLS Bootstrap 有關 flag：

serverTLSBootstrap: true：此將允許來自 certificates.k8s.io API
給予 kubelet 使用 kubelet serving certificates。然而此為一個已知安全性限制 [6]，此類型 certificate 無法透過 kube-controller-manager - kubernetes.io/kubelet-serving 自動化核准 CSR，則需要仰賴使用者或是第三方 controller。
featureGates.RotateKubeletServerCertificate: ture：kubelet 在 bootstrapping 自身 client credentials 後請求 serving certificate 並自動替換憑證。

kube-apiserver

The kube-apiserver 需要以下三點條件才能啟用 TLS bootstrapping：

設別 client certificate 的 CA
驗證（authenticate） bootstrapping kubelet 並設定於 system: bootstrappers group
授權（authorize） bootstrapping kubelet 建立 CSR

其中驗證流程，由上述提及透過 EKS optimized Amazon Linux AMI 所定義 bootstrap.sh script 定義 kubeconfig 使用 aws-iam-authenticator 。而 kubelet 在經由 API server 認證後，由 RBAC system: node role 和 Node authorizer [7] 將允許節點建立和讀取 CSRs。因此我們可以在 EKS 預設 eks: node-bootstrapper role 上檢視：

給予 kubelet 權限提交 CSR

$ kubectl get clusterrole eks:node-bootstrapper -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"annotations":{},"labels":{"eks.amazonaws.com/component":"node"},"name":"eks:node-bootstrapper"},"rules":[{"apiGroups":["certificates.k8s.io"],"resources":["certificatesigningrequests/selfnodeserver"],"verbs":["create"]}]}
  creationTimestamp: "2022-09-14T09:46:17Z"
  labels:
    eks.amazonaws.com/component: node
  name: eks:node-bootstrapper
  resourceVersion: "283"
  uid: eb23d8fe-dfdf-4f01-aba7-72ca32b52ad7
rules:
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests/selfnodeserver
  verbs:
  - create

cluster role eks: node-bootstrapper 綁定 system: bootstrappers 及 system: nodes group

$ kubectl get clusterrolebindings eks:node-bootstrapper -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"labels":{"eks.amazonaws.com/component":"node"},"name":"eks:node-bootstrapper"},"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole","name":"eks:node-bootstrapper"},"subjects":[{"apiGroup":"rbac.authorization.k8s.io","kind":"Group","name":"system:bootstrappers"},{"apiGroup":"rbac.authorization.k8s.io","kind":"Group","name":"system:nodes"}]}
  creationTimestamp: "2022-09-14T09:46:16Z"
  labels:
    eks.amazonaws.com/component: node
  name: eks:node-bootstrapper
  resourceVersion: "282"
  uid: 867196bc-b84a-410d-8d4b-cbf52d840108
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: eks:node-bootstrapper
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes

透過 CloudWatch Logs insight syntax 檢視 kube-apiserver logs 一樣可以查看到 --authorization-mode 使用了 Node 及 RBAC 兩種授權模式。

filter @logStream not like /^kube-apiserver-audit/
 | filter @logStream like /^kube-apiserver-/
 | fields @timestamp, @message
 | sort @timestamp asc
 | filter @message like "--authorization-mode"
 | limit 10000

I0914 09:46:06.295709 10 flags.go:59] FLAG: --authorization-mode="[Node,RBAC]"

kube-controller-manager

EKS 使用了 kubelet serving certificates 方式，此方式僅能透過第三方 controller 方式核准 CSR。此部分，我們也可以再次使用 CloudWatch Logs insight syntax 語法檢視此 flag 可以觀察到停用原生 Kubernetes csrsigning controller。

filter @logStream not like /^kube-controller-manager/
 | fields @timestamp, @message
 | sort @timestamp asc
 | filter @message like "--controller"
 | limit 10000

I0914 09:51:40.684692 11 flags.go:59] FLAG: --controllers="[*,-csrsigning]"

實際啟用 EKS 節點

我們透過 eksctl scale 命令 scale up 一個 node 來查看檢視相應元件設定，及 TLS bootstrap 流程：

$ eksctl scale ng --nodes=3 --name=ng1-public-ssh --cluster=ironman-2022
2022-09-19 09:44:52 [ℹ]  scaling nodegroup "ng1-public-ssh" in cluster ironman-2022
2022-09-19 09:44:53 [ℹ]  waiting for scaling of nodegroup "ng1-public-ssh" to complete
2022-09-19 09:46:31 [ℹ]  nodegroup successfully scaled

透過 kubectl get CSR 可以先觀察到由 node system: node: ip-192-168-65-212.eu-west-1.compute.internal 作為 requestor 發起 CSR。約 11 秒後，此 CSR 被 Approved。接續，CSR 被 Issued 給 node。


$ kubectl get node
NAME                                           STATUS     ROLES    AGE     VERSION
...
ip-192-168-65-212.eu-west-1.compute.internal   NotReady      0s      v1.22.12-eks-ba74326

$ kubectl get csr
NAME        AGE   SIGNERNAME                      REQUESTOR                                                  REQUESTEDDURATION   CONDITION
csr-kdkll   11s   kubernetes.io/kubelet-serving   system:node:ip-192-168-65-212.eu-west-1.compute.internal                 Approved

$ kubectl get csr
NAME        AGE   SIGNERNAME                      REQUESTOR                                                  REQUESTEDDURATION   CONDITION
csr-kdkll   16s   kubernetes.io/kubelet-serving   system:node:ip-192-168-65-212.eu-west-1.compute.internal                 Approved,Issued

$ kubectl get node
NAME                                                STATUS   ROLES    AGE     VERSION
node/ip-192-168-18-254.eu-west-1.compute.internal   Ready       4d17h   v1.22.12-eks-ba74326
node/ip-192-168-40-16.eu-west-1.compute.internal    Ready       19h     v1.22.12-eks-ba74326
node/ip-192-168-65-212.eu-west-1.compute.internal   Ready       43s     v1.22.12-eks-ba74326

$ kubectl describe csr csr-kdkll
Name:               csr-kdkll
Labels:             
Annotations:        
CreationTimestamp:  Mon, 19 Sep 2022 09:46:59 +0000
Requesting User:    system:node:ip-192-168-65-212.eu-west-1.compute.internal
Signer:             kubernetes.io/kubelet-serving
Status:             Approved,Issued
Subject:
  Common Name:    system:node:ip-192-168-65-212.eu-west-1.compute.internal
  Serial Number:
  Organization:   system:nodes
Subject Alternative Names:
         DNS Names:     ec2-52-211-162-59.eu-west-1.compute.amazonaws.com
                        ip-192-168-65-212.eu-west-1.compute.internal
         IP Addresses:  192.168.65.212
                        52.211.162.59
Events:

由於並未能查看到 EKS 是否有使用第三方 controller 進行核准（approve）CSR。我們透過 CloudWatch Logs insight syntax 語法檢視 kube-apiserver-audit 日誌內記錄 CSR csr-kdkll 流程變化。

filter @logStream like /^kube-apiserver-audit/
 | fields @timestamp, @message
 | sort @timestamp asc
 | filter @message like 'csr-kdkll'
 | limit 10000

2022-09-19 09:46:59 UTC+0：kubelet/v1.22.12 (linux/amd64) kubernetes/1fc8914 使用了建立了 CSR csr-kdkll
user.username：`system: node: ip-192-168-65-212.eu-west-1.compute.internal
user.uid：aws-iam-authenticator:111111111111: AROAYFMQSNSE3QYOZUIO6
responseObject.spec.signerName：kubernetes.io/kubelet-serving
responseObject.spec.usages
digital signature
key encipherment
server auth
2022-09-19 09:46:59 UTC+0：eks: certificate-controller 更新 CSR 並核准（approve）CSR
user.username：eks: certificate-controller
responseObject.status.conditions.0.message：Auto approving self kubelet server certificate after SubjectAccessReview.
responseObject.status.conditions.0.reason：AutoApproved

由上述可以了解此 CSR 符合 Kubernetes signers signerName kubernetes.io/kubelet-serving 而不被 kube-controller-manager 自動核准，此 CSR 則是由 EKS eks: certificate-controller 來自動核准。

而經由 CSR 核准並發證於 node 上。我們也可以於 /var/lib/kubelet/pki/ 目錄查看到 kubelet-server-current.pem 憑證。

[ec2-user@ip-192-168-65-212 ~]$ journalctl -u kubelet | grep "certificate signing request"
Sep 19 09:47:00 ip-192-168-65-212.eu-west-1.compute.internal kubelet[3347]: I0919 09:47:00.509710    3347 csr.go:262] certificate signing request csr-kdkll is approved, waiting to be issued
Sep 19 09:47:14 ip-192-168-65-212.eu-west-1.compute.internal kubelet[3347]: I0919 09:47:14.707915    3347 csr.go:258] certificate signing request csr-kdkll is issued

[ec2-user@ip-192-168-65-212 ~]$ sudo ls -al /var/lib/kubelet/pki/
total 4
drwxr-xr-x 2 root root   86 Sep 14 16:39 .
drwxr-xr-x 8 root root  182 Sep 14 16:39 ..
-rw------- 1 root root 1370 Sep 14 16:39 kubelet-server-2022-09-14-16-39-07.pem
lrwxrwxrwx 1 root root   59 Sep 14 16:39 kubelet-server-current.pem -> /var/lib/kubelet/pki/kubelet-server-2022-09-14-16-39-07.pem

總結

比對原生 Kubernetes 文件及 EKS 上 kube-apiserver、 kube-controller-manager 及 kubelet 元件設定後，我們可以確定 EKS 於 Control Plane 端設置了 eks: certificate-controller 提供自動核准（Auto-Aprroved）CSR 並 issue CSR 給予 kubelet 使用。在取得 CSR 後，node 則可以順利加入 EKS cluster。

上述資訊透過 EKS 所提供 Logs 來驗證上游 Kubernetes 運作原理，倘若上述內文有所錯誤，隨時可以留言或是私訊我。

參考文件

Add proposal for kubelet TLS bootstrap - https://github.com/kubernetes/kubernetes/pull/20439/files
TLS bootstrapping - https://kubernetes.io/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/
Certificate Signing Requests - https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/
AWS IAM Authenticator for Kubernetes - https://github.com/kubernetes-sigs/aws-iam-authenticator
kubelet - https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/
design: reduce scope of node on node object w.r.t ip #1982 - https://github.com/kubernetes/community/pull/1982
Using Node Authorization - https://kubernetes.io/docs/reference/access-authn-authz/node/
https://kubernetes.io/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/#certificate-rotation

為什麼 EKS worker node 可以自動加入 EKS cluster（一）

Mao-Lin Wang — Sun, 18 Sep 2022 15:40:43 GMT

如果是一個自建的 Kubernetes cluster，我們會需要使用 Certificate Authority 憑證給予 Kubernetes Components [1] 所使用，如 Kubernetes The Hard Way [2] 或是以下 kubeadm join [3] 命令則可以使 Kubernetes node 加入 cluster：

$ kubeadm join --discovery-token abcdef.1234567890abcdef --discovery-token-ca-cert-hash sha256:1234..cdef 1.2.3.4:6443

在 EKS 環境中，nodegroup 一詞代表了 Kubernetes nodes 集合，而我們也可以使用 eksctl scale [4] 命令 scale up Nodes。

$ eksctl scale ng --nodes=3 --nodes-max=10 --name=ng1-public-ssh --cluster=ironman-2022
2022-09-18 14:14:18 [ℹ]  scaling nodegroup "ng1-public-ssh" in cluster ironman-2022
2022-09-18 14:14:18 [ℹ]  waiting for scaling of nodegroup "ng1-public-ssh" to complete
2022-09-18 14:14:48 [ℹ]  nodegroup successfully scaled

而在短短幾分鐘內，我們並無需要手動設定 TLS 憑證，EKS node 則可以順利自動加入集群。

$ kubectl get node
NAME                                           STATUS   ROLES    AGE     VERSION
...
ip-192-168-40-16.eu-west-1.compute.internal    Ready       34s     v1.22.12-eks-ba74326
...

故將探討「為什麼 EKS 節點可以自動加入集群」，希望理解 EKS node 預設了哪些設置允許 node 可以自動加入 cluster。本文將著重於：

何謂 EKS node group？
EKS AMI bootstrap script 預設了哪些設定？

EKS node group

一個完整的 Kubernetes Components [1] 可以分成為 Control Plane 端及 Node 端。其中在 EKS cluster 環境中，AWS 服務整合了 EC2、Auto Scaling groups 等服務的 worker node 資源稱之為 nodegroup（節點組）。根據 EKS Node 文件，又可以分成 Managed node groups（託管節點組） [6] 及 Self-managed nodes（自行管理節點組） [7]。

於第一天的 eksctl ClusterConfig 文件定義使用了 Managed node groups ng1-public-ssh。

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: ironman-2022
  region: eu-west-1

managedNodeGroups:
  - name: "ng1-public-ssh"
    desiredCapacity: 2
    ssh:
      # Enable ssh access (via the admin container)
      allow: true
      publicKeyName: "ironman-2022"
    iam:
      withAddonPolicies:
        ebs: true
        fsx: true
        efs: true
        awsLoadBalancerController: true
        autoScaler: true

iam:
  withOIDC: true

cloudWatch:
  clusterLogging:
    enableTypes: ["*"]

在建立完環境後，我們會有名為 ng1-public-ssh 的 Managed node groups：

$ eksctl get ng --cluster=ironman-2022
CLUSTER NODEGROUP       STATUS  CREATED                 MIN SIZE        MAX SIZE        DESIRED CAPACITY        INSTANCE TYPE   IMAGE ID        ASG NAME                    TYPE
ironman-2022 ng1-public-ssh  ACTIVE  2022-09-14T09:54:36Z    0               10              3                       m5.large        AL2_x86_64      eks-ng1-public-ssh-62c19db5-f965-bdb7-373a-147e04d9f124      managed

因此我們也可以透過 aws eks describe-nodegroup [8] 命令查看 nodegroup 資源資訊：

$ aws eks describe-nodegroup --cluster-name ironman-2022 --nodegroup-name ng1-public-ssh
{
    "nodegroup": {
        "nodegroupName": "ng1-public-ssh",
        "nodegroupArn": "arn:aws:eks:eu-west-1:111111111111:nodegroup/ironman-2022/ng1-public-ssh/62c19db5-f965-bdb7-373a-147e04d9f124",
        "clusterName": "ironman-2022",
        "version": "1.22",
        "releaseVersion": "1.22.12-20220824",
        "createdAt": "2022-09-14T09:54:36.211000+00:00",
        "modifiedAt": "2022-09-18T13:45:07.322000+00:00",
        "status": "ACTIVE",
        "capacityType": "ON_DEMAND",
        "scalingConfig": {
            "minSize": 0,
            "maxSize": 2,
            "desiredSize": 2
        },
        "instanceTypes": [
            "m5.large"
        ],
        "subnets": [
            "subnet-0e863f9fbcda592a3",
            "subnet-00ebeb2e8903fb3f9",
            "subnet-02d98be342d8ab2a7"
        ],
        "amiType": "AL2_x86_64",
        "nodeRole": "arn:aws:iam::111111111111:role/eksctl-ironman-2022-nodegroup-ng1-publ-NodeInstanceRole-HN27OZ18JS6U",
        "labels": {
            "alpha.eksctl.io/cluster-name": "ironman-2022",
            "alpha.eksctl.io/nodegroup-name": "ng1-public-ssh"
        },
        "resources": {
            "autoScalingGroups": [
                {
                    "name": "eks-ng1-public-ssh-62c19db5-f965-bdb7-373a-147e04d9f124"
                }
            ]
        },
        "health": {
            "issues": []
        },
        "updateConfig": {
            "maxUnavailable": 1
        },
        "launchTemplate": {
            "name": "eksctl-ironman-2022-nodegroup-ng1-public-ssh",
            "version": "1",
            "id": "lt-000b4417a7baebbf8"
        },
        "tags": {
            "aws:cloudformation:stack-name": "eksctl-ironman-2022-nodegroup-ng1-public-ssh",
            "alpha.eksctl.io/cluster-name": "ironman-2022",
            "alpha.eksctl.io/nodegroup-name": "ng1-public-ssh",
            "aws:cloudformation:stack-id": "arn:aws:cloudformation:eu-west-1:111111111111:stack/eksctl-ironman-2022-nodegroup-ng1-public-ssh/2b5ad730-3413-11ed-9adb-0296792ac05b",
            "auto-delete": "never",
            "eksctl.cluster.k8s.io/v1alpha1/cluster-name": "ironman-2022",
            "aws:cloudformation:logical-id": "ManagedNodeGroup",
            "alpha.eksctl.io/nodegroup-type": "managed",
            "alpha.eksctl.io/eksctl-version": "0.111.0"
        }
    }
}

由上述輸出內容 nodegroup 資源包含了此 Node 使用的 IAM role、Auto Scaling Group 名稱、Launch Template 及 CloudFormation 資訊。其中我們可以得知 EKS 預設使用 AMI 為： AL2_x86_64，此為 AWS 基於 Amazon Linux 2 AMI 維護 Amazon EKS optimized Amazon Linux AMIs [9] 。

Amazon EKS AMI Build Specification

使用 aws ec2 describe-instance-attribute 命令查看 EC2 userdata [11] 並透過 base64 命令解析：

$ aws ec2 describe-instance-attribute --instance-id i-1234567890abcdef0 --attribute userData | jq -r .UserData.Value | base64 -d
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
set -ex
B64_CLUSTER_CA=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHV... SKIP    ...
... CA      ...
... CONTENT ...
kZtcnI5V1lZRExMUDdLCm0xVUJQUWdzTzRQQlREUjlaLzhpbnZDV0FiT0szM2Z6OVZqU3dBbjlhQ0lXbU5FY2dVMkFUWm1FN0N4WEUrbFkKOFhjPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
API_SERVER_URL=https://1234567890ABCDEFGHIJKLMNOPQRSTUV.gr7.eu-west-1.eks.amazonaws.com
K8S_CLUSTER_DNS_IP=10.100.0.10
/etc/eks/bootstrap.sh ironman-2022 --kubelet-extra-args '--node-labels=eks.amazonaws.com/sourceLaunchTemplateVersion=1,alpha.eksctl.io/nodegroup-name=ng1-public-ssh,alpha.eksctl.io/cluster-name=ironman-2022,eks.amazonaws.com/nodegroup-image=ami-0ec9e1727a24fb788,eks.amazonaws.com/capacityType=ON_DEMAND,eks.amazonaws.com/nodegroup=ng1-public-ssh,eks.amazonaws.com/sourceLaunchTemplateId=lt-000b4417a7baebbf8 --max-pods=29' --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL --dns-cluster-ip $K8S_CLUSTER_DNS_IP --use-max-pods false

--//--

我們可以得知 bash script 定義：

B64_CLUSTER_CA、API_SERVER_URL、K8S_CLUSTER_DNS_IP 環境變數。
執行了於 /etc/eks/bootstrap.sh 路徑 script。

根據 Amazon EKS optimized Amazon Linux AMI build script [12] 提及了 GitHub Amazon EKS AMI Build Specification [13] 定義了 EKS node bootstrap script 包含 certificate data、control plane endpoint、cluster 名稱等資訊。

而預設的 userdata script 則定義了以下幾個參數：

cluster 名稱。
--kubelet-extra-args：定義額外 kubelet 參數，方便增加設定 labels 或 taints。
--b64-cluster-ca：EKS cluster based64 編碼後的內容。此部分透過 AWS CLI aws eks describe-cluster 命令取得後設定儲存於 /etc/kubernetes/pki/ca.crt。
--apiserver-endpoint： EKS cluster API Server endpoint。與 --b64-cluster-ca 一樣皆是透過 aws eks describe-cluster 命令，並設定 kubelet 所使用的 kubeconfig 文件（/var/lib/kubelet/kubeconfig ）。
--dns-cluster-ip：設定 EKS cluster 內部使用的 DNS IP 地址於 kubelet 設定文件 /etc/kubernetes/kubelet/kubelet-config.json，正是預設 CoreDNS（kube-dns）Service Cluster IP。預設使用 10.100.0.10，若使用 CIDR 10. 為 prefix 則會使用 172.20.0.10[14]。

$ kubectl -n kube-system get svc
NAME       TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE
kube-dns   ClusterIP   10.100.0.10           53/UDP,53/TCP   4d5h

--use-max-pods：設置 kubelet 參數 --max-pods 於 kubelet 設定文件 /etc/kubernetes/kubelet/kubelet-config.json。

總結

以上是 EKS worker node 啟用前在預設 EKS optimized Amazon Linux AMI 內預設 script 及針對 kubelet 參數設定，在經過初步了解之後，下一篇我們將會探討 kubelet TLS bootstrapping 啟動過程，來了解實際自動加入 cluster 過程 kubelet 及 control plane 所需設置。

上述資訊透過 EKS 所提供 Logs 來驗證上游 Kubernetes 運作原理，倘若上述內文有所錯誤，隨時可以留言或是私訊我。

參考文件

Kubernetes Components - https://kubernetes.io/docs/concepts/overview/components/#node-components
Provisioning a CA and Generating TLS Certificates | Kubernetes The Hard Way - https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/04-certificate-authority.md
kubeadm join - https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-join/
Managing nodegroups | eksctl - https://eksctl.io/usage/managing-nodegroups/
Amazon EKS nodes - https://docs.aws.amazon.com/eks/latest/userguide/eks-compute.html
Managed node groups - https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html
Self-managed nodes - https://docs.aws.amazon.com/eks/latest/userguide/worker.html
aws eks describe-nodegroup - https://docs.aws.amazon.com/cli/latest/reference/eks/describe-nodegroup.html
Amazon EKS optimized Amazon Linux AMIs - https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html
aws ec2 describe-instance-attribute - https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-attribute.html
Run commands on your Linux instance at launch - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html
Amazon EKS optimized Amazon Linux AMI build script - https://docs.aws.amazon.com/eks/latest/userguide/eks-ami-build-scripts.html
Amazon EKS AMI Build Specification - https://github.com/awslabs/amazon-eks-ami
https://github.com/awslabs/amazon-eks-ami/blob/master/files/bootstrap.sh#L462-L485
TLS bootstrapping - https://kubernetes.io/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/

為什麼 kubectl 可以訪問 EKS cluster

Mao-Lin Wang — Fri, 16 Sep 2022 16:55:00 GMT

根據 EKS 文件 Installing or updating kubectl [1]，或是 Kubernetes 官方安裝 kubectl 文件 [2] 皆是直接安裝 kubectl binary 並設置相應 Linux 路徑及權限即可以直接使用。

而 EKS 前置作業則需要設置 AWS CLI 權限，那「為什麼原生的 kubectl 可以直接訪問 EKS cluster」以及「為什麼 AWS CLI 權限與訪問 EKS cluster」，本文將探討這兩個問題，希望理解在 kubectl 如何與 IAM 權限整合允許訪問 EKS cluster。

AWS IAM Authenticator

EKS 文件並未提及 kubectl 所需最低版本，而在檢視 eksctl [3] GitHub 頁面說明提及需搭配 AWS CLI（aws eks get-token）或是使用 AWS IAM Authenticator for Kubernetes [4]。

AWS IAM Authenticator for Kubernetes 提供使用 AWS IAM credentials 可以向 Kubernetes cluster 進行身份驗證（authenticate）的工具。

若希望 Kubernetes 在 AWS 環境上支援 IAM Authenticator，需要以下五個步驟：

建立 IAM 角色（role）
以 DaemonSet 方式執行 Authenticator server
設定 Kubernetes API Server 與 Authenticator server 整合
創建 IAM role/user 到 Kubernetes user/group 對照
設定 kubectl 使用 AWS IAM Authenticator 所提供的 authentication tokens

有趣的是，其中在對照 IAM role/user 致 Kubernetes user/group 所使用的正是 EKS 文件 [5] 所提及使用的 EKS-style kube-system/aws-auth ConfigMap。以下為預設 eksctl 建立時會自動將 EKS worker node IAM role 關聯至此 kube-system/aws-auth ConfigMap。

$ kubectl -n kube-system get cm aws-auth -o yaml
apiVersion: v1
data:
  mapRoles: |
    - groups:
      - system:bootstrappers
      - system:nodes
      rolearn: arn:aws:iam::111111111111:role/eksctl-ironman-2022-nodegroup-ng1-publ-NodeInstanceRole-HN27OZ18JS6U
      username: system:node:{{EC2PrivateDNSName}}
kind: ConfigMap
metadata:
  creationTimestamp: "2022-09-14T09:56:19Z"
  name: aws-auth
  namespace: kube-system
  resourceVersion: "2011"
  uid: f4cb3fae-17ef-421d-871c-5be15efbe73f

進一步查看 EKS control plane logging [6] 文件，我們也可以更近一步確認 EKS 除了以下原生元件 logs 之外，提供了

Kubernetes API server
Audit
Controller manager
Scheduler
Authenticator (authenticator) – Authenticator logs 為 EKS 環境獨有。這些 logs 代表 EKS 使用 IAM 憑證進行 Kubernetes RBAC 身分驗證的 control plane 元件。

驗證

kubectl

一般來說，kubectl 命令設定文建會儲存於 ~/.kube/config，透過 eksctl 命令建立 cluster 時也會自動生成此文件。在 kubeconfig 設定檔，使用了 users.user.exec 命令並調用外部 AWS CLI 命令。

$ cat ~/.kube/config
apiVersion: v1
clusters:
... 
... SKIPPING certificate-authority-data INFORMATION ... 
...
current-context: arn:aws:eks:eu-west-1:111111111111:cluster/ironman-2022
kind: Config
preferences: {}
users:
- name: cli@ironman-2022.eu-west-1.eksctl.io
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - eks
      - get-token
      - --cluster-name
      - ironman-2022
      - --region
      - eu-west-1
      command: aws

在 Kubernetes 1.10 alpha k8s.io/client-go library 支援了 exec-based credential providers [7]。而 kubectl 及 kubectl 命令正式使用相同 k8s.io/client-go library。

同時，我們也可以得知若 client 端使用 client-go credential plugins 方式與 API server 進行認證，有以下流程：

使用者發出 kubectl 命令
調用外部 credentials 並與外部服務取得 token
Credential plugin 取得 token 後返回給 client-go client，並使用 token 訪問 API server
API Server 使用 webhook token authenticator 元件向外部服務發出 TokenReview 請求。
外部服務驗證 token 上的 signature 後，返回 Kubernetes user name 及 group

因此我們也可以透過 aws eks get-token 命令查看此 token：

$ aws eks get-token --cluster ironman-2022 | jq .
{
  "kind": "ExecCredential",
  "apiVersion": "client.authentication.k8s.io/v1beta1",
  "spec": {},
  "status": {
    "expirationTimestamp": "2022-09-14T23:04:55Z",
    "token": "k8s-aws-v1.aHR0cHM6Ly9zdHMuZXUtd2VzdC0xLmFtYXpvbmF3cy5jb20vP0FjdGlvbj1HZXRDYWxsZXJJZGVudGl0eSZWZXJzaW9uPTIwMTEtMDYtMTUmWC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBWUZNUVNOU0U1SDVaVEpERSUyRjIwMjIwOTE0JTJGZXUtd2VzdC0xJTJGc3RzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyMjA5MTRUMjI1MDU1WiZYLUFtei1FeHBpcmVzPTYwJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCUzQngtazhzLWF3cy1pZCZYLUFtei1TaWduYXR1cmU9MDU5ZjQwNzEzNTY0OGYwNjdlMWZjOThjZjhhYmY3YjdkMmIzNDgxMjE1ZWEzNGE1NTI4YzcyODczMmU1YjJkYw"
  }
}

每個 token 皆是 AWS IAM Authenticator 定義以 k8s-aws-v1. 字串為前綴，並接續使用 base64 encoded string，故透過 base64 命令解析出 STS presigned URL[9]：

$ aws eks get-token --cluster ironman-2022 | jq -r .status.token | awk -F. '{print $2}' | base64 -d
https://sts.eu-west-1.amazonaws.com/?Action=GetCallerIdentity&Version=2011-06-15&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYFMQSNSE5H5ZTJDE%2F20220914%2Feu-west-1%2Fsts%2Faws4_request&X-Amz-Date=20220914T225747Z&X-Amz-Expires=60&X-Amz-SignedHeaders=host%3Bx-k8s-aws-id&X-Amz-Signature=9688c99e7abd33807bc7b3af3542a59235e501a0d9c807100707410fc3da5d33

Control plane

透過以下 CloudWatch Log Syntax 可查看 kube-apiserver 所使用 Flag：

filter @logStream not like /^kube-apiserver-audit/
 | filter @logStream like /^kube-apiserver-/
 | fields @timestamp, @message
 | sort @timestamp asc
 | filter @message like "--authentication"
 | limit 10000

kube-apiserver logs 定義了 --authentication-token-webhook-config-file 則代表使用了 Webhook Token Authentication [10] ，但非常遺憾的是我們無法查看到 API Server 元件內定義目錄查看細節。/etc/kubernetes/authenticator/apiserver-webhook-kubeconfig.yaml。

I0914 09:46:06.295581 10 flags.go:59] FLAG: --authentication-token-webhook-cache-ttl="7m0s"
I0914 09:46:06.295687 10 flags.go:59] FLAG: --authentication-token-webhook-config-file="/etc/kubernetes/authenticator/apiserver-webhook-kubeconfig.yaml"
I0914 09:46:06.295703 10 flags.go:59] FLAG: --authentication-token-webhook-version="v1beta1"

此外，一樣透過 CloudWatch Log Syntax 查看 authenticator log，也能觀察到相同的設置及 logs：

filter @logStream like /^authenticator/
 | fields @timestamp, @message
 | sort @timestamp asc
 | filter @message like "--authentication-token-webhook-config-file"
 | limit 10000

time="2022-09-14T09:46:01Z" level=info msg="reconfigure your apiserver with `--authentication-token-webhook-config-file=/etc/kubernetes/authenticator/apiserver-webhook-kubeconfig.yaml` to enable (assuming default hostPath mounts)"

總結

原生 kubectl 使用了 k8s.io/client-go library，並在 Kubernetes 1.10 版本開始支援 exec 命令使用外部 credentials，而在 API server 則設定了使用 webhook token authenticator 方式設定 AWS IAM Authenticator 整合 IAM STS 服務驗證 IAM 使用者，STS endpoint 驗證後交由 Kubernetes API server 讀取 kube-system/aws-auth ConfigMap 對照 Kubernetes 使用 RBAC groups，最終返回結果。

了解上述 aws eks get-token 為 Kubernetes Bearer Token，故我們也可以直接透過 curl 命令搭配此 token 訪問 EKS cluster endpoint。

$ TOKEN=$(aws eks get-token --cluster ironman-2022 | jq -r .status.token)
$ APISERVER=$(aws eks describe-cluster --name ironman-2022 | jq -r .cluster.endpoint) 
$ curl $APISERVER/api --header "Authorization: Bearer $TOKEN" --insecure
{
  "kind": "APIVersions",
  "versions": [
    "v1"
  ],
  "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "ip-10-0-51-57.eu-west-1.compute.internal:443"
    }
  ]

參考文件

Installing or updating kubectl - https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html
Install and Set Up kubectl on Linux | Kubernetes Documentation - https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/
eksctl - The official CLI for Amazon EKS - https://github.com/weaveworks/eksctl
AWS IAM Authenticator for Kubernetes - https://github.com/kubernetes-sigs/aws-iam-authenticator.
Enabling IAM user and role access to your cluster - Add IAM users or roles to your Amazon EKS cluster - https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html#aws-auth-users
Amazon EKS control plane logging - https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html
client-go: add an exec-based client auth provider #59495 - https://github.com/kubernetes/kubernetes/pull/59495
client-go credential plugins | Authenticating - https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins
https://github.com/kubernetes-sigs/aws-iam-authenticator#api-authorization-from-outside-a-cluster
Webhook Token Authentication | Authenticating - https://kubernetes.io/docs/reference/access-authn-authz/authentication/#webhook-token-authentication

建立 EKS Cluster

Mao-Lin Wang — Thu, 15 Sep 2022 16:55:00 GMT

前言

根據 Kubernetes 官方公告[1]，Kubernetes 發布週期約為 15 週一次，換言之每接近 4 個月就會發布一次新的版本。而其中 Cloud Provider 所提供的 Kubernetes 平台也必須持續更新，可以說是每隔三至五個月皆會有一次新的發布，如 GKE[2] 及 EKS[3] 發布頻率。

因此我們也能觀察到 GKE [2]及 EKS [4][5] 於文件上頻繁更新，並基於上游 Kubernetes 引入新的功能，進而整合 Cloud Provider 特性。

故本系列文章希望研究在 AWS 平台上 EKS 一些有趣的功能是如何實現，究竟哪些事情是 EKS 官方文件上並未提及的事情，如 kubectl 如何使用 IAM user 權限整合 EKS、EKS worker node 如何自動加入 cluster，最終了解 EKS 如何整合上游 Kubernetes 功能與 AWS 服務整合。

建立 EKS Cluster 環境

本系列文章將會使用eksctl 管理 EKS Cluster，以下為建立 EKS 步驟，及使用相關命令版本。

啟用 EC2 並選用 AMI Amazon Linux 2 Kernel 5.10 AMI 作為控制 EKS Cluster 主機。
安裝 AWS CLI[6]。

$ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 44.8M  100 44.8M    0     0  69.0M      0 --:--:-- --:--:-- --:--:-- 68.9M

$ unzip awscliv2.zip

$ sudo ./aws/install

$ aws --version
aws-cli/2.4.27 Python/3.8.8 Linux/4.14.290-217.505.amzn2.x86_64 exe/x86_64.amzn.2 prompt/off

3. 安裝 eksctl [7] 命令。

$ curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp

$ sudo mv /tmp/eksctl /usr/local/bin

$ eksctl version
0.111.0

4. 當前 eksctl 預設版本為 1.22 版本[8]，故安裝 kubectl 1.22 版本[9]。

$ curl -LO https://dl.k8s.io/release/v1.22.13/bin/linux/amd64/kubectl
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   138  100   138    0     0    836      0 --:--:-- --:--:-- --:--:--   841
100 44.7M  100 44.7M    0     0  82.8M      0 --:--:-- --:--:-- --:--:-- 82.8M

$ sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

$ kubectl version
Kubeconfig user entry is using deprecated API version client.authentication.k8s.io/v1alpha1. Run 'aws eks update-kubeconfig' to update.
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.13", GitCommit:"a43c0904d0de10f92aa3956c74489c45e6453d6e", GitTreeState:"clean", BuildDate:"2022-08-17T18:28:56Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.12-eks-6d3986b", GitCommit:"dade57bbf0e318a6492808cf6e276ea3956aecbf", GitTreeState:"clean", BuildDate:"2022-07-20T22:06:30Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

5. 設置 AWS CLI 憑證（credentials）[10]。eksctl 將會使用此 IAM 使用者權限作為 EKS Cluster 建立者（creator），並建立 EKS Cluster、節點組（nodegroup）等 AWS 資源。倘若希望 IAM 使用者可以限縮 IAM 最低權限，可以參考 eksctl 最小 IAM policies 文件[11]

$ aws configure

6. 建立 eksctl ClusterConfig 文件，並啟用 control plane logs[12]。

$ cat ./ironman-2022.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: ironman-2022
  region: eu-west-1

managedNodeGroups:
  - name: "ng1-public-ssh"
    desiredCapacity: 2
    ssh:
      # Enable ssh access (via the admin container)
      allow: true
      publicKeyName: "ironman-2022"
    iam:
      withAddonPolicies:
        ebs: true
        fsx: true
        efs: true
        awsLoadBalancerController: true
        autoScaler: true

iam:
  withOIDC: true

cloudWatch:
  clusterLogging:
    enableTypes: ["*"]

$ eksctl create cluster -f ./ironman-2022.yaml                                                                                                                                                                      [133/4203]
2022-09-14 09:39:32 [ℹ]  eksctl version 0.111.0
2022-09-14 09:39:32 [ℹ]  using region eu-west-1
2022-09-14 09:39:32 [ℹ]  setting availability zones to [eu-west-1a eu-west-1c eu-west-1b]
2022-09-14 09:39:32 [ℹ]  subnets for eu-west-1a - public:192.168.0.0/19 private:192.168.96.0/19
2022-09-14 09:39:32 [ℹ]  subnets for eu-west-1c - public:192.168.32.0/19 private:192.168.128.0/19
2022-09-14 09:39:32 [ℹ]  subnets for eu-west-1b - public:192.168.64.0/19 private:192.168.160.0/19
2022-09-14 09:39:32 [ℹ]  nodegroup "ng1-public-ssh" will use "" [AmazonLinux2/1.22]
2022-09-14 09:39:32 [ℹ]  using EC2 key pair "ironman-2022"
2022-09-14 09:39:32 [ℹ]  using Kubernetes version 1.22
2022-09-14 09:39:32 [ℹ]  creating EKS cluster "ironman-2022" in "eu-west-1" region with managed nodes
2022-09-14 09:39:32 [ℹ]  1 nodegroup (ng1-public-ssh) was included (based on the include/exclude rules)
2022-09-14 09:39:32 [ℹ]  will create a CloudFormation stack for cluster itself and 0 nodegroup stack(s)
2022-09-14 09:39:32 [ℹ]  will create a CloudFormation stack for cluster itself and 1 managed nodegroup stack(s)
2022-09-14 09:39:32 [ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=eu-west-1 --cluster=ironman-2022'
2022-09-14 09:39:32 [ℹ]  Kubernetes API endpoint access will use default of {publicAccess=true, privateAccess=false} for cluster "ironman-2022" in "eu-west-1"
2022-09-14 09:39:32 [ℹ]  configuring CloudWatch logging for cluster "ironman-2022" in "eu-west-1" (enabled types: api, audit, authenticator, controllerManager, scheduler & no types disabled)
2022-09-14 09:39:32 [ℹ]
2 sequential tasks: { create cluster control plane "ironman-2022",
    2 sequential sub-tasks: {
        4 sequential sub-tasks: {
            wait for control plane to become ready,
            associate IAM OIDC provider,
            2 sequential sub-tasks: {
                create IAM role for serviceaccount "kube-system/aws-node",
                create serviceaccount "kube-system/aws-node",
            },
            restart daemonset "kube-system/aws-node",
        },
        create managed nodegroup "ng1-public-ssh",
    }
}
2022-09-14 09:39:32 [ℹ]  building cluster stack "eksctl-ironman-2022-cluster"
2022-09-14 09:39:33 [ℹ]  deploying stack "eksctl-ironman-2022-cluster"
2022-09-14 09:40:03 [ℹ]  waiting for CloudFormation stack "eksctl-ironman-2022-cluster"
2022-09-14 09:40:33 [ℹ]  waiting for CloudFormation stack "eksctl-ironman-2022-cluster"
2022-09-14 09:41:33 [ℹ]  waiting for CloudFormation stack "eksctl-ironman-2022-cluster"
2022-09-14 09:42:33 [ℹ]  waiting for CloudFormation stack "eksctl-ironman-2022-cluster"
2022-09-14 09:43:33 [ℹ]  waiting for CloudFormation stack "eksctl-ironman-2022-cluster"
2022-09-14 09:44:33 [ℹ]  waiting for CloudFormation stack "eksctl-ironman-2022-cluster"
2022-09-14 09:45:33 [ℹ]  waiting for CloudFormation stack "eksctl-ironman-2022-cluster"
2022-09-14 09:46:33 [ℹ]  waiting for CloudFormation stack "eksctl-ironman-2022-cluster"
2022-09-14 09:47:33 [ℹ]  waiting for CloudFormation stack "eksctl-ironman-2022-cluster"
2022-09-14 09:48:33 [ℹ]  waiting for CloudFormation stack "eksctl-ironman-2022-cluster"
2022-09-14 09:49:33 [ℹ]  waiting for CloudFormation stack "eksctl-ironman-2022-cluster"
2022-09-14 09:50:33 [ℹ]  waiting for CloudFormation stack "eksctl-ironman-2022-cluster"
2022-09-14 09:51:33 [ℹ]  waiting for CloudFormation stack "eksctl-ironman-2022-cluster"
2022-09-14 09:53:34 [ℹ]  building iamserviceaccount stack "eksctl-ironman-2022-addon-iamserviceaccount-kube-system-aws-node"
2022-09-14 09:53:35 [ℹ]  deploying stack "eksctl-ironman-2022-addon-iamserviceaccount-kube-system-aws-node"
2022-09-14 09:53:35 [ℹ]  waiting for CloudFormation stack "eksctl-ironman-2022-addon-iamserviceaccount-kube-system-aws-node"
2022-09-14 09:54:05 [ℹ]  waiting for CloudFormation stack "eksctl-ironman-2022-addon-iamserviceaccount-kube-system-aws-node"
2022-09-14 09:54:05 [ℹ]  serviceaccount "kube-system/aws-node" already exists
2022-09-14 09:54:05 [ℹ]  updated serviceaccount "kube-system/aws-node"
2022-09-14 09:54:05 [ℹ]  daemonset "kube-system/aws-node" restarted
2022-09-14 09:54:05 [ℹ]  building managed nodegroup stack "eksctl-ironman-2022-nodegroup-ng1-public-ssh"
2022-09-14 09:54:05 [ℹ]  deploying stack "eksctl-ironman-2022-nodegroup-ng1-public-ssh"
2022-09-14 09:54:05 [ℹ]  waiting for CloudFormation stack "eksctl-ironman-2022-nodegroup-ng1-public-ssh"
2022-09-14 09:54:35 [ℹ]  waiting for CloudFormation stack "eksctl-ironman-2022-nodegroup-ng1-public-ssh"
2022-09-14 09:55:18 [ℹ]  waiting for CloudFormation stack "eksctl-ironman-2022-nodegroup-ng1-public-ssh"
2022-09-14 09:56:04 [ℹ]  waiting for CloudFormation stack "eksctl-ironman-2022-nodegroup-ng1-public-ssh"
2022-09-14 09:57:24 [ℹ]  waiting for CloudFormation stack "eksctl-ironman-2022-nodegroup-ng1-public-ssh"
2022-09-14 09:59:10 [ℹ]  waiting for CloudFormation stack "eksctl-ironman-2022-nodegroup-ng1-public-ssh"
2022-09-14 09:59:10 [ℹ]  waiting for the control plane availability...
2022-09-14 09:59:12 [✔]  saved kubeconfig as "/home/ec2-user/.kube/config"
2022-09-14 09:59:12 [ℹ]  no tasks
2022-09-14 09:59:12 [✔]  all EKS cluster resources for "ironman-2022" have been created
2022-09-14 09:59:12 [ℹ]  nodegroup "ng1-public-ssh" has 2 node(s)
2022-09-14 09:59:12 [ℹ]  node "ip-192-168-29-179.eu-west-1.compute.internal" is ready
2022-09-14 09:59:12 [ℹ]  node "ip-192-168-78-165.eu-west-1.compute.internal" is ready
2022-09-14 09:59:12 [ℹ]  waiting for at least 2 node(s) to become ready in "ng1-public-ssh"

參考文件

Kubernetes Release Cadence Change: Here’s What You Need To Know - https://kubernetes.io/blog/2021/07/20/new-kubernetes-release-cadence
Amazon EKS Kubernetes versions - Amazon EKS Kubernetes release calendar - https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-release-calendar
GKE release notes -https://cloud.google.com/kubernetes-engine/docs/release-notes
Document history for Amazon EKS - https://docs.aws.amazon.com/eks/latest/userguide/doc-history.html
Amazon EKS User Guide - https://github.com/awsdocs/amazon-eks-user-guide
Installing or updating the latest version of the AWS CLI - https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
Installing or updating eksctl - https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html
Introduction | eksctl - https://eksctl.io/introduction/
Install and Set Up kubectl on Linux - https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/
Configuration and credential file settings - https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html
Minimum IAM policies | eksctl - https://eksctl.io/usage/minimum-iam-policies/#minimum-iam-policies
Amazon EKS control plane logging - https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html

How to control your external display brightness on macOS and Arch Linux

Mao-Lin Wang — Mon, 14 Mar 2022 14:05:33 GMT

Preface

在筆記型電腦上基本上都會內建功能提供調整螢幕亮度(Brightness)。倘若電腦使用了外接螢幕，一般來說僅能透過外接螢幕功能鍵進行調整，而沒辦法使用內建螢幕功能調整外接螢幕。

有沒有辦法透過在筆記型電腦上直接控制外接螢幕的亮度及音量大小呢？

DDC/CI

首先，在作業系統上是否有相關界面可以對外接螢幕進操作。根據 Wiki，2016 年後螢幕多數都已經支援了 Display Data Channel Command Interface（DDC/CI） [1]：提供了由電腦與螢幕顯示溝通的訊號界面，並可以透過軟體定義調整外接螢幕亮度。

macOS

在 macOS 環境中，使用 MonitorControl [2] 來調整外接螢幕亮度。以下使用外接螢幕 LG 32ML600M 測試，可以觀察到外接螢幕音軌進行調整。

此外，macOS 系統已經提供內建螢幕及外接螢幕明亮，於 MonitorControl 4.0.0 [3] 後提供了同步內建螢幕與外接螢幕亮度功能。

當然，需要注意若習慣將 macbook 關閉上蓋，環境光度感測器（ambient light sensor）就不會自動感測而變化外接螢幕亮度。

Linux - Arch Linux

Linux 環境則可以使用 i2c-dev [4] 或 ddcci-driver-linux [5] 整合。

i2c-dev ：原生 kernel 已經支援，僅需要設定 /etc/modules-load.d 使 kernel 開機時載入即可，參考 ddcutil kernel module 設置 [6]。

最終目的希望透過桌面環境調整外接螢幕，以下為我的 Arch Linux 環境資訊：

$ uname -a
Linux nuc-arch 5.16.14-arch1-1 #1 SMP PREEMPT Fri, 11 Mar 2022 17:40:36 +0000 x86_64 GNU/Linux

$ plasmashell -v     
plasmashell 5.24.3

$ kf5-config --version
Qt: 5.15.3
KDE Frameworks: 5.92.0
kf5-config: 1.0

由於使用了 Arch Linux KDE Plasma 環境中整合，其 Plasma PowerDevil [7] module 提供電源管理控制，其包含螢幕亮度及裝置電源等。

不過目前 PowerDevil 套件預設 [8] 將 ddcutil（-DHAVE_DDCUTIL=Off）關閉，故需要手動編譯啟用 ddcutil功能。

當然 Arch user 維護了 powerdevil-ddcutil AUR [9] 提供大家使用：

$ sudo gpasswd -a $USER i2c
$ sudo sh -c 'echo i2c-dev > /etc/modules-load.d/ddc.conf'
$ sudo reboot

重起後，KDE Plasma 順利地直接調整外接螢幕：

References

Why logging system collects container logs from specific directory

Mao-Lin Wang — Fri, 04 Mar 2022 16:00:02 GMT

Logging Architecture

根據 Kubernetes Logging Architecture[1] 文件，大致上分為兩類 node-level 及 cluster-level：

Logging at the node level

node-level 透過 logrotate 命令或是 kubelet containerLogMaxSize 及 containerLogMaxFiles 參數 rotate log。

Cluster-level logging architectures

cluster-level 使用以下三種方式實現：

Using a node logging agent

於每一個節點上接部署 Logging Agent 收集 application log 並上傳回 logging 系統。例如：Flutend 或是 Grafana Loki 都是比較接近這種方式，於每一個節點透過 DaemonSet 部署方式，等同有一支 agent process 收集每個節點上的 log 。執得注意的是，倘若 logging system 也是依賴於 Kubernetes 環境，而 Kubernetes 無法正常運作時，logging system 也可能癱瘓，需要注意環境相依性及單點故障。

Using a sidecar container with the logging agent

Streaming sidecar container：一般來說，Container log 將 stdout 及 stderr 輸出於 Container run time 預設目錄，但是有些 application 可能預設並不會輸出至 stdout 或 stderr。如部分老舊系統有固定產出的目錄位置，則可以透過 sidecar container 搭配使用 tail 命令定期讀取該目錄位置，並由 logging agent 收集 log 回 logging system。

Sidecar container with a logging agent ：直接透過 sidecar container 收集 application 回 logging system。例如：倘若僅需要收集特殊 application log，可以自行撰寫 script 或是 HTTP API 將 log 更新至 logging system。

Exposing logs directly from the application：直接將透過 logging system 收集 log。

Why we collects logs from `/var/log/containers/` directory

那到底這些 Logging system 本質上是怎麼於節點上採集 application log 呢？

以下以 EKS 官方 CloudWatch Agent for Container Insights Kubernetes Monitoring 解決方案為例，提供了 Flutend 及 Fluent Bit 兩種 Logging system。

有趣的是，這兩種不同的 Logging system 卻同時收集了相同 /var/log/containers/*.log 目錄作為 application log，預設 config 分別如下方所示：

Flutend

預設 Flutend containers.conf 收集 /var/log/containers/*.log

  containers.conf: |
    
      @type tail
      @id in_tail_container_logs
      @label @containers
      path /var/log/containers/*.log
      exclude_path ["/var/log/containers/cloudwatch-agent*", "/var/log/containers/fluentd*"]
      pos_file /var/log/fluentd-containers.log.pos
      tag *
      read_from_head true
      
        @type json
        time_format %Y-%m-%dT%H:%M:%S.%NZ
      
    
    ...
    ...

Fluent Bit

預設 Fluent Bit application-log.conf 收集 /var/log/containers/*.log

  application-log.conf: |
    [INPUT]
        Name                tail
        Tag                 application.*
        Exclude_Path        /var/log/containers/cloudwatch-agent*, /var/log/containers/fluent-bit*, /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
        Path                /var/log/containers/*.log
        Docker_Mode         On
        Docker_Mode_Flush   5
        Docker_Mode_Parser  container_firstline
        Parser              docker
        DB                  /var/fluent-bit/state/flb_container.db
        Mem_Buf_Limit       50MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Rotate_Wait         30
        storage.type        filesystem
        Read_from_Head      ${READ_FROM_HEAD}
        ...

難道 Kubernetes cluster 本就會將 container log 輸出至 /var/log/containers 目錄嗎？

根據 Kubernetes Proposals，明確定義 Pod 是基於 cluster-level 收集 log 目的導向，都會 kubelet 透過 soft link 方式關聯 Container Runtime（如 Docker）至 /var/log/containers 目錄，並且依照 /var/log/containers/__-.log 格式作為 log 名稱。

In a production cluster, logs are usually collected, aggregated, and shipped to a remote store where advanced analysis/search/archiving functions are supported. In kubernetes, the default cluster-addons includes a per-node log collection daemon, `fluentd`. To facilitate the log collection, kubelet creates symbolic links to all the docker containers logs under `/var/log/containers` with pod and container metadata embedded in the filename.

	/var/log/containers/__-.log`

The fluentd daemon watches the `/var/log/containers/` directory and extract the metadata associated with the log from the path. Note that this integration requires kubelet to know where the container runtime stores the logs, and will not be directly applicable to CRI.

附註：Kubernetes Proposals 從 2021 4 月起已經將相關 Proposal 遷移至 kubernetes/enhancements GitHub。

Summary

故現況 Kubernetes - Cluster-level Using a node logging agent 架構，皆是收集 /var/log/contianers 目錄 log 作為一個 Kubernetes 收集 container application log 的共用規範。

參考文件

Why Linux OS takes over 5 minutes during boot-up process

Mao-Lin Wang — Mon, 14 Feb 2022 22:28:56 GMT

Problem

As the following output of journalctl command, we can view the network unit takes around 5 minutes between initial the LSB and failed to start:

Jun 01 15:59:45: Starting LSB: Bring up/down networking...
Jun 01 16:04:45: Failed to start LSB: Bring up/down networking.

$ journalctl --unit=network
-- Logs begin at Tue 2021-06-01 15:59:40 UTC, end at Tue 2021-06-01 16:05:37 UTC. --
Jun 01 15:59:45 ip-172-31-29-127.eu-west-1.compute.internal systemd[1]: Starting LSB: Bring up/down networking...
Jun 01 15:59:45 ip-172-31-29-127.eu-west-1.compute.internal network[666]: Bringing up loopback interface:  [  OK  ]
Jun 01 15:59:45 ip-172-31-29-127.eu-west-1.compute.internal network[666]: Bringing up interface eth0:
Jun 01 15:59:45 ip-172-31-29-127.eu-west-1.compute.internal dhclient[789]: DHCPREQUEST on eth0 to 255.255.255.255 port 67 (xid=0x7487dbae)
Jun 01 15:59:45 ip-172-31-29-127.eu-west-1.compute.internal dhclient[789]: DHCPACK from 172.31.16.1 (xid=0x7487dbae)
Jun 01 15:59:47 ip-172-31-29-127.eu-west-1.compute.internal NET[833]: /usr/sbin/dhclient-script : updated /etc/resolv.conf
Jun 01 15:59:47 ip-172-31-29-127.eu-west-1.compute.internal network[666]: Determining IP information for eth0... done.
Jun 01 15:59:48 ip-172-31-29-127.eu-west-1.compute.internal dhclient[856]: XMT: Solicit on eth0, interval 1020ms.
Jun 01 15:59:49 ip-172-31-29-127.eu-west-1.compute.internal dhclient[856]: XMT: Solicit on eth0, interval 2120ms.
Jun 01 15:59:51 ip-172-31-29-127.eu-west-1.compute.internal dhclient[856]: XMT: Solicit on eth0, interval 4450ms.
Jun 01 15:59:55 ip-172-31-29-127.eu-west-1.compute.internal dhclient[856]: XMT: Solicit on eth0, interval 9080ms.
Jun 01 16:00:05 ip-172-31-29-127.eu-west-1.compute.internal dhclient[856]: XMT: Solicit on eth0, interval 18980ms.
Jun 01 16:00:24 ip-172-31-29-127.eu-west-1.compute.internal dhclient[856]: XMT: Solicit on eth0, interval 37130ms.
Jun 01 16:01:01 ip-172-31-29-127.eu-west-1.compute.internal dhclient[856]: XMT: Solicit on eth0, interval 77810ms.
Jun 01 16:02:19 ip-172-31-29-127.eu-west-1.compute.internal dhclient[856]: XMT: Solicit on eth0, interval 113620ms.
Jun 01 16:04:12 ip-172-31-29-127.eu-west-1.compute.internal dhclient[856]: XMT: Solicit on eth0, interval 35520ms.
Jun 01 16:04:45 ip-172-31-29-127.eu-west-1.compute.internal systemd[1]: network.service start operation timed out. Terminating.
Jun 01 16:04:45 ip-172-31-29-127.eu-west-1.compute.internal systemd[1]: Failed to start LSB: Bring up/down networking.
Jun 01 16:04:45 ip-172-31-29-127.eu-west-1.compute.internal systemd[1]: Unit network.service entered failed state.
Jun 01 16:04:45 ip-172-31-29-127.eu-west-1.compute.internal systemd[1]: network.service failed.
Jun 01 16:04:48 ip-172-31-29-127.eu-west-1.compute.internal network[666]: Determining IPv6 information for eth0... failed.
Jun 01 16:04:48 ip-172-31-29-127.eu-west-1.compute.internal network[666]: WARN      : [/etc/sysconfig/network-scripts/ifup-eth] Unable to obtain IPv6 DHCP address eth0.

Find the systemd unit that take a long time

In this case, we can use systemd-analyze blame command to show a list of all running units, ordered by the time they took to initialize.

$ sudo systemd-analyze blame
  5min 115ms network.service
      3.866s dev-xvda1.device
      3.101s cloud-init-local.service
      1.206s cloud-init.service

From the output message of systemd-analyze blame, we can know the network.service take around 5 min. Moreover, we also can check the configuration of network.service that defined the TimeoutSec=5min by default[2].

$ sudo systemctl edit --full network.service
...
...
[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
ExecStart=/etc/rc.d/init.d/network start
ExecStop=/etc/rc.d/init.d/network stop

This means that network.service might be waiting for something and cause the timeout. Therefore, we should check the networking configurations.

$ cat /etc/sysconfig/network-scripts/ifcfg-ens5 

# Created by cloud-init on instance boot automatically, do not edit.
#
BOOTPROTO=dhcp
DEVICE=ens5
DHCPV6C=yes
IPV6INIT=yes
ONBOOT=yes
TYPE=Ethernet
USERCTL=no

From this configuration of the interface ens5, we can know IPv6 address is enabled. However, the configurations in /etc/sysconfig/network-scripts are created by cloud-init on instance boot automatically.

By checking the configurations in /etc/cloud/cloud.cfg.d/, we can confirm the IPv6 address is enabled in cloud-init.

...
    network:
      version: 1
      config:
      - type: physical
        name: eth0
        subnets:
          - type: dhcp
          - type: dhcp6

In the meantime, I can view the VPC did not be enabled the IPv6 support[3].

Summary

If we enable the IPv6 at the Linux OS level, but we don't enable IPv6 support for your VPC and resources. During the boot-up process, network.unit would like to retrieve IPv6 IP address and wait for DHCP server for 5 min. After the timeout period, the systemd.unit would continue the process, and cause a slow boot-up time over 5 minutes.

References

https://www.freedesktop.org/software/systemd/man/systemd-analyze.html
https://www.freedesktop.org/software/systemd/man/systemd.service.html
Get started with IPv6 for Amazon VPC - https://docs.aws.amazon.com/vpc/latest/userguide/get-started-IPv6.html

AWS VPC CNI plugin random livenessProbe failures after upgrading to Kubernetes 1.20

Mao-Lin Wang — Sun, 23 Jan 2022 20:42:07 GMT

Problem

After upgrading the EKS 1.19 to 1.20, we can find some pods were using readiness and liveness probes failures without reason message. Take AWS VPC CNI plugin aws-node pod as an example below:

# Pod event
...
Normal   Killing    85m                    kubelet  Container aws-node failed liveness probe, will be restarted
Normal   Pulling    85m (x2 over 106m)     kubelet  Pulling image "602401143452.dkr.ecr.eu-central-1.amazonaws.com/amazon-k8s-cni:v1.8.0-eksbuild.1"
Normal   Created    85m (x2 over 106m)     kubelet  Created container aws-node
Normal   Pulled     85m                    kubelet  Successfully pulled image "602401143452.dkr.ecr.eu-central-1.amazonaws.com/amazon-k8s-cni:v1.8.0-eksbuild.1" in 158.21763ms
Normal   Started    85m (x2 over 106m)     kubelet  Started container aws-node
Warning  Unhealthy  11m (x31 over 106m)    kubelet  Readiness probe failed:
Warning  Unhealthy  4m57s (x28 over 100m)  kubelet  Liveness probe failed:

What changes in Kubernetes 1.20

According to the Kubernetes 1.20: The Raddest Release[1]:

A longstanding bug regarding exec probe timeouts that may impact existing pod definitions has been fixed. Prior to this fix, the field timeoutSeconds was not respected for exec probes. Instead, probes would run indefinitely, even past their configured deadline, until a result was returned. With this change, the default value of 1 second will be applied if a value is not specified and existing pod definitions may no longer be sufficient if a probe takes longer than one second.

This means a bug fix in Kubernetes 1.20: Fixing Kubelet Exec Probe Timeouts[2] and KEP-1972: kubelet exec probe timeouts[3]. Now the default timeout 1s is respected but is periodically too short causing it to fail and pods to restart.

Workaround

Here are 2 methods to mitigate the issue:

Disable the feature gate ExecProbeTimeout on kubelet: As a cluster administrator, we can disable the feature gate ExecProbeTimeout (set it to false) on each kubelet to restore the behavior from older versions, then remove that override once all the exec probes in the cluster have a timeoutSeconds value set[4].
Increase the timeoutSeconds to a proper value: If you have pods that are impacted from the default 1 second timeout, you should update their probe timeout so that you’re ready for the eventual removal of that feature gate.

For VPC CNI plugin, however, issue #1425[5] is still considered a bug and needs to be followup.

Reference

Why my CDK VPC constructor does not respect maximum availability zones

Mao-Lin Wang — Sat, 15 Jan 2022 11:30:00 GMT

Problem

In the following code snippet, we configure the maximum number of availability zones(AZs). However, CDK does not respect the value.

$ cdk --version
2.5.0 (build 0951122)

$ cat ./bin/vpc-additional-subnet-stack.ts
#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import { VpcAdditionalSubnetStackStack } from '../lib/vpc-additional-subnet-stack-stack';

const app = new cdk.App();
new VpcAdditionalSubnetStackStack(app, 'VpcAdditionalSubnetStackStack', {

  env: { account: '111111111111', region: 'eu-west-1' },

});%

import { Stack, StackProps } from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as ec2 from "aws-cdk-lib/aws-ec2";

export class VpcAdditionalSubnetStackStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id);

    const subnetConfig = [
      {
          cidrMask: 22,
          name: "outputSubnet",
          subnetType: ec2.SubnetType.PUBLIC,
      },
      {
          cidrMask: 22,
          name: "database",
          subnetType: ec2.SubnetType.PRIVATE_WITH_NAT,
      },
      {
          cidrMask: 22,
          name: "application",
          subnetType: ec2.SubnetType.PRIVATE_WITH_NAT,
      },
    ];

    // VPC
    const vpc = new ec2.Vpc(this, "Lab-VPC", {
      cidr: "10.0.0.0/16",
      maxAzs: 3,
      subnetConfiguration: subnetConfig,
    });

  }
}

Also, we can view there are 3 AZs in eu-west-1 region.

$ aws ec2 describe-availability-zones

{
    "AvailabilityZones": [
        {
            "State": "available",
            "OptInStatus": "opt-in-not-required",
            "Messages": [],
            "RegionName": "eu-west-1",
            "ZoneName": "eu-west-1a",
            "ZoneId": "euw1-az1",
            "GroupName": "eu-west-1",
            "NetworkBorderGroup": "eu-west-1",
            "ZoneType": "availability-zone"
        },
        {
            "State": "available",
            "OptInStatus": "opt-in-not-required",
            "Messages": [],
            "RegionName": "eu-west-1",
            "ZoneName": "eu-west-1b",
            "ZoneId": "euw1-az2",
            "GroupName": "eu-west-1",
            "NetworkBorderGroup": "eu-west-1",
            "ZoneType": "availability-zone"
        },
        {
            "State": "available",
            "OptInStatus": "opt-in-not-required",
            "Messages": [],
            "RegionName": "eu-west-1",
            "ZoneName": "eu-west-1c",
            "ZoneId": "euw1-az3",
            "GroupName": "eu-west-1",
            "NetworkBorderGroup": "eu-west-1",
            "ZoneType": "availability-zone"
        }
    ]
}

Dive into the problem

According to the Stack.availabilityZones:

If the stack is environment-agnostic (either account and/or region are tokens), this property will return an array with 2 tokens that will resolve at deploy-time to the first two availability zones returned from CloudFormation's Fn::GetAZs intrinsic function.

From the code snippet, we had already set up the account and region. It seems that the Stack does not inherit the environment. Thus, we can check the account and region of Stack again as the following snippet.

export class VpcAdditionalSubnetStackStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);
    console.log('account: ', Stack.of(this).account);
    console.log('region: ', Stack.of(this).region);
    console.log('availability zones', Stack.of(this).availabilityZones);

$ cdk synth
account:  ${Token[AWS.AccountId.6]}
region:  ${Token[AWS.Region.10]}
availability zones [ '${Token[TOKEN.200]}', '${Token[TOKEN.202]}' ]
...
...

AWS CDK encodes a token whose value is not yet known at construction time[2]. We can know the environment variables are not been used in VpcAdditionalSubnetStackStack Stack.

    super(scope, id);

From the code snippet, which does not pass props in the call tosuper(), and the environment variable we pass when creating VpcAdditionalSubnetStackStack is ignored. Therefore, CDK considers this to be environment-agnostic and creates only 2 AZ.

$ cat ./lib/vpc-additional-subnet-stack-stack.ts
import { Stack, StackProps } from 'aws-cdk-lib';
import { Construct } from 'constructs';
// import * as sqs from 'aws-cdk-lib/aws-sqs';

export class VpcAdditionalSubnetStackStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    // The code that defines your stack goes here

    // example resource
    // const queue = new sqs.Queue(this, 'VpcAdditionalSubnetStackQueue', {
    //   visibilityTimeout: cdk.Duration.seconds(300)
    // });
  }
}

By default, the CDK helps us pull a template that defines the basic constructor. Here is the example template:

    super(scope, id, props);

$ cdk synth
account:  111111111111
region:  eu-west-1
availability zones [ 'eu-west-1a', 'eu-west-1b', 'eu-west-1c' ]
...

After updating the super() function, we can view the account and region output.

Summary

In order to create the availability zones of an AWS region, we must set up the following items:

Environment variable for account and region, it can be set up via AWS credential or env property.
Confirm the Stack inherits theStackProps, if not the stack will ignore the environment variable we set up in previous steps.

References

How OpsWorks mount EBS Volume automatically

Mao-Lin Wang — Sat, 08 Jan 2022 11:30:00 GMT

Add an Additional EBS Volumes with OpsWorks Layer's configuration.

$ aws opsworks --region us-west-2 describe-layers --stack-id 433a1bd3-699d-4306-9533-b44530293ab5 --query 'Layers[*].VolumeConfigurations'


[
    [
        {
            "MountPoint": "/mnt/workspace",
            "NumberOfDisks": 1,
            "Size": 10,
            "VolumeType": "gp2",
            "Encrypted": false
        }
    ]
]

Add SSH key with Stack configuration. So, we can access the instance via SSH.
Access to the OpsWorks instance. I can view the EBS is mounted on /mnt/workspace.

$ mount | grep "workspace"
/dev/xvdi on /mnt/workspace type xfs (rw,relatime,attr2,inode64,noquota)

The ephemeral device ephemeral0 is mounted on /media/ephemeral0 by cloud-init.

$ tail -n 5 /etc/cloud/cloud.cfg

mounts:
 - [ ephemeral0, /media/ephemeral0 ]
 - [ swap, none, swap, sw, "0", "0" ]
# vim:syntax=yaml

EBS volumes will be set up the current configuration in /var/lib/aws/opsworks/chef/xxxxxxxxx.json. After that, the default OpsWork recipes aws_opsworks_ebshelp us to mount the volume to the configured mount points.

$ cat /var/lib/aws/opsworks/chef/2022-01-06-10-35-28-01.json
...
      "volumes": [
        {
          "name": "Created for nodejs-server2",
          "mount_point": "/mnt/workspace",
          "device": "/dev/sdi",
          "volume_id": "vol-032b79ca4f686b2ee"
        }
      ]
    },

$ cat /var/lib/aws/opsworks/chef/2022-01-06-10-33-35-01.log
...
[2022-01-06T10:33:51+00:00] INFO: Processing ruby_block[delete_lines_from_fstab] action run (aws_opsworks_ebs::default line 1)
[2022-01-06T10:33:51+00:00] INFO: ruby_block[delete_lines_from_fstab] called
[2022-01-06T10:33:51+00:00] INFO: Processing yum_package[xfsprogs] action install (aws_opsworks_ebs::default line 9)
[2022-01-06T10:33:51+00:00] INFO: Processing ruby_block[add xfs to list of known filesystems] action run (aws_opsworks_ebs::default line 16)
[2022-01-06T10:33:51+00:00] INFO: ruby_block[add xfs to list of known filesystems] called
[2022-01-06T10:33:51+00:00] INFO: Processing apt_repository[add_required_repository_for_nvme-cli_for_ubuntu-14.04] action add (aws_opsworks_ebs::default line 23)
[2022-01-06T10:33:51+00:00] INFO: Processing yum_package[nvme-cli] action install (aws_opsworks_ebs::default line 28)
[2022-01-06T10:33:51+00:00] INFO: Processing ebs_volume[vol-032b79ca4f686b2ee] action mount (aws_opsworks_ebs::default line 44)
[2022-01-06T10:33:51+00:00] INFO: Processing execute[mkfs /dev/xvdi] action run (/var/lib/aws/opsworks/cache.internal/cookbooks/aws_opsworks_ebs/resources/ebs_volume.rb line 15)
[2022-01-06T10:33:52+00:00] INFO: execute[mkfs /dev/xvdi] ran successfully
[2022-01-06T10:33:52+00:00] INFO: Processing directory[/mnt/workspace] action create (/var/lib/aws/opsworks/cache.internal/cookbooks/aws_opsworks_ebs/resources/ebs_volume.rb line 29)
[2022-01-06T10:33:52+00:00] INFO: directory[/mnt/workspace] created directory /mnt/workspace
[2022-01-06T10:33:52+00:00] INFO: directory[/mnt/workspace] mode changed to 755
[2022-01-06T10:33:52+00:00] INFO: Processing ruby_block[delete existing fstab entries for this mount point and device] action run (/var/lib/aws/opsworks/cache.internal/cookbooks/aws_opsworks_ebs/resources/ebs_volume.rb line 35)
[2022-01-06T10:33:52+00:00] INFO: ruby_block[delete existing fstab entries for this mount point and device] called
[2022-01-06T10:33:52+00:00] INFO: Processing mount[/mnt/workspace] action mount (/var/lib/aws/opsworks/cache.internal/cookbooks/aws_opsworks_ebs/resources/ebs_volume.rb line 44)
[2022-01-06T10:33:52+00:00] INFO: mount[/mnt/workspace] mounted
[2022-01-06T10:33:52+00:00] INFO: Processing mount[/mnt/workspace] action enable (/var/lib/aws/opsworks/cache.internal/cookbooks/aws_opsworks_ebs/resources/ebs_volume.rb line 44)
[2022-01-06T10:33:52+00:00] INFO: mount[/mnt/workspace] enabled
...

The following is default aws_opsworks_ebs recipe.

$ cat /opt/aws/opsworks/current/cookbooks/aws_opsworks_ebs/recipes/default.rb
ruby_block "delete_lines_from_fstab" do
  block do
    file = Chef::Util::FileEdit.new("/etc/fstab")
    file.search_file_delete_line("/dev/nvme")
    file.write_file
  end
end

package "xfsprogs" do
  # RedHat 6 does not provide xfsprogs
  not_if { rhel6? }
  retries 2
end

# add xfs to list of known filesystems
ruby_block "add xfs to list of known filesystems" do
  block do
    Filesystems.add_xfs_to_known_filesystems
  end
  only_if { ::File.exist?("/etc/filesystems") && node["aws_opsworks_agent"]["resources"]["volumes"].size > 0 }
end

apt_repository 'add_required_repository_for_nvme-cli_for_ubuntu-14.04' do
  only_if { EbsVolumeHelpers.nvme_based? && !EbsVolumeHelpers.has_ebs_tooling? && platform?("ubuntu") && node[:platform_version] == "14.04" && node["aws_opsworks_agent"]["resources"]["volumes"].size > 0 }
  uri 'ppa:sbates'
end

package "nvme-cli" do
  only_if { EbsVolumeHelpers.nvme_based? && !EbsVolumeHelpers.has_ebs_tooling? && !rhel6? && node["aws_opsworks_agent"]["resources"]["volumes"].size > 0 }
  retries 2
end

node["aws_opsworks_agent"]["resources"]["volumes"].each do |volume|
  if rhel6?
    log "skipping volume #{volume["device"]} - no EBS volume support for Red Hat Enterprise Linux 6"
    next
  end

  if volume["mount_point"].nil? || volume["mount_point"].empty?
    log "skip mounting volume #{volume["device"]} (#{volume["volume_id"]}) because no mount_point specified"
    next
  end

  ebs_volume volume["volume_id"] do
    mount_point volume["mount_point"]
    volume_id volume["volume_id"]
    device volume["device"]
    fstype volume["fstype"] || "xfs"
  end
end

References

Editing an OpsWorks Layer's Configuration - EBS Volumes - https://docs.aws.amazon.com/opsworks/latest/userguide/workinglayers-basics-edit.html#workinglayers-basics-edit-ebs

Automatically validating the multiple SAN through DNS validation within multiple hosted zone by using CloudFormation template

Mao-Lin Wang — Tue, 18 May 2021 00:00:00 GMT

Summary

ACM support automatically renews DNS-validated certificates[1]. In this article will go through how to request a public certificate with multiple Subject Alternative Names(SAN)[2], which included the domain name in a public hosted zone and a private hosted zone.

Steps

Create a public hosted zone, and record the hosted zone id.

Create a CloudFormation template with following YAML file.

---
AWSTemplateFormatVersion: '2010-09-09'
Description: Test
Resources:
  MyPrivateHostedZone:
    Type: AWS::Route53::HostedZone
    Properties:
      Name: dev.example.com
      VPCs:
      - VPCId: vpc-000e0266
        VPCRegion: eu-west-1
  MyCertificate:
    Type: AWS::CertificateManager::Certificate
    Properties:
      DomainName: "*.example.com"
      DomainValidationOptions:
      - DomainName: example.com
        HostedZoneId: ZZYK0LLL1NN1XX
      - DomainName: "*.dev.example.com"
        HostedZoneId: !Ref MyPrivateHostedZone
      SubjectAlternativeNames:
      - "*.dev.example.com"
      - "*.example.com"
      ValidationMethod: DNS

Create the the CloudFormation stack.

During the certificate is creating, ACM will create the both CNAME in the hosted zone that specific in CloudFormation template.

$ aws cloudformation describe-stack-events --stack-name my53-certificate
...
{
    "StackId": "arn:aws:cloudformation:eu-west-1:111222333444:stack/ttttteeset/eb245330-b821-11eb-89af-061b29697291",
    "EventId": "MyCertificate-0820989e-77e7-480e-8f57-2b3aaf3d59f4",
    "StackName": "my53-certificate",
    "LogicalResourceId": "MyCertificate",
    "PhysicalResourceId": "",
    "ResourceType": "AWS::CertificateManager::Certificate",
    "Timestamp": "2021-05-18T21:44:15.531000+00:00",
    "ResourceStatus": "CREATE_IN_PROGRESS",
    "ResourceStatusReason": "Content of DNS Record is: {Name: _7758ffb3838c6cf7c3ec68de36d03fe0.example.com.,
Type: CNAME,Value: _3402713d1c23665051ea05c1963caf81.olprtlswtu.acm-validations.aws.}",
    "ClientRequestToken": "Console-CreateStack-b05e9d4a-9da0-dfe3-387c-4890b84060c2"
},
...

Wrapping up

The AWS ACM is supported DNS validation for both public and private hosted zone.
We can add multiple SAN with DomainValidationOptions[3] in CloudFormation.
ACM will create the domain's DNS record for a validation CNAME.

References

Validating domain ownership - https://docs.aws.amazon.com/acm/latest/userguide/domain-ownership-validation.html
Subject Alternative Name - https://en.wikipedia.org/wiki/Subject_Alternative_Name
AWS::CertificateManager::Certificate DomainValidationOption - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-certificatemanager-certificate-domainvalidationoption.html

Push a Helm Chart to AWS ECR

Mao-Lin Wang — Tue, 04 May 2021 00:00:00 GMT

Summary

AWS ECR supports pushing Open Container Initiative (OCI) artifacts[1][2] to your repositories, and this article will go through how to push a Helm chart to ECR. Also, I try to use the Helm Dependency on ECR. The ECR supported the Helm template written in Helm Dependency, but ECR repositories are still not can not add as Helm repo currently.

Testing Steps

Following the steps in document[2] to install the Helm client version 3.

$ helm version

version.BuildInfo{Version:"v3.5.3", GitCommit:"041ce5a2c17a58be0fcd5f5e16fb3e7e95fea622", GitTreeState:"dirty", GoVersion:"go1.15.8"}

Enable OCI support in the Helm 3 client.
```
$ export HELM_EXPERIMENTAL_OCI=1
```

Create a repository to store your Helm chart

$ aws ecr create-repository \
        --repository-name helm-test \
        --region eu-west-1
{
    "repository": {
        "repositoryArn": "arn:aws:ecr:eu-west-1:123456789012:repository/helm-test",
        "registryId": "123456789012",
        "repositoryName": "helm-test",
        "repositoryUri": "123456789012.dkr.ecr.eu-west-1.amazonaws.com/helm-test",
        "createdAt": "2021-05-04T15:07:49+00:00",
        "imageTagMutability": "MUTABLE",
        "imageScanningConfiguration": {
            "scanOnPush": false
        },
        "encryptionConfiguration": {
            "encryptionType": "AES256"
        }
    }
}

Authenticate your Helm client to the Amazon ECR registry to which you intend to push your Helm chart.

$ aws ecr get-login-password \
    --region eu-west-1 | helm registry login \
    --username AWS \
    --password-stdin 123456789012.dkr.ecr.eu-west-1.amazonaws.com
Login succeeded

Use the following steps to create a test Helm chart.

$ mkdir helm-tutorial
$ cd helm-tutorial

$ helm create mychart
Creating mychart

$ rm -rf ./mychart/templates/*

$ cd mychart/templates
$ cat < configmap.yaml
> apiVersion: v1
> kind: ConfigMap
> metadata:
>   name: mychart-configmap
> data:
>   myvalue: "Hello World"
> EOF

Save the chart locally and create an alias for the chart with your registry URI.

$ cd ..
$ helm chart save . mychart
ref:     mychart:0.1.0
digest:  66ac23dc1e383370778e2a8db2bf1d93e73f169af14569618dcec38a086405a4
size:    1.4 KiB
name:    mychart
version: 0.1.0
0.1.0: saved

$ helm chart save . 123456789012.dkr.ecr.eu-west-1.amazonaws.com/helm-test:mychart
ref:     123456789012.dkr.ecr.eu-west-1.amazonaws.com/helm-test:mychart
digest:  2935917ffbc7492eae674f1e82308a2acd4e41b3b6db4c25b79a25095481ebca
size:    1.4 KiB
name:    mychart
version: 0.1.0
mychart: saved

Identify the Helm chart to push. Run the helm chart list command to list the Helm charts on your system.

$ helm chart list
REF                                                             NAME    VERSION DIGEST  SIZE    CREATED
123456789012.dkr.ecr.eu-west-1.amazonaws.com/helm-test:my...    mychart 0.1.0   66ac23d 1.4 KiB 2 minutes
mychart:0.1.0

Push the Helm chart using the helm chart push command.

$ helm chart push 123456789012.dkr.ecr.eu-west-1.amazonaws.com/helm-test:mychart
The push refers to repository [123456789012.dkr.ecr.eu-west-1.amazonaws.com/helm-test]
ref:     123456789012.dkr.ecr.eu-west-1.amazonaws.com/helm-test:mychart
digest:  2935917ffbc7492eae674f1e82308a2acd4e41b3b6db4c25b79a25095481ebca
size:    1.4 KiB
name:    mychart
version: 0.1.0
mychart: pushed to remote (1 layer, 1.4 KiB total)

Describe your Helm chart.

$ aws ecr describe-images \
    --repository-name helm-test \
    --region eu-west-1
{
    "imageDetails": [
        {
            "registryId": "123456789012",
            "repositoryName": "helm-test",
            "imageDigest": "sha256:2935917ffbc7492eae674f1e82308a2acd4e41b3b6db4c25b79a25095481ebca",
            "imageTags": [
                "mychart"
            ],
            "imageSizeInBytes": 1610,
            "imagePushedAt": "2021-05-04T15:16:24+00:00",
            "imageManifestMediaType": "application/vnd.oci.image.manifest.v1+json",
            "artifactMediaType": "application/vnd.cncf.helm.config.v1+json"
        }
    ]
}

Copy the mychart directory to mychart-dependency.
```
$ cp -r mychart/ mychart-dependency
```

Create another new ECR repository named helm-dependency.

$ aws ecr create-repository \
        --repository-name helm-dependency \
        --region eu-west-1

{
    "repository": {
        "repositoryArn": "arn:aws:ecr:eu-west-1:123456789012:repository/helm-dependency",
        "registryId": "123456789012",
        "repositoryName": "helm-dependency",
        "repositoryUri": "123456789012.dkr.ecr.eu-west-1.amazonaws.com/helm-dependency",
        "createdAt": "2021-05-04T15:21:25+00:00",
        "imageTagMutability": "MUTABLE",
        "imageScanningConfiguration": {
            "scanOnPush": false
        },
        "encryptionConfiguration": {
            "encryptionType": "AES256"
        }
    }
}

Update the Chart.yaml and add the configuration for Helm dependencies.

$ cat Chart.yaml
apiVersion: v2
name: mychart-dependency
description: A Helm chart for Kubernetes

type: application
version: 0.1.0
appVersion: "1.16.0"

dependencies:
- name: mychart
  version: "0.1.0"
  repository: "123456789012.dkr.ecr.eu-west-1.amazonaws.com/helm-test:mychart"

Push Helm chart mychart-dependency using the helm chart push command.

$ helm chart save . mychart-dependency
ref:     mychart-dependency:0.1.0
digest:  5d948b5d95b7e455215fd957930c057a2e556b437656713a7deefae0c8017dfa
size:    1.5 KiB
name:    mychart-dependency
version: 0.1.0
0.1.0: saved

$ helm chart save . 123456789012.dkr.ecr.eu-west-1.amazonaws.com/helm-dependency:mychart-dependency
ref:     123456789012.dkr.ecr.eu-west-1.amazonaws.com/helm-dependency:mychart-dependency
digest:  a1200cf17bfae3a733c83b8981da2b2e2b44a79963ffc1dab53cbc58e6f8e8c9
size:    1.5 KiB
name:    mychart-dependency
version: 0.1.0
mychart-dependency: saved

$ helm chart list
REF                                                             NAME                    VERSION DIGEST  SIZE    CREATED
123456789012.dkr.ecr.eu-west-1.amazonaws.com/helm-depende...    mychart-dependency      0.1.0   a1200cf 1.5 KiB 20 seconds
123456789012.dkr.ecr.eu-west-1.amazonaws.com/helm-test:my...    mychart                 0.1.0   a1200cf 1.4 KiB 14 minutes
mychart-dependency:0.1.0                                        mychart-dependency      0.1.0   a1200cf 1.5 KiB 58 seconds
mychart:0.1.0                                                   mychart                 0.1.0   a1200cf 1.4 KiB 15 minutes

$ helm chart push 123456789012.dkr.ecr.eu-west-1.amazonaws.com/helm-dependency:mychart-dependency
The push refers to repository [123456789012.dkr.ecr.eu-west-1.amazonaws.com/helm-dependency]
ref:     123456789012.dkr.ecr.eu-west-1.amazonaws.com/helm-dependency:mychart-dependency
digest:  a1200cf17bfae3a733c83b8981da2b2e2b44a79963ffc1dab53cbc58e6f8e8c9
size:    1.5 KiB
name:    mychart-dependency
version: 0.1.0
mychart-dependency: pushed to remote (1 layer, 1.5 KiB total)

Helm dependency is using the "repository" URL should point to a Chart Repository[4], which is ECR not supported yet currently[5].

References

OCI Artifact Support In Amazon ECR - https://docs.aws.amazon.com/AmazonECR/latest/userguide/push-oci-artifact.html
ECR supports pushing Open Container Initiative (OCI) artifacts to your repositories - https://docs.aws.amazon.com/AmazonECR/latest/userguide/push-oci-artifact.html
Using Amazon ECR Images with Amazon EKS - Installing a Helm chart hosted on Amazon ECR with Amazon EKS - https://docs.aws.amazon.com/AmazonECR/latest/userguide/ECR_on_EKS.html#using-helm-charts-eks
Helm Dependency - https://helm.sh/docs/helm/helm_dependency/
Allow Helm to automatically install from a chart stored in an ECR repository - https://github.com/aws/containers-roadmap/issues/1116

EKS service with internal NLB, but gets timeouts randomly

Mao-Lin Wang — Fri, 23 Apr 2021 00:00:00 GMT

Summary

While deploying both client and server on their EKS cluster and connecting through internal NLB, however, the communication gets timeout randomly.

Problem

In Kubernetes, we can expose the pod to cluster with Kubernetes Service. For sure, we can create a Kubernetes Service of type LoadBalancer, AWS NLB, or CLB is provisioned that load balances network traffic.

Figure source: https://www.docker.com/blog/designing-your-first-application-kubernetes-communication-services-part3/

To manage the workload easily, we might want to deploy both the client and server sides on Kubernetes. However, in the following conditions, you might notice the timeout issue happen randomly.

This use case includes both server-side and client-side in the same EKS cluster, and the server-side must use Kubernetes service with internal NLB - instance target type.

Create a Kubernetes Deployment as server-side. For example, self-host Redis, Nginx server, etc.
Create a Kubernetes Service to expose the server application through the internal NLB.
Create another Kubernetes Deployment to test the application as client-side, and try to connect with the server's service.

Reproduce

In my testing, I launched the Kubernetes cluster with Amazon EKS (1.8.9) with default deployments (such as CoreDNS, AWS CNI Plugin, and kube-proxy). The issue can be reproduced as the steps below:

Deploy a Nginx Deployment and a troubleshooting container. I used the netshoot as troubleshooting pod, which had preinstalled the curl command. Also, you can find the nginx-nlb-service.yaml and netshoot.yaml in the attachments.
```
$ kubectl apply -f ./netshoot.yaml
$ kubectl apply -f ./nginx-nlb-service.yaml
```
Use the kubctl exec command to attach into troubleshooting(netshoot) container.
```
$ kubectl exec -it netshoot -- bash
bash-5.1#
```

Access the internal NLB domain name(Nginx's Service) via curl command, the connection will timeout randomly. At the same time, I capture the packets through with tcpdump command in EKS worker node.

# In the EKS worker node(192.168.2.3)
$ sudo tcpdump -i any  -w packets.pcap

bash-5.1# date
Mon Mar 22 23:35:13 UTC 2021

bash-5.1# curl a7487a27201f2434490bada8096adce3-221f6e1d21fadd5b.elb.eu-west-1.amazonaws.com
... It will timeout randomly, you might need to try more times.
...

$ date
Mon Mar 22 23:35:32 UTC 2021

Why do we get the timeout from the Internal NLB

Firstly, we must know the NLB introduced the source IP address preservation feature[3]: the original IP addresses and source ports for the incoming connections remain unmodified. When the backend answers a request, the VPC internals capture this packet and forward it to the NLB, which will forward it to its destination.

Therefore, if the worker node is running the client-side, and NLB forward to the same nodes. It will generate a random connection timeout. We can dive into this process:

The troubleshooting container starts from an ephemeral port and sends a SYN packet.
- src: 192.168.27.239:49134
- dst: 192.168.168.103:80
The NLB receives the SYN packet and forwards it to the backend EKS worker node 192.168.2.3 with target port 31468, which was registered by Kubernetes Service. NLB modified the destination IP address as Client IP preservation feature. Thus, the EKS worker node receives a SYN packet:
- src: 192.168.27.239:49134
- dst: 192.168.2.3:31468

Base on the iptables rules(Nginx Service) on the EKS worker node, this SYN packet was forwarded to the Nginx pod.

src: 192.168.2.3:19969

dst: 192.168.16.142:80

# Kubernetes Service will update the following iptables rules in every node.
$ sudo iptables-save | grep "service-nginx-demo"
-A KUBE-NODEPORTS -p tcp -m comment --comment "nginx-demo/service-nginx-demo:" -m tcp --dport 31919 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "nginx-demo/service-nginx-demo:" -m tcp --dport 31919 -j KUBE-SVC-7D7VEXWNCKBQRZ7W
-A KUBE-SEP-2IF7DICDPRGPK5UI -s 192.168.39.2/32 -m comment --comment "nginx-demo/service-nginx-demo:" -j KUBE-MARK-MASQ
-A KUBE-SEP-2IF7DICDPRGPK5UI -p tcp -m comment --comment "nginx-demo/service-nginx-demo:" -m tcp -j DNAT --to-destination 192.168.39.2:80
-A KUBE-SEP-2V4THTSJVFKRX4LC -s 192.168.16.142/32 -m comment --comment "nginx-demo/service-nginx-demo:" -j KUBE-MARK-MASQ
-A KUBE-SEP-2V4THTSJVFKRX4LC -p tcp -m comment --comment "nginx-demo/service-nginx-demo:" -m tcp -j DNAT --to-destination 192.168.16.142:80
-A KUBE-SEP-73EDW25F4C3YFWYZ -s 192.168.49.70/32 -m comment --comment "nginx-demo/service-nginx-demo:" -j KUBE-MARK-MASQ
-A KUBE-SEP-73EDW25F4C3YFWYZ -p tcp -m comment --comment "nginx-demo/service-nginx-demo:" -m tcp -j DNAT --to-destination 192.168.49.70:80
-A KUBE-SERVICES -d 10.100.235.20/32 -p tcp -m comment --comment "nginx-demo/service-nginx-demo: cluster IP" -m tcp --dport 80 -j KUBE-SVC-7D7VEXWNCKBQRZ7W
-A KUBE-SVC-7D7VEXWNCKBQRZ7W -m comment --comment "nginx-demo/service-nginx-demo:" -m statistic --mode random --probability 0.33333333349 -j KUBE-SEP-2IF7DICDPRGPK5UI
-A KUBE-SVC-7D7VEXWNCKBQRZ7W -m comment --comment "nginx-demo/service-nginx-demo:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-73EDW25F4C3YFWYZ
-A KUBE-SVC-7D7VEXWNCKBQRZ7W -m comment --comment "nginx-demo/service-nginx-demo:" -j KUBE-SEP-2V4THTSJVFKRX4LC

Nginx pod replies the SYN-ACK packet back to the EKS worker node. We can view the tcp.stream eq 9 for step 3 and step4.

VPC CNI plugin maintains the route table for binding the ENI with pod. Base on the route table in Node 1, it will reply the SYN-ACK packet for the SYN packet for troubleshooting pod in step 2.

# Node 1 - routing table

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.32.1    0.0.0.0         UG    0      0        0 eth0
169.254.169.254 0.0.0.0         255.255.255.255 UH    0      0        0 eth0
192.168.32.0    0.0.0.0         255.255.224.0   U     0      0        0 eth0
192.168.27.239  0.0.0.0         255.255.255.255 UH    0      0        0 eni708fb089496
...

The troubleshooting pod notices this SYN-ACK connection is an abnormal connection, and send RST packet to close the connection to 192.168.2.3:31468.
The RST packet is forwarded to the Nginx pod through iptables rules(Nginx server) as well. Also, the troubleshooting container will send both TCP_RETRANSMISSION and RST several times until it times out.
The initiating socket(src: 192.168.27.239:49134, dst: 192.168.168.103:80) still expects the SYN-ACK from the NLB 192.168.168.103:80. However, the troubleshooting pod did not receive the SYN-ACK, so the troubleshooting pod will send several TCP RETRANSMISSION - SYN until it times out.

Workaround

If you don't care about keeping the source IP address, we can suggest that you can consider using the NLB IP mode[4].

metadata:
      name: my-service
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: "nlb-ip"

2021-05-21 Update

The NLB adds support to configure client IP preservation[5] with instance mode, which is supported in version v2.2.0[6]. Thus, we can use the following annotations for the client IP preservation with instance mode.

Note: Please use the annotation service.beta.kubernetes.io/aws-load-balancer-type: "external" to ignore in-tree cloud-controller-manager[7] to create an in-tree NLB service.

...
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "external"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
    service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: "preserve_client_ip.enabled=false"
...

References

Connecting Applications with Services - https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/
Network load balancing on Amazon EKS - https://docs.aws.amazon.com/eks/latest/userguide/load-balancing.html
Target groups for your Network Load Balancers - Client IP preservationhttps://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#client-ip-preservation
https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/service/nlb_ip_mode/#nlb-ip-mode
https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#client-ip-preservation
AWS Load Balancer Controller - v2.2.0 - https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases/tag/v2.2.0
AWS Load Balancer Controller - Network Load Balancer - https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/guide/service/nlb/#configuration

WordPress Tuning

Mao-Lin Wang — Sun, 03 Mar 2019 22:38:00 GMT

前言

以前在兼差幫忙管理系所網站時，那時候學校主流 CMS 使用的是 Joomla，當年也從 apache2 移轉使用 NGINX 做了些微的調教，但是也沒有認真做個筆記。剛好公司的官網也要從 windows 移轉到 linux 伺服器上，不過使用的是 WordPress，也稍微 survey 了相關調教，本篇為移轉過程及相關調教筆記。

安裝與移轉

使用伺服器架構為基本的 LNMP，可以參考 DigitalOcean - How To Install Linux, Nginx, MySQL, PHP (LEMP stack) in Ubuntu 16.04，額外安裝的 php 套件有 php7.0-curl、php7.0-zip。

$ apt install php7.0-curl php7.0-zip

比較麻煩的點在於 windows 換行符號為 ^M ，因此必須把每個檔案的換行符號切換成 linux 的換行符號。

# 替換換行符號
find . -type f -exec dot2unix {} \;
# 確保資料夾及檔案權限
find . -type d -exec chmod 755 {} \;
find . -type f -exec chmod 644 {} \;

Nginx 設定

gzip 及 cache


# gzip
gzip on;
gzip_static on;
gzip_disable "msie6";

gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_buffers 16 8k;
gzip_http_version 1.1;
# Don't compress files smaller than 256 bytes, as size reduction will be negligible.
gzip_min_length 256;
gzip_types
    application/atom+xml
    application/javascript
    application/json
    application/ld+json
    application/manifest+json
    application/rss+xml
    application/vnd.geo+json
    application/vnd.ms-fontobject
    application/x-font-ttf
    application/x-web-app-manifest+json
    application/xhtml+xml
    application/xml
    font/opentype
    image/bmp
    image/svg+xml
    image/x-icon
    text/cache-manifest
    text/css
    text/plain
    text/vcard
    text/vnd.rim.location.xloc
    text/vtt
    text/x-component
    text/x-cross-domain-policy;

# cache
open_file_cache max=100000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 2;
open_file_cache_errors on;

網站設定

參考官網 Wordpress for Nginx config

General WordPress rules：一般 WordPress 及 php 相關設定。
Global restrictions file：不該開放權限給人存取相關設定。

Tuning

NGINX 網站設定

盡可能的 cache 一些靜態檔案。

location ~* \.(ogg|ogv|svg|svgz|eot|otf|woff|mp4|ttf|css|rss|atom|js|jpg|jpeg|gif|png|ico|zip|tgz|gz|rar|bz2|doc|xls|exe|ppt|tar|mid|midi|wav|bmp|rtf)$ {
    expires max;
    log_not_found off;
    access_log off;
}

HTTP2 設定。

WordPress Plugin

WP Super Cache： WordPress 仰賴的是 php 動態存取資料庫產生出對應的 HTML 頁面呈現，Super Cache 功能先把一些常用的網頁先產出成 Static HTML 檔案暫存起來，讓使用者存取時可以直接存取。依照官網教學安裝方式即可。
Fast Velocity Minify：把散落各個 plugin 的 css, js merge 並 minify，同時也有將 WordPress 版本號的檔案也一併整理壓縮達到隱藏版本號的效果。
WP SMush：壓縮上傳的 image，免費版則是每次幫你壓縮 50 張圖片，每次都得自己重複點擊壓縮。
All in One SEO：在我接收以前就已經設定好了，但是也幫助了網站 SEO 評分，每個頁面也都可以設定對應 SEO tag 及 Sitemap 自動產出，大幅提高了網站被搜尋到的排名。

安全性

Nginx

location ~* ^/wp-content/uploads/.*.(html|htm|shtml|php|js|swf)$ {
    deny all;
}

location ~ /(wp-config|xmlrpc).php$ {
    deny all;
}

location ~ /\.ht {
    deny all;
}

# 只允許 localhost 可以登入及查看 admin 後台
location ~ /(wp-admin|wp-login\.php)$ {
    include snippets/fastcgi-php.conf;
    fastcgi_pass unix:/run/php/php7.0-fpm.sock;
    allow 127.0.0.1;
    deny  all;
}

SSL

參考 Cipherli.st - Strong Ciphers for Apache, nginx and Lighttpd。

WordPress、Joomla 相關套件大量使用了 iFrame 遷入在 plugin 設定，因此以下這個設定可以看有沒有影響操作斟酌加入設定檔。

add_header X-Frame-Options DENY;

測效網站

GTMetrix：提供 PageSpeed 和 YSlow 個項目評分，可以查看有哪些可能可以改善的圖片壓縮，或是伺服器沒有設定 cache 等驗證。
Lighthouse： Google Opensource 的自動化工具，但是滿多都以非常「嚴格」的標準來查看網頁存取速度，也提供了 Progressive Web App 的檢驗。

focaaby's Notes

為什麼 EKS cluster 知道預設 CNI Plugin 為 Amazon VPC CNI plugin

work node

透過 EC2 啟用 EKS AMI 的 node

已經有 VPC CNI plugin Pod 的 node

那為什麼需要這樣設定一個 entrypoint script 安裝 CNI Binary 呢？

API server

總結

參考文件

為什麼 EKS worker node 可以自動加入 EKS cluster（二）？

kubelet bootstrap 初始化步驟

驗證 EKS 設定

kubelet

kube-apiserver

kube-controller-manager

實際啟用 EKS 節點

總結

參考文件

為什麼 EKS worker node 可以自動加入 EKS cluster（一）

EKS node group

Amazon EKS AMI Build Specification

總結

參考文件

為什麼 kubectl 可以訪問 EKS cluster

AWS IAM Authenticator

驗證

kubectl

Control plane

總結

參考文件

建立 EKS Cluster

前言

建立 EKS Cluster 環境

參考文件

How to control your external display brightness on macOS and Arch Linux

Preface

DDC/CI

macOS

Linux - Arch Linux

References

Why logging system collects container logs from specific directory

Logging Architecture

Logging at the node level

Cluster-level logging architectures

Using a node logging agent

Using a sidecar container with the logging agent

Why we collects logs from /var/log/containers/ directory

Flutend

Fluent Bit

Summary

參考文件

Why Linux OS takes over 5 minutes during boot-up process

Problem

Find the systemd unit that take a long time

Summary

References

AWS VPC CNI plugin random livenessProbe failures after upgrading to Kubernetes 1.20

Problem

What changes in Kubernetes 1.20

Workaround

Reference

Why my CDK VPC constructor does not respect maximum availability zones

Problem

Dive into the problem

Summary

References

How OpsWorks mount EBS Volume automatically

References

Automatically validating the multiple SAN through DNS validation within multiple hosted zone by using CloudFormation template

Summary

Steps

Wrapping up

References

Push a Helm Chart to AWS ECR

Summary

Testing Steps

References

EKS service with internal NLB, but gets timeouts randomly

Summary

Problem

Why we collects logs from `/var/log/containers/` directory