kubernetes

install kubernetes cluster 1.25.2 on local ubuntu, centos

PSAwesome 2022. 10. 9. 01:12

➜ ~ k version --client --short

Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.

Client Version: v1.25.2

Kustomize Version: v4.5.7

➜ ~ kubelet --version

Kubernetes v1.25.2

➜ ~ kubeadm version

kubeadm version: &version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.2", GitCommit:"5835544ca568b757a8ecae5c153f317e5736700e", GitTreeState:"clean", BuildDate:"2022-09-21T14:32:18Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"linux/amd64"}

➜ ~ sudo containerd -version

containerd containerd.io 1.6.8 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6

➜ ~ docker --version

Docker version 20.10.18, build b40c2f6

MASTER Ubuntu

➜ ~ lsb_release -a

No LSB modules are available.

Distributor ID: Ubuntu

Description: Ubuntu 22.04.1 LTS

Release: 22.04

Codename: jammy

WORKER CentOs

➜ ~ rpm --query centos-release

centos-release-7-9.2009.1.el7.centos.x86_64

➜ ~ cat /etc/centos-release

CentOS Linux release 7.9.2009 (Core)

control plane ports

protocol	방향	포트 범위	용도	사용 주체
TCP	인바운드	6443	쿠버네티스 API 서버	전부
TCP	인바운드	2379-2380	etcd 서버 클라이언트 API	kube-apiserver, etcd
TCP	인바운드	10250	Kubelet API	Self, 컨트롤 플레인
TCP	인바운드	10259	kube-scheduler	Self
TCP	인바운드	10257	kube-controller-manager	Self

worker node ports

protocol	방향	포트 범위	용도	사용 주체
TCP	인바운드	10250	Kubelet API	Self, 컨트롤 플레인
TCP	인바운드	30000-32767	NodePort 서비스†	전부

ubuntu

ufw

vim /etc/ufw/applications.d/kubernetes-profiles

# body

[k8s-master]
title=master
description=required master port api, etc client api, kubelet api, kube-scheduler, kube-controller-manager
ports=6443,8080,10250,10259,10257/tcp|2379:2380/tcp

sudo ufw allow from 172.30.1.0/24 to any app k8s-master && sudo ufw reload

sudo ufw status

change hostname

hostnamectl set-hostname ps-master
exec bash

swap off

swapoff -a
sudo sed -i '/swap/d' /etc/fstab

cgroup

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# 필요한 sysctl 파라미터를 설정하면, 재부팅 후에도 값이 유지된다.
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# 재부팅하지 않고 sysctl 파라미터 적용하기
sudo sysctl --system

sudo apt install -y curl gnupg2 software-properties-common apt-transport-https ca-certificates

add apt repository for docker

sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmour -o /etc/apt/trusted.gpg.d/docker.gpg
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

containerd usin systemd cgroup

containerd config default | sudo tee /etc/containerd/config.toml >/dev/null 2>&1
sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml

containerd restart

sudo systemctl restart containerd
sudo systemctl enable containerd

add apt repository for kubernetes

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"

install kubernetes components kubectl, kubeadm, kubelet

sudo apt update
sudo apt install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

sudo kubeadm init --control-plane-endpoint=ps.k8smaster.org --pod-network-cidr=10.224.0.0/16

mkdir -p $HOME/.kube &&
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config &&
sudo chown $(id -u):$(id -g) $HOME/.kube/config

network add-on : wave net

kubectl apply -f https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s.yaml

uninstall wave net

sudo rm /opt/cni/bin/weave-*

ip link delete flannel.1

별첨 (설치 도중에 발생한 오류 내용)

아래와 같은 에러가 확인되는 경우

아래 명령어 입력 후 init을 다시 해줍니다.

# no such file or success deleted
sudo rm /etc/containerd/config.toml

sudo systemctl restart containerd
sudo kubeadm init

kinit 이후 --token과 --discovery-token-ca-cert-hash 값이 확인되는데, 백업이 필요합니다.

- root에서 config 복사 및 소유자 변경

# To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

# Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

- user에서 config 복사

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

network add-on 재설치 수 차례(calico, flannel, wave)
kubelet 재시작을 수 차례
kubeadm reset, init을 수 차례

마지막 루틴은 아래와 같습니다.

1. network add-on 설치
2. docker, containerd, kubelet 재시작
3. node ready 확인
4. coredns pending -> create 확인

sudo systemctl restart kubelet containerd docker

kubectl get nodes

zsh auto-completion

The kubectl completion script for Zsh can be generated with the command kubectl completion zsh. Sourcing the completion script in your shell enables kubectl autocompletion.

To do so in all your shell sessions, add the following to your ~/.zshrc file:

source <(kubectl completion zsh)

아래와 같은 오류가 뜬다면

The connection to the server 192.168.1.0:6443 was refused - did you specify the right host or port?

root가 아닌 유저로 `- config 복사 및 소유자 변경`이 정상적으로 됐는지 체크
user에서 export를 한번 더 하진 않았는지 체크.
Docker가 정상적으로 실행됐는지 체크.
user가 docker 명령어 입력이 가능한지.
1. `sudo chown $(id -u):$(id -g) /var/run/docker.sock`
2. Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json": dial unix /var/run/docker.sock: connect: permission denied

install calico pod network add-on

curl https://projectcalico.docs.tigera.io/manifests/calico.yaml -O
kubectl apply -f calico.yaml

kubectl get pods -n kube-system --watch

reference

https://www.linuxtechi.com/install-kubernetes-on-ubuntu-22-04/

centos 7

firewall

iptables-save | grep 10250

master node ports

sudo firewall-cmd --add-port={6443,2379-2380,10250,10251,10252,5473,179,5473}/tcp --permanent
sudo firewall-cmd --add-port={4789,8285,8472}/udp --permanent
sudo firewall-cmd --reload

worker node ports

sudo firewall-cmd --permanent --add-port={10250,30000-32767,5473,179,5473}/tcp && 
sudo firewall-cmd --permanent --add-port={4789,8285,8472}/udp &&
sudo firewall-cmd --reload

sudo firewall-cmd --list-ports

change hostname

sudo hostnamectl set-hostname ps.k8sworker.org
hostname

add the following entries in /etc/hosts file on each node

172.30.1.34 ps.k8smaster.org
172.30.1.11 ps.k8sworker.org

install kubelet, kubeadm and kubectl

add kubernetes repository

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

# Set SELinux in permissive mode (effectively disabling it)
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

- you have to do this until SELinux support is improved in the kubelet

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

sudo systemctl enable --now kubelet
sudo systemctl start kubelet

install

sudo yum clean all && sudo yum -y makecache
sudo yum -y install epel-release vim git curl wget kubeadm kubectl --disableexcludes=kubernetes

disabled swap

swapoff -a
sudo sed -i '/swap/d' /etc/fstab

disable SELinux

sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

configure sysctl

sudo modprobe overlay
sudo modprobe br_netfilter

sudo tee /etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

sudo sysctl --system

docker container runtime

# remove
sudo yum remove docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-engine
                  
                  
# install
sudo yum install -y yum-utils

# add docker repository
sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

# install docker container
sudo yum install docker-ce docker-ce-cli containerd.io docker-compose-plugin

# specify version
# VERSION_STRING=20.10.9
# sudo yum install docker-ce-$VERSION_STRING docker-ce-cli-$VERSION_STRING containerd.io docker-compose-plugin

# root
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/kubelet.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config


# for user
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/kubelet.conf $HOME/.kube/config

# change own
sudo chown $(id -u):$(id -g) $HOME/.kube/config
sudo chown $(id -u):$(id -g) /var/lib/kubelet/pki/kubelet-client-current.pem

finally...

#etc

change hostname

kubectl delete node <original-nodename>
kubeadm reset
hostnamectl set-hostname [changed hostname]
kubeadm init--control-plane-endpoint=[changed hostname]

NotReady issue

you can see kubelet log

journalctl -u kubelet

remove all then sudo kubeadm init ...

# step 1
sudo kubeadm reset
sudo systemctl stop docker && sudo systemctl stop kubelet

# step 2
sudo rm -rf /etc/kubernetes/

sudo rm -rf .kube/
sudo rm -rf /var/lib/kubelet/
sudo rm -rf /var/lib/cni/
sudo rm -rf /etc/cni/
sudo rm -rf /var/lib/etcd/

# step3 - Bootstrap K8s cluster via kubeadm:

sudo systemctl start docker && 
sudo systemctl start kubelet && 
systemctl daemon-reload

init kubernetes

sudo kubeadm init --control-plane-endpoint=ps.k8smaster.org

centos all delete

sudo kubeadm reset

sudo systemctl stop docker && sudo systemctl stop kubelet

sudo yum remove -y kubeadm kubectl kubernetes-cni kubelet kube*

sudo rm -rf ~/.kube
sudo rm -rf /etc/kubernetes /var/lib/kubelet /var/lib/etcd /var/lib/cni/
sudo rm -rf /etc/cni/

occurred error while cluster join

msg="getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"

you can remove containerd config and restart containerd then join

sudo rm /etc/containerd/config.toml
sudo systemctl restart containerd

kubeadm join ps.k8smaster.org:6443 --token 8ghs5q.fig8uafkna9pwwbz \
    --discovery-token-ca-cert-hash sha256:3518c28e8bba2a64ddc5abe6ca243b350e805a4ffb4bb9f6bb0639dcafa44b08

network error

journalctl -f -u kubelet | egrep "kubelet.go|cni.go"

kubectl describe node ps.k8sworker.org

false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

Switch to root: su - root

Uninstall k8s
(Although on master node, I did this a few times and included draining the node the last time)

kubectl drain mynodename --delete-local-data --force --ignore-daemonsets
kubectl delete node mynodename
kubeadm reset
systemctl stop kubelet
yum remove kubeadm kubectl kubelet kubernetes-cni kube*
yum autoremove
rm -rf ~/.kube
rm -rf /var/lib/kubelet/*

Uninstall docker:

docker rm docker ps -a -q``
docker stop (as needed)
docker rmi -f docker images -q``
Check that all containers and images were deleted: docker ps -a; docker images
systemctl stop docker
yum remove yum-utils device-mapper-persistent-data lvm2
yum remove docker docker-client docker-client-latest docker-common docker-latest docker-latest-logrotate docker-logrotate docker-selinux docker-engine-selinux docker-engine
yum remove docker-ce
rm -rf /var/lib/docker
12. rm -rf /etc/docker

Uninstall flannel

rm -rf /var/lib/cni/
rm -rf /run/flannel
rm -rf /etc/cni/
Remove interfaces related to docker and flannel:
ip link
For each interface for docker or flannel, do the following
ifconfig <name of interface from ip link> down
ip link delete <name of interface from ip link>

마지막 정리

pending 또는 에러 로그를 하나씩 보는게 중요하다.

kubectl describe node ps.k8smaster.org
kubectl describe pod coredns-565d847f94-rx7kw -n kube-system
파일 이름이 달라서 안되는 경우 (버전 상이)
network add-on이 정상 실행되지 않는 경우 (flannel을 읽고, running은 wavenet인 경우 등)
kubelet을 아무리 재시작 해도 로그가 바뀌지 않는다면 containerd, docker를 재실행 (이상한 경우)

1.25.2 network add-on plugin 내부를 수정하지 않고 쓰는 건 calico가 되더라..

(flannel, wavenet, cni 등등 해봄.. 제가 잘 몰라서)

worker node는 명령어를 날리지 않는다 (kubernetes 설정이 굳이 필요 없다.)

`/run/systemd/resolve/resolv.conf` 는 손으로 추가했습니다.

cluster 구축할 때 os는 가능하면 통일하자 (애매하게 달라서 고통 가중)

centos 7
ubuntu 22.04

ansible을 활용하자.

이렇게 기록한들 보기 힘들고, 새로 구축할 때 결국 고생이 반복될 것

network add-on 수정 후에도 지속적으로 node가 notReady 상태인 경우 세 개를 리셋시키자

docker
containerd
kubelet

master 하나만 띄우는 경우 Taint 설정을 먼저 바꾸자

master node에 배포하지 않는 정책을 배포하도록 수정하자.
1.22 < version 인 경우 `master`가 아닌 `control-plane`이다.
위 예제에서 `kubectl taint node --all node-role.kubernetes.io/control-plane:NoSchedule-`
문법 `kubectl taint nodes node1 key1=value1:NoSchedule` (master는 node-role.kubernetes.io 정책이 NoSchedule임)

nginx 테스트 띄우고 local에서 접속, scale out 테스트한 결과입니다.

참조

저작자표시 동일조건

'kubernetes' 카테고리의 다른 글

installation kubernetes by kubespray on-premise (0)	2022.12.09
delete kubernetes all (0)	2022.10.09
쿠버네티스 버전 낮추기(설정하기) - minikube (0)	2022.08.15
kubernetes dashboard 생성 (0)	2022.08.12
kubernetes localhost 연결 - sonarqube (0)	2022.07.21

현재글install kubernetes cluster 1.25.2 on local ubuntu, centos

현실과 긍정의 경계에서 긍정을 바라보려고 노력하는 사람

k8s, kubectl, logstash, install, Spring Boot, error, elasticsearch, Helm, filebeat, IntelliJ, Elk, GitHub, ubuntu, docker-compose, GCP, docker hub, spark, ssh, kubernetes, docker,

Today :
Yesterday :

지혜를 창출할 창고

install kubernetes cluster 1.25.2 on local ubuntu, centos

control plane ports

worker node ports

ubuntu

ufw

change hostname

swap off

cgroup

add apt repository for docker

add apt repository for kubernetes

install kubernetes components kubectl, kubeadm, kubelet

zsh auto-completion

install calico pod network add-on

centos 7

firewall

change hostname

install kubelet, kubeadm and kubectl

disabled swap

finally...

마지막 정리

'kubernetes' 카테고리의 다른 글

'kubernetes'의 다른글

티스토리툴바

« 2024/06 »
일	월	화	수	목	금	토
						1
2	3	4	5	6	7	8
9	10	11	12	13	14	15
16	17	18	19	20	21	22
23	24	25	26	27	28	29
30

install kubernetes cluster 1.25.2 on local ubuntu, centos

control plane ports

worker node ports

ubuntu

ufw

change hostname

swap off

cgroup

add apt repository for docker

add apt repository for kubernetes

install kubernetes components kubectl, kubeadm, kubelet

zsh auto-completion

install calico pod network add-on

centos 7

firewall

change hostname

install kubelet, kubeadm and kubectl

disabled swap

finally...

마지막 정리

'kubernetes' 카테고리의 다른 글

'kubernetes'의 다른글

관련글

티스토리툴바