Contents

kubespray本质是一堆ansible的role文件,通过这种方式,即ansible方式可以自动化的安装高可用k8s集群,目前支持v2.12.7

1.kubespray部署kubernetes高可用集群 ¶

参考文献

https://blog.csdn.net/networken/article/details/106623080

使用 kubeplay 来离线部署 kubernetes 集群

https://blog.k8s.li/deploy-k8s-by-kubeplay.html

2. 环境需求 ¶

环境	版本
Centos	centos 7.6
Kernel	4.4.224-1.el7.elrepo.x86_64
kubernetes	v1.16.9
kubespray	v2.12.6
Docker	v19.03.9

硬件需求：CPU>=2c ,内存>=2G

3. 环境角色 ¶

主机名 \| IP		角色	安装软件
i1-ma ster- 1	172.16 .60.17 8	k8s-Ma ster01	ansible kubespray etcd ingress-nginx calico kube-apiserver kube-controller-manager kube-proxy kube-scheduler nodelocaldns node-exporte kubelet
i1-wo rker- 1	172.16 .60.22 6	k8s-no de01	calico kube-proxy nginx-proxy nodelocaldns node-exporte kubelet
i1-wo rker- 2	172.16 .60.9	k8s-no de02	calico kube-proxy nginx-proxy nodelocaldns node-exporte kubelet

4. 系统环境初始化 ¶

1.设置主机名和hosts

hostnamectl set-hostname i1-master-1
hostnamectl set-hostname i1-worker-1
hostnamectl set-hostname i1-worker-2

2.配置hosts

cat >> /etc/hosts <<'EOF'
172.16.60.178 i1-master-1
172.16.60.226 i1-worker-1
172.16.60.9   i1-worker-2
EOF

3.关闭防火墙,防火墙（可选）和网络设置，所有的主机都执行以下命令：

systemctl disable firewalld && systemctl stop firewalld && systemctl status firewalld
modprobe br_netfilter
echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables
sysctl -w net.ipv4.ip_forward=1

4.关闭swap分区

#临时
swapoff -a && echo "vm.swappiness=0" >> /etc/sysctl.conf && sysctl -p && free –h

#永久
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

5.关闭selinux:

setenforce  0
sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/sysconfig/selinux

sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/sysconfig/selinux

sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/selinux/config

6.设置内核—将桥接的IPv4流量传递到iptables的链

modprobe br_netfilter

cat <<EOF >> /etc/sysctl.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

sysctl -p /etc/sysctl.conf

7.设置系统同步时间服务器

yum install -y ntp

echo "0 6 * * *  /usr/sbin/ntpdate pool.ntp.org  >/dev/null 2>&1" >> /var/spool/cron/root

8.配置ssh key 免密认证

ssh-keygen -t rsa -N ""
cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys

#配置其他节点免密登录
ssh-copy-id -i /root/.ssh/id_rsa.pub root@i1-worker-1
ssh-copy-id -i /root/.ssh/id_rsa.pub root@i1-worker-2

9.设置k8s内核参数

sudo cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.ip_forward = 1
vm.swappiness=0
EOF

10.由于kubespay安装方式会检测docker是否安装，如果没有安装会安装docker，但是使用的源是https://yum.dockerproject.org/repo/main/centos/7，速度会比较慢，建议提前安装好。

使用阿里云yum镜像,docker安装速度快

#docker yum源
cat >> /etc/yum.repos.d/docker.repo <<EOF
[docker-repo]
name=Docker Repository
baseurl=http://mirrors.aliyun.com/docker-engine/yum/repo/main/centos/7
enabled=1
gpgcheck=0
EOF

同时配置好阿里云加速器

mkdir -p /etc/docker
tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://5md0553g.mirror.aliyuncs.com"]
}
EOF

11 .手动安装docker:

#查看docker版本
yum list docker-engine –showduplicates
#安装docker
yum install -y docker-engine-1.13.1-1.el7.centos.x86_64

12.更新系统内核为 4.4.x , CentOS 默认为3.10.x

#安装：
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm

yum --enablerepo=elrepo-kernel install -y kernel-lt kernel-lt-devel

grub2-set-default 0

#重启操作系统：
reboot

5. 部署Kubespray基础环境 ¶

1.安装 python 及 epel (在Ansible主机上安装并配置好与各node的免秘钥登录)

#master任意一台机器上操作
#安装ansible
#----ansible 必须 >= 2.7
yum install -y epel-release
yum install -y vim wget python-pip ansible python36 git  python36-pip

2.取消key检查：

vim /etc/ansible/ansible.cfg

host_key_checking = False

Linux中修改pip源

mkdir ~/.pip
cat > ~/.pip/pip.conf << EOF
[global]
trusted-host=mirrors.aliyun.com
index-url=https://mirrors.aliyun.com/pypi/simple/
EOF

3.安装jinja2

#master任意一台机器上操作
pip install --upgrade pip && pip install netaddr \
 && pip install --upgrade jinja2

6. kubespray安装部署k8s ¶

克隆项目kubespray

wget https://github.com/kubernetes-incubator/kubespray/archive/v2.12.4.tar.gz
tar -zxvf v2.12.4.tar.gz
mv kubespray-2.12.4 kuberspray

Install dependencies from requirements.txt

sudo /usr/bin/pip3.6 install -r requirements.txt

Copy inventory/sample as inventory/mycluster

cp -rfp inventory/sample inventory/mycluster

Update Ansible inventory file with inventory builder

declare -a IPS=(172.16.60.226 172.16.60.9)

CONFIG_FILE=inventory/mycluster/hosts.yaml /usr/bin/python3.6 contrib/inventory_builder/inventory.py ${IPS[@]}

cp -rf inventory/mycluster/inventory.ini{,_.bak}

vim inventory/mycluster/inventory.ini

[all]
node1    ansible_host=172.16.10.226 ip=172.16.10.226
node2    ansible_host=172.16.10.9 ip=172.16.10.9

[kube-master]
node1

[kube-node]
node1
node2

[etcd]
node1

[k8s-cluster:children]
kube-node
kube-master

[calico-rr]

[vault]
node1

6.1 替换镜像 ¶

在kuberspay源码源代码中搜索包含 gcr.io/google_containers 和 quay.io 镜像的文件，并替换为我们之前已经上传到阿里云的镜像，替换脚本如下：

grc_image_files=(
./kubespray/extra_playbooks/roles/dnsmasq/templates/dnsmasq-autoscaler.yml
./kubespray/extra_playbooks/roles/download/defaults/main.yml
./kubespray/extra_playbooks/roles/kubernetes-apps/ansible/defaults/main.yml
./kubespray/roles/download/defaults/main.yml
./kubespray/roles/dnsmasq/templates/dnsmasq-autoscaler.yml
./kubespray/roles/kubernetes-apps/ansible/defaults/main.yml
)

在执行

for file in ${grc_image_files[@]} ; do
    sed -i 's/gcr.io\/google_containers/registry.cn-hangzhou.aliyuncs.com\/szss_k8s/g' $file
done

同理替换quay_image：

quay_image_files=(
./kubespray/extra_playbooks/roles/download/defaults/main.yml
./kubespray/roles/download/defaults/main.yml
)

再执行：

for file in ${quay_image_files[@]} ; do
    sed -i 's/quay.io\/coreos\//registry.cn-hangzhou.aliyuncs.com\/szss_quay_io\/coreos-/g' $file
    sed -i 's/quay.io\/calico\//registry.cn-hangzhou.aliyuncs.com\/szss_quay_io\/calico-/g' $file
    sed -i 's/quay.io\/l23network\//registry.cn-hangzhou.aliyuncs.com\/szss_quay_io\/l23network-/g' $file
done

使用ansible playbook部署Kubernetes集群以上全部完成，执行安装操作：

cd kubespray

ansible-playbook -i inventory/mycluster/inventory.ini cluster.yml -b -v --private-key=~/.ssh/id_rsa

7. 快捷命令 ¶

alias kk='kubectl get pod --all-namespaces -o wide --show-labels'
alias ks='kubectl get svc --all-namespaces -o wide'
alias kss='kubectl get svc --all-namespaces -o wide --show-labels'
alias kd='kubectl get deploy --all-namespaces -o wide'
alias wk='watch kubectl get pod --all-namespaces -o wide --show-labels'
alias kv='kubectl get pv -o wide'
alias kvc='kubectl get pvc -o wide --all-namespaces --show-labels'
alias kbb='kubectl run -it --rm --restart=Never busybox --image=busybox sh'
alias kbbc='kubectl run -it --rm --restart=Never curl --image=appropriate/curl sh'
alias kd='kubectl get deployment --all-namespaces --show-labels'
alias kcm='kubectl get cm --all-namespaces -o wide'
alias kin='kubectl get ingress --all-namespaces -o wide'