来源: DevOpSec公众号 作者: DevOpSec
背景
国内安装k8s集群，下载镜像往往失败或者半天下载不下来，导致安装不成功，影响摸鱼心情。
是否可以通过kubespray离线部署k8s集群。
完全可以的，准备就绪后，只需10多分钟即可搭建出一套高可用的k8s集群，心情畅悦。
准备工作
- 首先需要一台
科学上网机器，用于下载k8s相关镜像和安装包 - 搭建nginx下载站，用于存放
k8s集群安装所需的安装包。 - 搭建harbor镜像服务，存储安装
k8s集群所需的镜像。 - 需要一台独立
管理机器，安装docker环境用于部署k8s集群。
开始实施
资源
| 主机名 | 角色 | ip | 操作系统 |
|---|---|---|---|
| master1 | master | 192.168.1.11 | Centos 7.9.x 内核版本4.19.x |
| master2 | master | 192.168.1.12 | Centos 7.9.x 内核版本4.19.x |
| master3 | master | 192.168.1.13 | Centos 7.9.x 内核版本4.19.x |
| node1 | node | 192.168.1.14 | Centos 7.9.x 内核版本4.19.x |
| manager1 | 管理机器 | 192.168.1.253 | Centos 7.9.x 内核版本4.19.x |
| nogfw001 | 科学上网机器 | 192.168.2.111 | Centos 7.9.x 内核版本4.19.x |
| yourdownload.domian.com | 下载站 | Centos 7.9.x 内核版本4.19.x | |
| yourharbor.domian.com | 镜像源 | Centos 7.9.x 内核版本4.19.x |
在科学上网机器上做如下操作
- 下载
kubespray包，选最新最稳定版
a. 检出kubespray代码
git clone git@github.com:kubernetes-sigs/kubespray.git
b. 此时对外release版本v.2.21.0
cd kubespray
git clone v.2.21.0- 离线下载需要的安装包
a. 生成下载包和镜像列表
yum install ansible -y #安装ansible
cd contrib/offline
sh generate_list.sh
b. 查看目录结构
tree temp/
temp/
├── files.list #依赖包的列表
├── files.list.template
├── images.list #依赖镜像的列表
└── images.list.template
c. 下载软件包到目录 temp/files
wget -x -P temp/files -i temp/files.list
- 配置
nginx代理，代理可以不在科学上网机器上
a. nginx配置块
server {
listen 80 yourdownload.domian.com;
location /k8s/ {
alias /path/kubespray/contrib/offline/temp/files/;
}
}
reload nginx 进程
b. 测试，注意版本，如果能正常下载则代表代理搭建正常
wget http://yourdomain.domian.com/k8s/etcd-io/etcd/releases/download/v3.5.6/etcd-v3.5.6-linux-amd64.tar.gz
- 下载依赖镜像
a. 安装并启动docker-ce
yum install docker-ce
service docker start
b. 安装 skopeo 将依赖镜像同步到自建私有镜像仓库
yum install skopeo ## 注意centos操作系统版本要在7.x
c. 创建docker公共的项目k8s，harbor域名：youharbor.domain.com
如果需要登录则执行
docker login youharbor.domain.com
d. 同步镜像
for image in $(cat temp/images.list); do skopeo copy docker://${image} docker://youharbor.domain.com/k8s/${image#*/}; done
- 下载
kubespray镜像，省去安装kubespray各种依赖下载镜像 docker pull quay.io/kubespray/kubespray:v2.21.0
镜像打tag docker tag quay.io/kubespray/kubespray:v2.21.0 youharbor.domain.com/k8s/quay.io/kubespray/kubespray:v2.21.0
推送到自己的harbor仓库 docker push youharbor.domain.com/k8s/quay.io/kubespray/kubespray:v2.21.0
### 在<font color="red">`管理机器`</font>上操作
1. 安装docker，下载镜像a. 安装docker-ce并启动 b. 下载kubespray image docker pull youharbor.domain.com/k8s/quay.io/kubespray/kubespray:v2.21.0
2. 免密登录k8s集群机器- 生产公私钥对 ssh-keygen -b 4096 遇到输密码直接回车 d. 公钥copy到master和node机器 ssh-copy-id master1 ssh-copy-id master2 ssh-copy-id master3 ssh-copy-id node1 ```
- 下载
kubespray包选最新最稳定版，或者从科学上网的机器上copy
a. 检出kubespray代码
git clone git@github.com:kubernetes-sigs/kubespray.git
b. 此时对外release版本v.2.21.0
cd kubespray
git clone v.2.21.0
或者从`科学上网的机器`上copy
cp -rp 192.168.2.111:/path/kubespray .- 配置离线安装
cp -rp /path/kubespray/inventory/sample/ /path/kubespray/inventory/mycluster/ vim /path/kubespray/inventory/mycluster/group_vars/all/offline.yml
替换镜像地址
registry_host: "yourharbor.domian.com/k8s" kube_image_repo: "{{ registry_host }}" gcr_image_repo: "{{ registry_host }}" github_image_repo: "{{ registry_host }}" docker_image_repo: "{{ registry_host }}" quay_image_repo: "{{ registry_host }}"
替换软件包地址
files_repo: "http://yourdownload.domian.com/k8s" kubeadm_download_url: "{{ files_repo }}/storage.googleapis.com/kubernetes-release/release/{{ kube_version }}/bin/linux/{{ image_arch }}/kubeadm" kubectl_download_url: "{{ files_repo }}/storage.googleapis.com/kubernetes-release/release/{{ kube_version }}/bin/linux/{{ image_arch }}/kubectl" kubelet_download_url: "{{ files_repo }}/storage.googleapis.com/kubernetes-release/release/{{ kube_version }}/bin/linux/{{ image_arch }}/kubelet" cni_download_url: "{{ files_repo }}/github.com/containernetworking/plugins/releases/download/{{ cni_version }}/cni-plugins-linux-{{ image_arch }}-{{ cni_version }}.tgz" crictl_download_url: "{{ files_repo }}/github.com/kubernetes-sigs/cri-tools/releases/download/{{ crictl_version }}/crictl-{{ crictl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz" etcd_download_url: "{{ files_repo }}/github.com/etcd-io/etcd/releases/download/{{ etcd_version }}/etcd-{{ etcd_version }}-linux-{{ image_arch }}.tar.gz" calicoctl_download_url: "{{ files_repo }}/github.com/projectcalico/calico/releases/download/{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}" calico_crds_download_url: "{{ files_repo }}/github.com/projectcalico/calico/archive/{{ calico_version }}.tar.gz" flannel_cni_download_url: "{{ files_repo }}/github.com/flannel-io/cni-plugin/releases/download/{{ flannel_cni_version }}/flannel-{{ image_arch }}" helm_download_url: "{{ files_repo }}/get.helm.sh/helm-{{ helm_version }}-linux-{{ image_arch }}.tar.gz" crun_download_url: "{{ files_repo }}/github.com/containers/crun/releases/download/{{ crun_version }}/crun-{{ crun_version }}-linux-{{ image_arch }}" kata_containers_download_url: "{{ files_repo }}/github.com/kata-containers/kata-containers/releases/download/{{ kata_containers_version }}/kata-static-{{ kata_containers_version }}-{{ ansible_architecture }}.tar.xz" runc_download_url: "{{ files_repo }}/github.com/opencontainers/runc/releases/download/{{ runc_version }}/runc.{{ image_arch }}" containerd_download_url: "{{ files_repo }}/github.com/containerd/containerd/releases/download/v{{ containerd_version }}/containerd-{{ containerd_version }}-linux-{{ image_arch }}.tar.gz" nerdctl_download_url: "{{ files_repo }}/github.com/containerd/nerdctl/releases/download/v{{ nerdctl_version }}/nerdctl-{{ nerdctl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz" krew_download_url: "{{ files_repo }}/github.com/kubernetes-sigs/krew/releases/download/{{ krew_version }}/krew-{{ host_os }}{{ image_arch }}.tar.gz" cri_dockerd_download_url: "{{ files_repo }}/github.com/Mirantis/cri-dockerd/releases/download/{{ cri_dockerd_version }}/cri-dockerd-{{ cri_dockerd_version }}-linux-{{ image_arch }}.tar.gz" gvisor_runsc_download_url: "{{ files_repo }}/storage.googleapis.com/gvisor/releases/release/{{ gvisor_version }}/{{ ansible_architecture }}/runsc" gvisor_containerd_shim_runsc_download_url: "{{ files_repo }}/storage.googleapis.com/gvisor/releases/release/{{ gvisor_version }}/{{ ansible_architecture }}/containerd-shim-runsc-v1" youki_download_url: "{{ files_repo }}/github.com/containers/youki/releases/download/v{{ youki_version }}/youki_v{{ youki_version | regex_replace('\.', '') }}_linux.tar.gz"
5. 配置机器资源vim /path/kubespray/inventory/mycluster/inventory.ini
[all] master1 ansible_host=192.168.1.11 master2 ansible_host=192.168.1.12 master3 ansible_host=192.168.1.13 node1 ansible_host=192.168.1.14
[kube_control_plane] master1 master2 master3
[etcd] master1 master2 master3
[kube_node] node1
[calico_rr]
[k8s_cluster:children] kube_control_plane kube_node calico_rr
6. 配置安装集群信息vim /path/kubespray/inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
a. 配置pod网段 kube_service_addresses: 10.233.0.0/18 ## 有1.6万多个ip，更具自己环境配置
b. 配置pod网段 kube_service_addresses: 10.233.0.0/18 ## 有1.6万多个ip，更具自己环境配置
c. 选择网络插件，默认calico kube_network_plugin: calico
d. 容器运行时 container_manager: containerd #推荐containerd
e. 是否开启kata容器，建议开启 kata_containers_enabled: true
f. 证书是否自动续期，建议开启 auto_renew_certificates: true
也可以通过手动续期，在所有的master节点执行 sh /usr/local/bin/k8s-certs-renew.sh
以上工作准备完之后，就开始我们的集群安装之旅，内网环境大概10多分钟就能搞定集群安装。
## `kubespray`运维k8s集群，在<font color="red">`管理机器`</font>上操作
0. 启动`kubespray`容器
`kubespray`容器启动脚本如下cat runc.sh
把我们刚刚准备的配置挂载到容器里
docker run --rm -it --mount type=bind,source=/path/kubespray/inventory/mycluster,dst=/inventory
--mount type=bind,source=/path/kubespray,dst=/apps/kubespray
--mount type=bind,source="${HOME}"/.ssh/id_rsa,dst=/root/.ssh/id_rsa
yourharbor.domian.com/k8s/quay.io/kubespray/kubespray:v2.21.0 bash
1. `k8s`集群安装a. 启动kubespray容器 sh runc.sh
b. 在容器里执行命令安装集群
ANSIBLE_HOST_KEY_CHECKING=False 作用是首次访问集群机器不用输入yes验证
注意集群节点都能访问yourharbor.domian.com和yourdownload.domian.com域名
ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i /inventory/inventory.ini --private-key /root/.ssh/id_rsa cluster.yml
如果安装过程中有报jmespath依赖包的错误，见下文问题部分
2. `k8s`集群新增节点a. 修改/path/kubespray/inventory/mycluster/inventory.ini [kube_node] node1 node2 #新增节点
b. 启动容器 sh runc.sh
b. 执行扩容 ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i /inventory/inventory.ini --private-key /root/.ssh/id_rsa scale.yml -b -v
3. `k8s`集群删除节点a. 修改/path/kubespray/inventory/mycluster/inventory.ini [kube_node] node1 node2 #删除节点 node3 #删除节点
b. 启动容器 sh runc.sh
b. 执行删除 ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i /inventory/inventory.ini --private-key /root/.ssh/id_rsa remove-node.yml -b -v --extra-vars "node=node2,node3"
4. `k8s`集群升级a. 下载文件和对应的镜像，具体操作见上文
b. 启动容器 sh runc.sh
b. 执行升级 ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -b -i /inventory/inventory.ini --private-key /root/.ssh/id_rsa upgrade-cluster.yml -e kube_version=v1.26.3
5. 卸载`k8s`集群a. 启动容器 sh runc.sh
b. ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i /inventory/inventory.ini --private-key /root/.ssh/id_rsa reset.yml
## 问题
1. jmespath 依赖包没有如果安装过程中有报jmespath依赖包的错误，重新打镜像 cat Dockerfile FROM youharbor.domain.com/k8s/quay.io/kubespray/kubespray:v2.21.0 RUN pip3 install jmespath -i http://pypi.douban.com/simple/ --trusted-host=pypi.douban.com/simple
docker build youharbor.domain.com/k8s/quay.io/kubespray/kubespray:v2.21.0 . docker push youharbor.domain.com/k8s/quay.io/kubespray/kubespray:v2.21.0
2. node节点上`kubelet`报错error: Apr 27 13:06:37 plat-data-mysql-sh-001 kubelet[26809]: E0427 13:06:37.782074 26809 summary_sys_containers.go:48] "Failed to get system container stats" err="failed to get cgroup stats for "/kube.slice/containerd.service": failed to get container info for "/kube.slice/containerd.service": unknown container "/kube.slice/containerd.service"" containerName="/kube.slice/containerd.service"
解决： sed -i 's#/kube.slice/containerd.service#/system.slice/kubelet.service#g' /etc/kubernetes/kubelet.env ; service kubelet restart
