本次通过Gitlab+Jenkins+K8s集群+Kuboard+Harbor实现自动化CICD,当容器镜像推送到 Harbor镜像库中之后,自动触发docker pull 更新 Kubernetes 环境中的容器镜像。
实验拓扑图:
由于在之前的文章中已经部署过k8s集群、harbor、kuboard,所以就不在这里介绍了,大家可以结合以下文章进行部署;本次重点介绍CICD自动化的实现与落地。
Breeze1.19.0部署Kubernetes1.19.0高可用集群:
https://blog.51cto.com/890909/2538107
Kuboard图形化管理k8s集群及harbor镜像库集成:
https://blog.51cto.com/890909/2538334
解决Jenkins初始化插件失败:
https://blog.51cto.com/890909/2488882
一. Gitlab配置:
去gitlab官网下载最新版rpm包,拷贝到gitlab服务器,我这里下载了gitlab-ce-13.4.1-ce.0.el7.x86_64.rpm
# yum install -y gitlab-ce-13.4.1-ce.0.el7.x86_64.rpm
修改Hosts:
# vim /etc/hosts
修改gitlab.rb文件,添加gitlab登录地址:
# vim /etc/gitlab/gitlab.rb
external_url 'http://192.168.50.17'
重新配置gitlab:
# gitlab-ctl reconfigure
等待完成后,用浏览器登录“http://192.168.50.17”,修改登录密码,用root用户登录。
创建项目:
⑴ 通过http克隆项目:
登录到client客户端:
# yum install -y git
# vim /etc/hosts
# git clone http://192.168.50.17/root/cicd.git
# ls
# cd cicd/
# ls -a //查看隐藏文件
测试提交文件:
# vim index.html
666666 v6
# git add .
# git config --global user.name "Aizenwong"
# git config --global user.email "test@123.com"
# git commit -m "v1"
# git push -u origin master //输入root和密码
登录gitlab进行查看:
# cd
# rm -rf cicd/
⑵ 通过SSH克隆:
登录到client客户端:
# ssh-keygen -N ""
# ls .ssh/
# cat .ssh/id_rsa.pub //复制红色框内密钥
登录到gitlab网站:
点击右上角,"settings"
添加客户端SSH KEY:
以后可以通过SSH免密提交代码了
测试:
# git clone git@192.168.50.17:root/cicd.git
# cd cicd/
# vim index.html
88888888 v8
# git add .
# git commit -m "v8"
# git config --global user.name "Aizenwong"
# git config --global user.email "test@123.com"
# git push
二. 配置Jenkins:
到jenkins和java官网下载rpm包,不在赘述,我这里下载的是jenkins-2.259-1.1.noarch.rpm、jdk-8u261-linux-x64.rpm
# yum install -y jenkins-2.259-1.1.noarch.rpm
# yum install -y git docker
# rpm -ivh jdk-8u261-linux-x64.rpm
# service jenkins start
# chkconfig jenkins on
⑴ 登录jenkins:
在浏览器输入“http://192.168.50.18:8080”,复制地址/var/lib/jenkins/secrets/initialAdminPassword
登录jenkins服务器解锁:
# cat /var/lib/jenkins/secrets/initialAdminPassword //复制密钥
⑵ 安装推荐的插件:
此处附上解决初始化插件问题解决地址:https://blog.51cto.com/890909/2488882
⑶ 用admin登录,自行更改密码:
① 插件管理:
插件管理----可选插件----“查找docker”,选择"docker-build-step和Docker",进行安装。
安装完成后的效果:
⑶ 系统配置:
① 进入系统配置----Docker Builder
Docker URL tcp://192.168.50.15:2375
点击,Test Connection进行测试。
由于harbor默认没有启用docker远程连接,还需要先设置docker远程连接,才能访问2375端口,方法如下:
登录到harbor主机:
# systemctl edit docker.service
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375
# systemctl daemon-reload ; systemctl restart docker
# netstat -tnlp | grep 2375
② 新增一个docker云:
新版jenkins更改了docker云配置位置:
输入一样的URL:
⑷ 权限设置:
# vim /etc/sudoers.d/jenkins
jenkins jenkins=(root)NOPASSWD:/bin/rm
# gpasswd -a jenkins root
# service jenkins restart
⑸ jenkins主机登录harbor镜像库报错解决方法:
在docker login harbor主机时,发现无法访问,从网上找来解决方案:
# find / -name docker.service -type f
# vim /usr/lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd-current \
--insecure-registry=192.168.50.15
# systemctl daemon-reload ; systemctl restart docker
# docker login 192.168.50.15 -u admin -p Harbor12345
⑹ 创建任务:
新建任务,CICD,选择自由风格
配置:
构建触发器:
勾选"触发远程构建",身份验证令牌输入相关令牌码
记住jenkins URL,gitlab webhook会用到,http://192.168.50.18:8080/job/CICD/build?token=password
构建:
选择执行shell:
在执行shell里输入相关命令:
cd /zz
sudo rm -rf *
git clone http://192.168.50.17/root/cicd.git
version=$(date +"%Y.%m.%d.%H.%M.%S")
name=192.168.50.15/cicd/cicd-nginx:$version
docker build -t $name cicd
docker login 192.168.50.15 -u admin -p password
docker push $name
bash /root/rm_all_images.sh
curl -X PATCH \
-H "content-type: application/strategic-merge-patch+json" \
-H "Authorization:Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6InlUT1lkZjgwNHhBdjlQNnRtSHVoYjg4MGlFUzI4S21sOE1RVkJSZzlNY3MifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJvYXJkLXVzZXItdG9rZW4tdDk0N2oiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoia3Vib2FyZC11c2VyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYjJhOTBiMzQtOGQxMS00NDI5LTlkMjItYWIwYzIzMjU4YzZmIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmt1Ym9hcmQtdXNlciJ9.7TkJZf21AdNnnYRpLi--ft3sC_iBy6qQk_BgKNREP1hrfOsSpqe_rI4YFg2MKlm6nMadhEL6WC3zSFdgwcI9em9iAeZ5-DxIliDXhm620dClN5ILzO4n0NC9pARB7uRo_KAFIdwJEV4Bjw5hTlPiFNFX6bqY0P9Lx7iM7uluDP2cEvcIKeHSQ_WLfbgzTAs8wvudQbSFVgbdSnJ8npG1lgDT2lXn3Dpqq4FSLipA5GE-J-BWnsJTLuZObTlHyMDs7EKwnWFq7vImVS9HgntUmh48h0qFJmh1LQSbwmfdiVMMsrZuUaFPYhQ-oNqqFrNYZTo6o5gb7pnilvpiIj4kYw" \
-d '{"spec":{"template":{"spec":{"containers":[{"name":"nginx","image":"192.168.50.15/cicd/cicd-nginx:'$version'"}]}}}}' \
"http://192.168.50.16:32567/k8s-api/apis/apps/v1/namespaces/cicd/deployments/nginx"
保存,后面我再讲每条命令的作用。
⑺ kuboard创建Deployment:
在cicd命名空间创建nginx的Deployment,NodePort为32568
点击CI/CD集成,会发现有相关更新镜像教程:
我们找到自动触发教程,查看如何集成:
教程的大致意思是,镜像版本改成$变量,这样在每次触发时,会更新为最新镜像,在我的脚本中为"192.168.50.15/cicd/cicd-nginx:'$version'"
⑻ 这里我来讲解一下我所写脚本的作用:
cd /zz ##jenkins服务新建zz目录
sudo rm -rf * ##每次进入zz都先清空目录,不然会导致下次构建时无法克隆目录
git clone ##http模式克隆gitlab项目目录
version=$(date +"%Y.%m.%d.%H.%M.%S") ##镜像版本,我利用系统时间来进行镜像版本$version
name=192.168.50.15/cicd/cicd-nginx:$version ##上传镜像到指定harbor项目下
docker build -t $name cicd ##制作dockerfile文件,
docker login 192.168.50.15 -u admin -p password ##登录harbor,否则无法docker push
docker push $name ##docker push 镜像到harbor,镜像为变量$name
bash /root/rm_all_images.sh ##这里我写了一个jenkins服务器,清空docker镜像脚本,节约空间
curl -X PATCH \ ##这里就是kuboard自动生成的登录远程master vip ip的Authorization:Bearer,注意修改镜像信息
-H "content-type: application/strategic-merge-patch+json" \
-H "Authorization:Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6InlUT1lkZjgwNHhBdjlQNnRtSHVoYjg4MGlFUzI4S21sOE1RVkJSZzlNY3MifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJvYXJkLXVzZXItdG9rZW4tdDk0N2oiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoia3Vib2FyZC11c2VyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYjJhOTBiMzQtOGQxMS00NDI5LTlkMjItYWIwYzIzMjU4YzZmIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmt1Ym9hcmQtdXNlciJ9.7TkJZf21AdNnnYRpLi--ft3sC_iBy6qQk_BgKNREP1hrfOsSpqe_rI4YFg2MKlm6nMadhEL6WC3zSFdgwcI9em9iAeZ5-DxIliDXhm620dClN5ILzO4n0NC9pARB7uRo_KAFIdwJEV4Bjw5hTlPiFNFX6bqY0P9Lx7iM7uluDP2cEvcIKeHSQ_WLfbgzTAs8wvudQbSFVgbdSnJ8npG1lgDT2lXn3Dpqq4FSLipA5GE-J-BWnsJTLuZObTlHyMDs7EKwnWFq7vImVS9HgntUmh48h0qFJmh1LQSbwmfdiVMMsrZuUaFPYhQ-oNqqFrNYZTo6o5gb7pnilvpiIj4kYw" \
-d '{"spec":{"template":{"spec":{"containers":[{"name":"nginx","image":"192.168.50.15/cicd/cicd-nginx:'$version'"}]}}}}' \
"http://192.168.50.16:32567/k8s-api/apis/apps/v1/namespaces/cicd/deployments/nginx"
⑼ 查看nginx服务:
在浏览器输入master vip ip + 32568端口:
⑽ 配置gitlab与jenkins的联动:
在jenkins主机上:
# mkdir /zz
# chown jenkins.jenkins /zz
登录jenkins网站:
安全----全局安全配置
① 授权策略:
"登录用户可以做任何事",勾选"匿名用户具有可读权限"
② 跨站请求伪造保护:
目前高版本的jenkins无法手动关闭,需要在jenkins主机里修改:
# vim /etc/sysconfig/jenkins
JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Dhudson.security.csrf.GlobalCrumbIssuerConfiguration.DISABLE_CSRF_PROTECTION=true"
添加蓝框部分:
# service jenkins restart
③ 登录到gitlab网页:
点击"扳手"按钮,Settings----Network----Outbound requests,勾选"Allow requests to the local network from web hooks and services"
"Save changes"
进入cicd项目,Settings----Integrations----Go to Webhooks
URL输入:http://192.168.50.18:8080/job/CICD/build?token=password
Add webhook
三. 实战CICD项目:
登录client客户端:
# cd cicd/
# vim index.html
Hello CICD! V10
# vim Dockerfile //编写Dockerfile,利用harbor已有镜像创建
FROM 192.168.50.15/cicd/nginx
MAINTAINER Aizenwong
ADD index.html /usr/share/nginx/html
EXPOSE 80
CMD ["nginx", "-g","daemon off;"]
# git add .
# git commit -m "v10"
# git push
四. 镜像回滚:
先登录harbor,查看镜像版本:
找一个时间最早的镜像,进行回滚
登录到kuboard: