jenkins部署k8s项目-CICD
一 环境准备
1 docker仓库，harbor我这里用的是线上的hub https://hub.docker.com/ 自己注册即可
2 准备git仓库，我这里用的是码云 https://gitee.com/huningfei/java.git
3 准备好k8s环境，一主两从，可以二进制安装，也可以kubeadmin安装
master:172.16.78.103 node1:172.16.78.102 node2:172.16.78.101
4 准备好jenkins环境，（我是利用jenkins.war包直接部署的，也可以用k8s部署jenkins）
我们提供的服务有：成都网站设计、成都网站制作、微信公众号开发、网站优化、网站认证、平顺ssl等。为千余家企事业单位解决了网站和推广的问题。提供周到的售前咨询和贴心的售后服务，是有科学管理、有技术的平顺网站制作公司
二 k8s部署
我是用kubeadmin快速部署的 参考 https://cloud.tencent.com/developer/article/1509412
安装之前，把iptables和selinux关掉。
1 master节点安装
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
269 yum -y install docker-ce-18.06.1.ce-3.el7
271 curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://f1361db2.m.daocloud.io
272 systemctl enable docker && systemctl start docker
# 添加kubernetes YUM软件源
275 cat > /etc/yum.repos.d/kubernetes.repo << EOF
276 [kubernetes]
277 name=Kubernetes
278 baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
279 enabled=1
280 gpgcheck=0
281 repo_gpgcheck=0
282 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
283 EOF
284 ll /etc/yum.repos.d/kubernetes.repo
#安装kubeadm,kubelet和kubectl
285 yum install -y kubelet-1.15.0 kubeadm-1.15.0 kubectl-1.15.0
286 kubectl systemctl enable kubelet
# 部署Kubernetes Master 只需要在Master 节点执行，这里的apiserve需要修改成自己的master地址
kubeadm init --apiserver-advertise-address=172.16.78.103 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.15.0 --service-cidr=10.1.0.0/16 --pod-network-cidr=10.244.0.0/16
最后出现 Your Kubernetes control-plane has initialized successfully! 即为安装成功
# 最后根据提示操作
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
# 安装网络插件
wget https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml
修改镜像地址：(有可能默认不能拉取，确保能够访问到quay.io这个registery,否则修改如下内容)
cat -n kube-flannel.yml|grep lizhenliang/flannel:v0.11.0-amd64
106 image: lizhenliang/flannel:v0.11.0-amd64
120 image: lizhenliang/flannel:v0.11.0-amd64
kubectl apply -f kube-flannel.yml
2 node节点安装
#!/bin/bash
#install docker
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
yum -y install docker-ce-18.06.1.ce-3.el7
curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://f1361db2.m.daocloud.io
systemctl enable docker && systemctl start docker
# repo
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
#install k8s
yum install -y kubelet-1.15.0 kubeadm-1.15.0 kubectl-1.15.0
systemctl enable kubelet
echo "1" >/proc/sys/net/bridge/bridge-nf-call-iptables
#join master
kubeadm join 172.16.78.103:6443 --token culxlo.ovuhhraplwritzgh --discovery-token-ca-cert-hash sha256:3dfb2e64dfc2c603c1cfccceff82dbc1b680b117fdbbe0512889d3a74240a3b8
# 在加入的时候提示报错:
[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1
[ERROR Swap]: running with swap on is not supported. Please disable swap
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
# 解决办法
echo "1" >/proc/sys/net/bridge/bridge-nf-call-iptables
3 最后查看查看集群node状态
三设置jenkins
1 安装Kubernetes 和git，pipeline 安装这三个插件，安装maven和docker环境。
yum install maven
# install docker
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
yum -y install docker-ce-18.06.1.ce-3.el7
curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://f1361db2.m.daocloud.io
systemctl enable docker && systemctl start docker
# 还需要注意一点，如果你的jenkins是用普通用户启动的，当构建的时候可能会提示权限问题， 要改一下/var/run/docker.sock 这个文件的权限2 设置jenkins去连接k8s api
系统管理---系统设置-cloud
kubernetes 服务证书从哪里来？
2-1 如果是kubeadmin部署的k8s,就比较麻烦，进入到 /etc/kubernetes目录下面打开admin.conf配置文件，文件中有三个值 certificate-authority-data 、client-certificate-data 、 client-key-data 分别用这三部分生成证书，这里我把生成的证书都放到了tmp目录下面
echo LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRFNU1USXlOVEE0TVRFMU4xb1hEVEk1TVRJeU1qQTRNVEUxTjFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh3Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBT0d6Ckl0RzFDMTMxL09ObXNJeERraTFrWDFyZ25kZ0hLRnBLTmJxNVl2R21jamh4a21SVjdnc3g2VGVIakpCY01welUKUFFGU3FkcjRldUJQTTM1MGhTSnkrODZpSWl5NWRzZTNONGw2bng1cjVyQzB1TEk1c1BLTXdLb3pGaHc4a0pFUwpLQnBpTFd2Q0VsTTljVlJKWnpiRENISWNHc09ZUklaWDBPcDJWNEh0U21FdlRuU0wrL1hEZmEzcUR1Y0NLVGNWCnNVWjNwdmRvMERCbEZzUUZPZDVQVDF4Rm13WmlSeDBneVJDS1VITDc0V1A3akk1eFZsYWMwczJ2QlRzaEp5L2IKTjBIcDM2RXQ2NUlLRHFrVjM4RS9ONEZEN3B2eWkxTGtxcWI3QXF5VG1VQlpsd2taclpPTnhuZFhUeGdWeXZVVApSRTBtZW1KNnAxaWp1MVFGKzZjQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFDUERiRXBzdlhVWVc2bW1LZjB2TjNkS0ZGN0kKbldUcUJBUXdTcnBFK0oxL0FueWRHOTFVT2NtQnZMdTBkTzhJbk56VXJvWnNWbEVqNjVpUDNMU2hEbzdqQzEzbQp5MFFrM2Zzc1ltd1JpY3NwcE9RMmpLZ1dETUF5dGVoRm1zTnBybmxQTWJueXVLQktHOTdDeWpYdExvam5jbGtQCnBqb1BYdFRNTmZSemJOdEVEeVNzcHhTaFZEaDQ1eWdBMk9yK3BTbGlaRXY4YUFJcWFTbFU2MU5rV3FuVGtlS0oKNzJ4SW9JRmpyV3ZrQzZsTDdMUVQrcjJxQlFjTDBGbDNiQS9zbnFoM2Irck1rTDZYQkZ6UmFnK1pwdEV3bktIdgordG85bVBaR0FVUldLakNvRHdXN05Da3NjdEZRTnBUbGZUMWppaUJXRHFNd1pqclkxdWlhcW5Lb3BuTT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= | base64 -d > /tmp/ca.crt
echo LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcFFJQkFBS0NBUUVBelBKRllPQVdFZm94WDh7dVh3NWhjUE5DS1oxVUZ4RGF6T3lSYjNaZnRXTVNmUDk0Ck92dW5Ocmx5MGh2Yk9MMmxrWllXM3ZUb2dsbXhiaVNKdGw4SDM3cHlzbXJ6bjJSMk5LMzZkRmQ4YVgxeDkzYXoKS2xWK1YwMnhuUUVreFZlL3lRUmE0L0RVUzdVUHdOdU85Q2FDVWoxV2dzVStwcjA0UlBEYVhpaE9VTGR0cVEyagpKeVJkeGhTQVBBMDdSWGM5cmFKODMzMXFBcUVJQkl2bG56Vm9zM1d6WUlhUzI2d29OOHhIaStpTXJGT2pkVnV3CjM0OGlvTlhIUUZuaVk4c0ZJTXpua0dEQmljZ2N3R1IzeGRGM1NkdldudFZlQ0xlWFZBaS9iOStOVTBBZmlEdjkKNmsrRVJQUEJ5UFIrWXIxR1ZtTWNncG40Q0lwaUlRZG11SmhxQlFJREFRQUJBb0lCQVFERkhONk5aZmpVSVlZTApWZGFObHF1eExzV2JzSzJINGJZUEVoaHUwN01KaWRmbVVPOGFDNSt2aW4xY2h6Q3JXZnM4SXRhc2FYQ1dyaDF3CmljUEptMnkvU1Z1M0xoWlNYeFV4WllOVkVpK2VaZ0NPdWFOb1BBSGZoZUlsMTB1K1liOGJYd3pyM2x6UTA3dnIKQ1VCY29nMEZlK2tRclJTczJhQ2M3ZW1ZVmJ1ODlIUXlGV2dsK21VQ1FYVzFXWWo1OWlmT3hkRlNEdVlaUUo5bwo1OHhPOWRHQXJQRll6WmdmZ0g4UkU2dXhiRlJUNUlXQlg1aEpLL2hQMDE1WW04M0tPWk9pNGZLOEg5cVhyK0VjCkZob3VIU0ZoRng1TTVURjgyVHpXWjNLS2FFYmhOMWxoUXdnNWIyVHQ3QzNDSlNoL01VOGx4UUJmbUVxR24yZW0KZGJveVNtOEJBb0dCQVBwZEtacG9hNngzNTFKSytNeW5lck05MVBMQXRXMXZMc3V6Z1Z1dGE2MGZIZlhSYWcwLwpTTFhpem5nb3FIMzducE55T28yTGl3aEtYb0NQNXpXMVVwQkt4UlJmeS8yYk9rWFBHMk9Qb2U5anNpeHFUV3NrCnBFakpEaEp2WnZHbVJVaEIvdVhRaGxuSmEzNWpWMzdzMEVnU1lVd0V5RmlPeXZQSXRIWmhhRDdOQW9HQkFOR1AKWG5QcnBaTlVVdFpSTVVxUjV6Z2RySmZFWTU2MUpTRjQ4YUVWVzRVWThsT1V4T3ZYT2xuTGFWNnZ6TWx6T3VPYQo0RStVRW1TaFYvNkt5Qm5jV2R1dmtnMmVBSEo2U2lnZlhJeGhYcHp0b2o0NFNTNlJ3M2Z3UE5NVEF5VjNQd2VtCjBWaTFRa1NtS0JMY2pYVE5VZ1FOaU5zNk8rVFJFN0EydlEyZGcyZ1pBb0dCQU5xS3JnRXB1eXVPY3E3RDROQU0KTmEvMmZrYjBicW83RGpjajF2d21za0lwVW1hOFNQMk5TelB4NmxhNjd2RWh7ZmRaRThGd21JbEJHYUxSZW5ySQpiMnlpeDh2VUg2V2RkVTF3anJEbXRPMFpNbk5ReTRtR0w1MlZXeUkrZWpiZjg4UXlUNFZkODVpMm1JMTN1KzJBClhBTTlnQTd5Y2N2VHdWYngwSTB4VStUVkFvR0FmcXAzQUFCV2s0ZnJqaW1EYnJ4a0V4STBxWU9HWjM2OGs4L2oKMVdid2E3SFpmMVd6OVkyaGNuYzJHSXFRY00rYXI4cUVmUFZXYk1idFJpa2lyQ2I1bFlVNDljd2tIdzMxMXV5bAo1eUJQclBFaUdST3lnRmlRMnVVMkRxczJRcVlpVGdDeUZ6bHdkY2dzL3NHYkt4ZVQxR2xONkp6NWFPUkxUejYyCjVRenBTYmtDZ1lFQWllUlY4b0FUNUdKQlRGK3FnMFhPZFA3ZFJUZFZHek1wb05rWVpzOEZ3bzBrT2ZZRHE2eVoKOTdQV2VHeEkxNlJWS2V0N1U0NlpTUGNOUWdkMWdmcklmenZmQndaeVA3NkVtTkRGTDJCZWJNaVBkTGtXRUp3RgozbmliUUZxd1pkdkw1L2tuSklhdkpKY3RGZ2NxOGFvREtWSVNYU255YzRrUVBJZUlKbEZWWk5nPQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo= | base64 -d > /tmp/client.key
echo LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM4ekNDQWR1Z0F3SUJBZ0lJZVVLY0NoaXNBRTR3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB4T1RFeU1qVXdPREV4TlRkYUZ3MHlNREV5TWpRd09ERXlNREZhTURVeApGVEFUQmdOVkJBb1RESE41YzNSbGJUcHViMlJsY3pFY01Cb0dBMVVFQXhNVGMzbHpkR1Z0T201dlpHVTZjMkZ6CmN6QXdNVENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFNenlSV0RnRmhINk1WL00KN2w3K1lYRHpRaW1kVkJjUTJzenNrVzkyWDdWakVuei9lRHI3cHphNWN0SWJtemk5cFpHV0Z0NzA2SUpac1c0awppYlpmQjkrNmNySnE4NTlrZGpTdCtuUlhmR2w5Y2ZkMnN5cFZmbGROc1owQkpNVlh3OGtFV3VQdzFFdTFEOERiCmp2UW1nbEk5Vm9MRlBxYTlPRVR3Mmw0b1RsQzNiYWtOb3lja1hjWVVnRHdOTzBWM1BhMmlmTjk5YWdLaENBU0wKNVo4MWFMTjFzMkNHa3R1c0tEZk1SNHZvakt4VG8zVmJzTitQSXFEVngwQlo0bVBMQlNETTU1Qmd3WW5JSE1CawpkOFhSZDBuYjFwN1ZYZ2kzbDFRSXYyL2ZqVk5BSDRnNy9lcFBoRVR6d2NqMGZtSzlSbFpqSElLWitBaUtZaUVIClpyaVlhZ1VDQXdFQUFhTW5NQ1V3RGdZRFZSMFBBUUgvQkFRREFnV2dNQk1HQTFVZEpRUU1NQW9HQ0NzR0FRVUYKQndNQ01BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQkpaWlh6SEdhM1BMb1Rtb0I4RmtMRDN4OHRBTk9oRW8vMQozY0lBZ1NsMFd0SFU3Lzh0Wlo5alBIa3djNWZCeE94Uk5lZk4wWVdoSUR4UFFmSkZwWlEyL0JxTmw1dnpRNGlICko4ZGJZblJxTmcyYWJWUHQvdEtERGtKanBNQ2U4LzlJbFFZY3M5L0gxVUtpaW5WOHFPUmwvKzBvTFFDdkRMRzcKYWdXc1pMb3M1MWVWM3Z3WnBzSTZvSFNSVlFuaTZLVXc3RnJpUWNtYS94WkVweitPWlBlUFBrZVUzQXR2OXQyRQpWcHhDNlZudkMvaWJZU2xEWENmU1lYeEZNZFh3bzQxamgvSk11QU1KRGZqRDJOODM4TFhVMVd3d1hqUHRQUWFICllsbEtENjhoRmoycTJVR0xOMUZqSDZSMEJxeDV2cTkrRUNBQW5GOVgwYU9KYlpxdnVmNDgKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= | base64 -d > /tmp/client.crt
然后到tmp目录，在生成一个ca.crt证书和一个cert.pfx
openssl pkcs12 -export -out cert.pfx -inkey client.key -in client.crt -certfile ca.crt
生成一个pfx的证书，会提示你输入密码 111然后将ca.crt里面的内容放到jenkisn服务证书key的地方即可。
如何添加凭据？
点击add,选择kind类型，然后上传你刚才生成的cert.pfx这个证书，并且输入刚才的密码
最终去测试jenkins能否连接成功，如图
2-2 如果你的jenkins是直接部署在k8s之内的，就比较简单，不需要填写证书，直接这样，如图：
.jpg)
2-3如果你是用的二进制安装的k8s,这个证书在安装过程中，会产生，我测试了用ca.pem和server.pem发现都可以连接，并且不需要添加凭据。
四 用pipeline发布k8s项目
脚本内容如下：
// 公共
def registry = "huningfei"
// 项目
def project = "welcome"
def app_name = "demo"
def image_name = "${registry}/${project}:${BUILD_NUMBER}"
def git_address = "https://gitee.com/huningfei/java.git"
// 认证
def secret_name = "registry-pull-secret"
def docker_registry_auth = "450fb8b4-62c9-43fe-8a69-ec30705e724e"
def git_auth = "b363af0f-a96c-465c-a14e-60f7127727eb"
def k8s_auth = "a60b4b7f-2aef-4622-94a5-6efef67c4ac9"
node(){
// 第一步
stage('拉取代码'){
checkout([$class: 'GitSCM', branches: [[name: '${Branch}']], userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]]])
}
// 第二步
stage('代码编译'){
sh "mvn clean package -Dmaven.test.skip=true"
}
// 第三步
stage('构建镜像'){
withCredentials([usernamePassword(credentialsId: "${docker_registry_auth}", passwordVariable: 'password', usernameVariable: 'username')]) {
sh """
echo '
FROM lizhenliang/tomcat
RUN rm -rf /usr/local/tomcat/webapps/*
ADD target/*.war /usr/local/tomcat/webapps/ROOT.war
' > Dockerfile
docker build -t ${image_name} .
docker login -u ${username} -p '${password}'
docker push ${image_name}
"""
}
}
// 第四步
stage('部署到K8S平台'){
sh """
sed -i 's#\$IMAGE_NAME#${image_name}#' deploy.yml
sed -i 's#\$SECRET_NAME#${secret_name}#' deploy.yml
"""
kubernetesDeploy configs: 'deploy.yml', kubeconfigId: "${k8s_auth}"
}
}
###注意： 脚本里面的deploy.yml是部署k8s的文件，这个文件必须存到gitlab仓库里面就是要跟代码在同一级目录里面
脚本上面三个认证的id从哪里获取？
添加完成之后，去凭据里面查看即可
最后构建项目
出现下面即可成功
然后到k8s-master上面查看是否生成新的pod容器
查看ingress
绑定host即可访问
五 测试修复一个bug,提交代码，重新部署
这个java项目需要连接数据库，在配置文件改一下数据库的地址，然后提交代码，重新构建
1 准备一个数据库，授权一个用户
grant all on *.* to 'demo'@'%'identified by '123456';2 修改代码里面连接数据库的地址
vim tomcat-java-demo-master/src/main/resources/application.yml
url: jdbc:MySQL://db-0.mysql:3306/test?characterEncoding=utf-83 安装mysql，我这里用k8s直接创建
[root@sass001 java-demo]# cat mysql.yaml
apiVersion: v1
kind: Service
metadata:
name: mysql
labels:
project: java-demo
app: mysql
spec:
ports:
- port: 3306
name: mysql
clusterIP: None
selector:
project: java-demo
app: mysql
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: db
spec:
selector:
matchLabels:
project: java-demo
app: mysql
serviceName: "mysql"
template:
metadata:
labels:
project: java-demo
app: mysql
spec:
containers:
- name: mysql
image: mysql:5.7
ports:
- containerPort: 3306
env:
- name: MYSQL_ROOT_PASSWORD
value: "123456"
volumeMounts:
- mountPath: /var/lib/mysql
name: data
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: "managed-nfs-storage"
resources:
requests:
storage: 2Gi kubectl create -f mysql.yaml
修改完代码之后，提交到git仓库，最后在重新发布项目
最后测试功能
分享标题：jenkins部署k8s项目-CICD
标题路径：http://shouzuofang.com/article/ihojpp.html