jenkins发布Kubernetes(K8s)集群(基于containerd)网站首页 学无止境

jenkins发布Kubernetes(K8s)集群(基于containerd)

Wielun 2024-10-17 00:01:02

简介jenkins发布Kubernetes(K8s)集群(基于containerd)

一、实验环境

1、k8s环境

版本v1.26.5，容器为containerd
二进制安装Kubernetes(K8s)集群(基于containerd)—从零安装教程（带证书）

主机名	IP	系统版本	安装服务
master01	10.10.10.21	rhel7.5	nginx、etcd、api-server、scheduler、controller-manager、kubelet、proxy
master02	10.10.10.22	rhel7.5	nginx、etcd、api-server、scheduler、controller-manager、kubelet、proxy
master03	10.10.10.23	rhel7.5	nginx、etcd、api-server、scheduler、controller-manager、kubelet、proxy
node01	10.10.10.24	rhel7.5	nginx、kubelet、proxy
node02	10.10.10.25	rhel7.5	nginx、kubelet、proxy

2、jenkins环境

jenkins入门与安装
容器为docker

主机	IP	系统版本
jenkins	10.10.10.10	rhel7.5

二、docker-compose安装

jenkins服务器上面安装

1、下载

https://github.com/docker/compose/releases/
下载版本：v2.18.0

在这里插入图片描述

2、安装

[root@jenkins ~]# cp docker-compose-linux-x86_64 /usr/local/bin/docker-compose
[root@jenkins ~]# chmod +x /usr/local/bin/docker-compose

3、查看版本

[root@jenkins ~]# docker-compose --version
Docker Compose version v2.18.0

三、cfssl证书生成

此处记录使用cfssl工具生成harbor私有证书，并使用证书搭建Harbor仓库，此证书使用按照kubernetes时使用的ca证书安装

1、安装cfssl

https://imroc.cc/kubernetes/trick/certs/sign-certs-with-cfssl.html
安装包下载地址：https://github.com/cloudflare/cfssl/releases

[root@jenkins ~]# ls cfssl*
cfssl_1.6.2_linux_amd64  cfssl-certinfo_1.6.2_linux_amd64  cfssljson_1.6.2_linux_amd64
[root@jenkins ~]# mv cfssl_1.6.2_linux_amd64 /usr/bin/cfssl
[root@jenkins ~]# mv cfssl-certinfo_1.6.2_linux_amd64  /usr/bin/cfssl-certinfo
[root@jenkins ~]# mv cfssljson_1.6.2_linux_amd64 /usr/bin/cfssljson
[root@jenkins ~]# chmod +x /usr/bin/cfssl*

2、ca生成证书

[root@jenkins ~]# mkdir -p pki &&  cd pki
[root@jenkins pki]# cat > ca-csr.json   << EOF 
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "Kubernetes",
      "OU": "Kubernetes-manual"
    }
  ],
  "ca": {
    "expiry": "876000h"
  }
}
EOF

[root@jenkins pki]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca
[root@jenkins pki]# ls
ca.csr  ca-csr.json  ca-key.pem  ca.pem

3、过期时间查看

[root@jenkins pki]# openssl x509 -noout -text -in ca.pem|grep -A 5 Validity
        Validity
            Not Before: Jun  4 12:32:00 2023 GMT
            Not After : May 11 12:32:00 2123 GMT
        Subject: C=CN, ST=Beijing, L=Beijing, O=Kubernetes, OU=Kubernetes-manual, CN=kubernetes
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption

4、创建Harbor证书

[root@jenkins pki]# cat > ca-config.json << EOF 
{
  "signing": {
    "default": {
      "expiry": "438000h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "438000h"
      }
    }
  }
}
EOF

[root@jenkins pki]# cat > harbor-csr.json  << EOF 
{
  "CN": "harbor",
  "hosts": [
    "127.0.0.1",
    "10.10.10.10",  
    "harbor.wielun.com"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "Kubernetes",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF

[root@jenkins pki]# cfssl gencert 
-ca=ca.pem 
-ca-key=ca-key.pem 
-config=ca-config.json 
-profile=kubernetes harbor-csr.json | cfssljson -bare harbor

[root@jenkins pki]# mkdir -p /etc/harbor/pki
[root@jenkins pki]# cp harbor.pem harbor-key.pem /etc/harbor/pki

四、安装harbor

jenkins服务器上面安装

1、下载

下载地址：https://github.com/goharbor/harbor/releases#install
安装官网：https://goharbor.io/docs/2.8.0/install-config/

在这里插入图片描述

2、安装

（1）解压文件

[root@jenkins ~]#  tar xf harbor-offline-installer-v2.8.1.tgz -C /usr/local

（2）修改harbor.yml

[root@jenkins ~]# cd /usr/local/harbor/
[root@jenkins harbor]# cp harbor.yml.tmpl harbor.yml
[root@jenkins harbor]# vim harbor.yml

在这里插入图片描述
（3）启动

[root@jenkins harbor]# docker load -i harbor.v2.8.1.tar.gz
[root@jenkins harbor]# ./prepare
[root@jenkins harbor]# ./install.sh
[root@jenkins harbor]# docker-compose up -d     #手动启动命令

3、创建登录证书

[root@jenkins ~]# mkdir -p /etc/docker/certs.d/10.10.10.10
[root@jenkins ~]# mkdir -p /etc/docker/certs.d/harbor.wielun.com
[root@jenkins ~]# cp pki/ca.pem /etc/docker/certs.d/10.10.10.10
[root@jenkins ~]# cp pki/ca.pem /etc/docker/certs.d/harbor.wielun.com

4、修改daemon.json

[root@jenkins ~]# cat /etc/docker/daemon.json
{
 "registry-mirrors": ["https://docker.mirrors.ustc.edu.cn"],
  "insecure-registries": [
    "10.10.10.10",
    "harbor.wielun.com"
  ]
}

[root@jenkins ~]# systemctl restart docker

5、添加hosts

[root@jenkins ~]# vim /etc/hosts
10.10.10.10 harbor.wielun.com

6、登录验证

账号密码：admin/Harbor12345

（1）docker login

[root@jenkins ~]# docker login 10.10.10.10
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

[root@jenkins ~]# docker login harbor.wielun.com
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

（2）浏览器登录

https://10.10.10.10/

在这里插入图片描述

7、docker上传镜像测试

（1）拉取镜像

[root@jenkins ~]# docker pull nginx
[root@jenkins ~]# docker images|grep nginx
nginx                           latest              f9c14fe76d50        10 days ago         143MB
goharbor/nginx-photon           v2.8.1              cea1bb2450ee        3 weeks ago         127MB

（2）打包上传

[root@jenkins ~]# docker tag nginx:latest harbor.wielun.com/library/nginx:latest
[root@jenkins ~]# docker push harbor.wielun.com/library/nginx

（3）浏览器中查看
在这里插入图片描述

五、K8s（containerd）拉取镜像（每台机器）

选择一种即可，这边我使用得是跳过证书

1、K8s（containerd）拉取镜像（跳过证书）

（1）删除之前containerd配置

[root@master01 ~]# rm -rf /etc/containerd/config.toml
[root@master01 ~]# containerd config default | tee /etc/containerd/config.toml
[root@master01 ~]# sed -i "s#SystemdCgroup = false#SystemdCgroup = true#g" /etc/containerd/config.toml
[root@master01 ~]# sed -i "s#registry.k8s.io#registry.cn-hangzhou.aliyuncs.com/chenby#g" /etc/containerd/config.toml
[root@master01 ~]# sed -i "s#config_path = ""#config_path = "/etc/containerd/certs.d"#g" /etc/containerd/config.toml

（2）配置hosts.toml

[root@master01 ~]# mkdir -p /etc/containerd/certs.d/harbor.wielun.com
[root@master01 ~]# cat > /etc/containerd/certs.d/harbor.wielun.com/hosts.toml << EOF
server = "https://harbor.wielun.com"
[host."https://harbor.wielun.com"]
  capabilities = ["pull", "resolve", "push"]
  skip_verify = true
EOF

（3）重启containerd

[root@master01 ~]# systemctl restart containerd

（4）添加hosts

[root@master01 ~]# cat /etc/hosts
10.10.10.10 harbor.wielun.com

2、K8s（containerd）拉取镜像（通过证书）

（1）证书配置

[root@master01 ~]# mkdir -p /etc/containerd/certs.d/harbor.wielun.com

[root@jenkins ~]# cd pki/
[root@jenkins pki]# scp ca.pem harbor.pem harbor-key.pem root@10.10.10.21:/etc/containerd/certs.d/harbor.wielun.com

（2）删除之前containerd配置

[root@master01 ~]# rm -rf /etc/containerd/config.toml
[root@master01 ~]# containerd config default | tee /etc/containerd/config.toml
[root@master01 ~]# sed -i "s#SystemdCgroup = false#SystemdCgroup = true#g" /etc/containerd/config.toml
[root@master01 ~]# sed -i "s#registry.k8s.io#registry.cn-hangzhou.aliyuncs.com/chenby#g" /etc/containerd/config.toml

（3）配置config.toml

[root@master01 ~]# vim /etc/containerd/config.toml
    [plugins."io.containerd.grpc.v1.cri".image_decryption]
      key_model = "node"

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = ""

      [plugins."io.containerd.grpc.v1.cri".registry.auths]

      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugin."io.containerd.grpc.v1.cri".registry.configs."harbor.wielun.com".tls]
          ca_file = "/etc/containerd/certs.d/harbor.wielun.com/ca.pem"
          cert_file = "/etc/containerd/certs.d/harbor.wielun.com/harbor.pem"
          key_file = "/etc/containerd/certs.d/harbor.wielun.com/harbor-key.pem"
        [plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.wielun.com".auth]
          username = "admin"
          password = "Harbor12345"

      [plugins."io.containerd.grpc.v1.cri".registry.headers]

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.wielun.com"]
          endpoint = ["https://harbor.wielun.com"]

    [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""

在这里插入图片描述

（4）重启containerd

[root@master01 ~]# systemctl restart containerd

（5）添加hosts

[root@master01 ~]# cat /etc/hosts
10.10.10.10 harbor.wielun.com

3、测试拉取镜像

（1）拉取镜像

# -k：跳过证书认证
[root@master01 ~]# ctr -n harbor.wielun.com  images  pull harbor.wielun.com/library/nginx:latest -k
harbor.wielun.com/library/nginx:latest:                                           resolved       |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:6b06964cdbbc517102ce5e0cef95152f3c6a7ef703e4057cb574539de91f72e6: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:831f51541d386c6d0d86f6799fcfabb48e91e9e5aea63c726240dd699179f495:    done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:f9c14fe76d502861ba0939bc3189e642c02e257f06f4c0214b1f8ca329326cda:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:f03b40093957615593f2ed142961afb6b540507e0b47e3f7626ba5e02efbbbf1:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:eed12bbd64949353649476b59d486ab4c5b84fc5ed2b2dc96384b0b892b6bf7e:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:fa7eb8c8eee8792b8db1c0043092b817376f096e3cc8feeea623c6e00211dad1:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:7ff3b2b12318a41d4b238b643d7fcf1fe6da400ca3e02aa61766348f90455354:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:0f67c7de5f2c7e0dc408ce685285419c1295f24b7a01d554517c7a72374d4aeb:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.1 s                                                                    total:   0.0 B (0.0 B/s)
unpacking linux/amd64 sha256:6b06964cdbbc517102ce5e0cef95152f3c6a7ef703e4057cb574539de91f72e6...

[root@master01 ~]# ctr -n harbor.wielun.com images pull harbor.wielun.com/library/nginx:latest --tlscacert  /etc/containerd/certs.d/harbor.wielun.com/ca.pem --tlscert   /etc/containerd/certs.d/harbor.wielun.com/harbor.pem  --tlskey  /etc/containerd/certs.d/harbor.wielun.com/harbor-key.pem
harbor.wielun.com/library/nginx:latest:                                           resolved       |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:6b06964cdbbc517102ce5e0cef95152f3c6a7ef703e4057cb574539de91f72e6: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:831f51541d386c6d0d86f6799fcfabb48e91e9e5aea63c726240dd699179f495:    done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:f9c14fe76d502861ba0939bc3189e642c02e257f06f4c0214b1f8ca329326cda:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:f03b40093957615593f2ed142961afb6b540507e0b47e3f7626ba5e02efbbbf1:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:eed12bbd64949353649476b59d486ab4c5b84fc5ed2b2dc96384b0b892b6bf7e:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:fa7eb8c8eee8792b8db1c0043092b817376f096e3cc8feeea623c6e00211dad1:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:7ff3b2b12318a41d4b238b643d7fcf1fe6da400ca3e02aa61766348f90455354:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:0f67c7de5f2c7e0dc408ce685285419c1295f24b7a01d554517c7a72374d4aeb:    done           |++++++++++++++++++++++++++++++++++++++|

（2）查看镜像

[root@master01 ~]# ctr -n harbor.wielun.com images ls
REF                                    TYPE                                                 DIGEST                                                                  SIZE     PLATFORMS   LABELS
harbor.wielun.com/library/nginx:latest application/vnd.docker.distribution.manifest.v2+json sha256:6b06964cdbbc517102ce5e0cef95152f3c6a7ef703e4057cb574539de91f72e6 54.5 MiB linux/amd64 -

（3）crictl拉取镜像

[root@master01 ~]# crictl  pull harbor.wielun.com/library/nginx:latest
Image is up to date for sha256:f9c14fe76d502861ba0939bc3189e642c02e257f06f4c0214b1f8ca329326cda

[root@master01 ~]# crictl images ls|grep harbor.wielun.com/library/nginx
harbor.wielun.com/library/nginx                                                latest              f9c14fe76d502       57.2MB

六、jenkins发布到K8S

1、推送tomcat到harbor

这里用Java项目做演示，一般项目打包成jar包，我们这里打包成war包

[root@jenkins ~]# docker pull tomcat:8.5.59
[root@jenkins ~]# docker tag tomcat:8.5.59 harbor.wielun.com/library/tomcat:8.5.59
[root@jenkins ~]# docker push harbor.wielun.com/library/tomcat:8.5.59

2、创建项目

这里我们使用Jenkinsfile

在这里插入图片描述

3、查看项目文件

在这里插入图片描述

（1）配置Dockerfile

FROM harbor.wielun.com/library/tomcat:8.5.59
MAINTAINER Wielun
RUN rm -rf /usr/local/tomcat/webapps/*
ADD target/*.war /usr/local/tomcat/webapps/ROOT.war

（2）配置Jenkinsfile

pipeline {
    agent any
	environment {
		harborUser = 'admin'
		harborPasswd = 'Harbor12345'
		HarborAddress = 'harbor.wielun.com'
		harborRepo = 'library'
	}
    stages {
        stage('git拉取代码') {
            steps {
				git credentialsId: '0c71c0f9-8277-493b-xxxx-540a9324cf08', url: 'https://jihulab.com/xxxx/java-demo.git'
            }
        }
    
        stage('maven编译') { 
           steps {
                    sh '''JAVA_HOME=/usr/local/jdk
                    PATH=$PATH:$JAVA_HOME/bin
                    /usr/local/maven/bin/mvn clean package -Dmaven.test.skip=true'''
                }
        }
        stage('生成自定义镜像') { 
           steps {
                    sh '''docker build -t ${JOB_NAME}:latest .'''
                }
        }
        stage('上传自定义镜像到harbor') { 
           steps {
                    sh '''docker login -u ${harborUser} -p ${harborPasswd} ${HarborAddress}
                    docker tag ${JOB_NAME}:latest ${HarborAddress}/${harborRepo}/${JOB_NAME}:latest
                    docker push ${HarborAddress}/${harborRepo}/${JOB_NAME}:latest'''
                }
        }
        stage('发送yaml到k8s-master并部署') { 
           steps {
					sshPublisher(publishers: [sshPublisherDesc(configName: 'k8s-master', transfers: [sshTransfer(cleanRemote: false, excludes: '', execCommand: '''
					/usr/local/bin/kubectl apply -f /tmp/${JOB_NAME}/pipeline.yaml''', execTimeout: 120000, flatten: false, makeEmptyDirs: false, noDefaultExcludes: false, patternSeparator: '[, ]+', remoteDirectory: '${JOB_NAME}', remoteDirectorySDF: false, removePrefix: '', sourceFiles: 'pipeline.yaml')], usePromotionTimestamp: false, useWorkspaceInPromotion: false, verbose: false)])
                }
        } 
    }
}

（3）配置pipeline.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: test
  name: pipeline
  labels:
    app: pipeline
spec:
  replicas: 2
  selector:
    matchLabels:
      app: pipeline
  template:
    metadata:
      labels:
        app: pipeline
    spec:
      containers:
      - name: pipeline
        image: harbor.wielun.com/library/java-k8s:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  namespace: test
  name: pipeline
  labels:
    app: pipeline
spec:
  ports:
  - port: 8081
    targetPort: 8080
  selector:
    app: pipeline
  type: NodePort
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: test
  name: pipeline
spec:
  ingressClassName: nginx
  rules:
  - host: "harbor.wielun.com"
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: pipeline
            port:
              number: 8081

4、创建namespace

[root@master01 ~]# kubectl create ns test

5、构建并查看结果

[root@master01 ~]# kubectl get pod -n test
NAME                        READY   STATUS    RESTARTS   AGE
pipeline-556759f7b4-7x8ml   1/1     Running   0          11s
pipeline-556759f7b4-zwdgr   1/1     Running   0          11s

风语者！平时喜欢研究各种技术，目前在从事后端开发工作，热爱生活、热爱工作。

上一篇
计算机网络3（网络层）

下一篇
paddleseg模型解说——fcn

站长推荐

stm32使用HAL库配置串口中断收发数据（保姆级教程）
stm32使用HAL库配置串口中断收发数据（保姆级教程）
SpringSecurity实现前后端分离认证授权
SpringSecurity实现前后端分离认证授权
U8W/U8W-Mini使用与常见问题解决
U8W/U8W-Mini使用与常见问题解决
【社区图书馆】伴我前行的一本书《The C Programming Language》
【社区图书馆】伴我前行的一本书《The C Programming Language》
【云原生进阶之容器】第六章容器网络6.6.1--Cilium网络方案概述
【云原生进阶之容器】第六章容器网络6.6.1--Cilium网络方案概述

您现在的位置是：首页 >学无止境 >jenkins发布Kubernetes(K8s)集群(基于containerd)网站首页学无止境

jenkins发布Kubernetes(K8s)集群(基于containerd)

一、实验环境

1、k8s环境

2、jenkins环境

二、docker-compose安装

1、下载

2、安装

3、查看版本

三、cfssl证书生成

1、安装cfssl

2、ca生成证书

3、过期时间查看

4、创建Harbor证书

四、安装harbor

1、下载

2、安装

3、创建登录证书

4、修改daemon.json

5、添加hosts

6、登录验证

7、docker上传镜像测试

五、K8s（containerd）拉取镜像（每台机器）

1、K8s（containerd）拉取镜像（跳过证书）

2、K8s（containerd）拉取镜像（通过证书）

3、测试拉取镜像

六、jenkins发布到K8S

1、推送tomcat到harbor

2、创建项目

3、查看项目文件

4、创建namespace

5、构建并查看结果

上一篇 计算机网络3（网络层）

下一篇 paddleseg模型解说——fcn

站长推荐

上一篇
计算机网络3（网络层）

下一篇
paddleseg模型解说——fcn