您现在的位置是：首页 >技术教程 >JuiceFS-K8s部署网站首页技术教程

JuiceFS-K8s部署

IsIands 2024-06-11 18:01:02
简介JuiceFS-K8s部署

目录

官网文档地址：https://juicefs.com/docs/zh/csi/introduction/

1、部署JuiceFS-CSI驱动

部署yaml如下，镜像已进行更改为镜像仓库内地址

# DO NOT EDIT: generated by 'kustomize build'
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/instance: juicefs-csi-driver
    app.kubernetes.io/name: juicefs-csi-driver
    app.kubernetes.io/version: master
  name: juicefs-csi-controller-sa
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/instance: juicefs-csi-driver
    app.kubernetes.io/name: juicefs-csi-driver
    app.kubernetes.io/version: master
  name: juicefs-csi-node-sa
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/instance: juicefs-csi-driver
    app.kubernetes.io/name: juicefs-csi-driver
    app.kubernetes.io/version: master
  name: juicefs-csi-external-node-service-role
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - pods/log
  - secrets
  verbs:
  - get
  - list
  - create
  - update
  - delete
  - patch
  - watch
- apiGroups:
  - batch
  resources:
  - jobs
  verbs:
  - get
  - list
  - create
  - update
  - delete
  - patch
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/proxy
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - persistentvolumes
  verbs:
  - get
  - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/instance: juicefs-csi-driver
    app.kubernetes.io/name: juicefs-csi-driver
    app.kubernetes.io/version: master
  name: juicefs-external-provisioner-role
rules:
- apiGroups:
  - ""
  resources:
  - persistentvolumes
  verbs:
  - get
  - list
  - watch
  - create
  - delete
- apiGroups:
  - ""
  resources:
  - persistentvolumeclaims
  verbs:
  - get
  - list
  - watch
  - update
- apiGroups:
  - storage.k8s.io
  resources:
  - storageclasses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
- apiGroups:
  - storage.k8s.io
  resources:
  - csinodes
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
  - list
  - create
  - update
  - patch
  - delete
- apiGroups:
  - ""
  resources:
  - pods
  - pods/log
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
- apiGroups:
  - batch
  resources:
  - jobs
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
- apiGroups:
  - ""
  resources:
  - endpoints
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/instance: juicefs-csi-driver
    app.kubernetes.io/name: juicefs-csi-driver
    app.kubernetes.io/version: master
  name: juicefs-csi-node-service-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: juicefs-csi-external-node-service-role
subjects:
- kind: ServiceAccount
  name: juicefs-csi-node-sa
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/instance: juicefs-csi-driver
    app.kubernetes.io/name: juicefs-csi-driver
    app.kubernetes.io/version: master
  name: juicefs-csi-provisioner-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: juicefs-external-provisioner-role
subjects:
- kind: ServiceAccount
  name: juicefs-csi-controller-sa
  namespace: kube-system
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: juicefs-csi-driver
    app.kubernetes.io/name: juicefs-csi-driver
    app.kubernetes.io/version: master
  name: juicefs-csi-controller
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: juicefs-csi-controller
      app.kubernetes.io/instance: juicefs-csi-driver
      app.kubernetes.io/name: juicefs-csi-driver
      app.kubernetes.io/version: master
  serviceName: juicefs-csi-controller
  template:
    metadata:
      labels:
        app: juicefs-csi-controller
        app.kubernetes.io/instance: juicefs-csi-driver
        app.kubernetes.io/name: juicefs-csi-driver
        app.kubernetes.io/version: master
    spec:
      containers:
      - args:
        - --endpoint=$(CSI_ENDPOINT)
        - --logtostderr
        - --nodeid=$(NODE_NAME)
        - --v=5
        env:
        - name: CSI_ENDPOINT
          value: unix:///var/lib/csi/sockets/pluginproxy/csi.sock
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: JUICEFS_MOUNT_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: HOST_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        - name: KUBELET_PORT
          value: "10250"
        - name: JUICEFS_MOUNT_PATH
          value: /var/lib/juicefs/volume
        - name: JUICEFS_CONFIG_PATH
          value: /var/lib/juicefs/config
        image: harbor.t3caic.com/juicefs/juicefs-csi-driver:v0.17.5
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /healthz
            port: healthz
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 3
        name: juicefs-plugin
        ports:
        - containerPort: 9909
          name: healthz
          protocol: TCP
        resources:
          limits:
            cpu: 1000m
            memory: 1Gi
          requests:
            cpu: 100m
            memory: 512Mi
        securityContext:
          capabilities:
            add:
            - SYS_ADMIN
          privileged: true
        volumeMounts:
        - mountPath: /var/lib/csi/sockets/pluginproxy/
          name: socket-dir
        - mountPath: /jfs
          mountPropagation: Bidirectional
          name: jfs-dir
        - mountPath: /root/.juicefs
          mountPropagation: Bidirectional
          name: jfs-root-dir
      - args:
        - --csi-address=$(ADDRESS)
        - --timeout=60s
        - --v=5
        env:
        - name: ADDRESS
          value: /var/lib/csi/sockets/pluginproxy/csi.sock
        image: harbor.t3caic.com/juicefs/csi-provisioner:v1.6.0
        name: csi-provisioner
        volumeMounts:
        - mountPath: /var/lib/csi/sockets/pluginproxy/
          name: socket-dir
      - args:
        - --csi-address=$(ADDRESS)
        - --health-port=$(HEALTH_PORT)
        env:
        - name: ADDRESS
          value: /csi/csi.sock
        - name: HEALTH_PORT
          value: "9909"
        image: harbor.t3caic.com/juicefs/livenessprobe:v1.1.0
        name: liveness-probe
        volumeMounts:
        - mountPath: /csi
          name: socket-dir
      priorityClassName: system-cluster-critical
      serviceAccount: juicefs-csi-controller-sa
      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
      volumes:
      - emptyDir: {}
        name: socket-dir
      - hostPath:
          path: /var/lib/juicefs/volume
          type: DirectoryOrCreate
        name: jfs-dir
      - hostPath:
          path: /var/lib/juicefs/config
          type: DirectoryOrCreate
        name: jfs-root-dir
  volumeClaimTemplates: []
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app.kubernetes.io/component: node
    app.kubernetes.io/instance: juicefs-csi-driver
    app.kubernetes.io/name: juicefs-csi-driver
    app.kubernetes.io/version: master
  name: juicefs-csi-node
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: juicefs-csi-node
      app.kubernetes.io/instance: juicefs-csi-driver
      app.kubernetes.io/name: juicefs-csi-driver
      app.kubernetes.io/version: master
  template:
    metadata:
      labels:
        app: juicefs-csi-node
        app.kubernetes.io/instance: juicefs-csi-driver
        app.kubernetes.io/name: juicefs-csi-driver
        app.kubernetes.io/version: master
    spec:
      containers:
      - args:
        - --endpoint=$(CSI_ENDPOINT)
        - --logtostderr
        - --nodeid=$(NODE_NAME)
        - --v=5
        - --enable-manager=true
        env:
        - name: CSI_ENDPOINT
          value: unix:/csi/csi.sock
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: JUICEFS_MOUNT_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: HOST_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        - name: KUBELET_PORT
          value: "10250"
        - name: JUICEFS_MOUNT_PATH
          value: /var/lib/juicefs/volume
        - name: JUICEFS_CONFIG_PATH
          value: /var/lib/juicefs/config
        image: harbor.t3caic.com/juicefs/juicefs-csi-driver:v0.17.5
        lifecycle:
          preStop:
            exec:
              command:
              - /bin/sh
              - -c
              - rm /csi/csi.sock
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /healthz
            port: healthz
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 3
        name: juicefs-plugin
        ports:
        - containerPort: 9909
          name: healthz
          protocol: TCP
        resources:
          limits:
            cpu: 1000m
            memory: 1Gi
          requests:
            cpu: 100m
            memory: 512Mi
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: /var/lib/kubelet
          mountPropagation: Bidirectional
          name: kubelet-dir
        - mountPath: /csi
          name: plugin-dir
        - mountPath: /dev
          name: device-dir
        - mountPath: /jfs
          mountPropagation: Bidirectional
          name: jfs-dir
        - mountPath: /root/.juicefs
          mountPropagation: Bidirectional
          name: jfs-root-dir
      - args:
        - --csi-address=$(ADDRESS)
        - --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
        - --v=5
        env:
        - name: ADDRESS
          value: /csi/csi.sock
        - name: DRIVER_REG_SOCK_PATH
          value: /var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock
        image: harbor.t3caic.com/juicefs/csi-node-driver-registrar:v2.1.0
        name: node-driver-registrar
        volumeMounts:
        - mountPath: /csi
          name: plugin-dir
        - mountPath: /registration
          name: registration-dir
      - args:
        - --csi-address=$(ADDRESS)
        - --health-port=$(HEALTH_PORT)
        env:
        - name: ADDRESS
          value: /csi/csi.sock
        - name: HEALTH_PORT
          value: "9909"
        image: harbor.t3caic.com/juicefs/livenessprobe:v1.1.0
        name: liveness-probe
        volumeMounts:
        - mountPath: /csi
          name: plugin-dir
      dnsPolicy: ClusterFirstWithHostNet
      priorityClassName: system-node-critical
      serviceAccount: juicefs-csi-node-sa
      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
      volumes:
      - hostPath:
          path: /var/lib/kubelet
          type: Directory
        name: kubelet-dir
      - hostPath:
          path: /var/lib/kubelet/csi-plugins/csi.juicefs.com/
          type: DirectoryOrCreate
        name: plugin-dir
      - hostPath:
          path: /var/lib/kubelet/plugins_registry/
          type: Directory
        name: registration-dir
      - hostPath:
          path: /dev
          type: Directory
        name: device-dir
      - hostPath:
          path: /var/lib/juicefs/volume
          type: DirectoryOrCreate
        name: jfs-dir
      - hostPath:
          path: /var/lib/juicefs/config
          type: DirectoryOrCreate
        name: jfs-root-dir
---
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  labels:
    app.kubernetes.io/instance: juicefs-csi-driver
    app.kubernetes.io/name: juicefs-csi-driver
    app.kubernetes.io/version: master
  name: csi.juicefs.com
spec:
  attachRequired: false
  podInfoOnMount: false

2、创建OBS认证信息Secret

需要修改的信息
metaurl: “用户名:密码@IP:端口/库名”
postgres://用户名:密码@IP:端口/库名 数据库需要提前进入进行创建库

storage: “obs” 存储类型标识
bucket: “” 桶的域名
access-key: “”
secret-key: “”

apiVersion: v1
kind: Secret
metadata:
  name: juicefs-sc-secret
  namespace: kube-system
type: Opaque
stringData:
  name: "test"
  metaurl: ""
  storage: ""
  bucket: ""
  access-key: ""
  secret-key: ""

3、创建存储类

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: juicefs-sc
provisioner: csi.juicefs.com
reclaimPolicy: Retain
volumeBindingMode: Immediate
parameters:
  csi.storage.k8s.io/node-publish-secret-name: juicefs-sc-secret
  csi.storage.k8s.io/node-publish-secret-namespace: kube-system
  csi.storage.k8s.io/provisioner-secret-name: juicefs-sc-secret
  csi.storage.k8s.io/provisioner-secret-namespace: kube-system

4、创建PVC–PVC创建时会自动创建PV

目前而言，JuiceFS CSI 驱动不支持设置存储容量。在 PersistentVolume 和 PersistentVolumeClaim 中指定的容量会被忽略，填写任意有效值即可，例如 10Gi

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: "juicefs-test-pvc"
  namespace: test-juicefs
spec:
  storageClassName: sc-topolvm
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 10Gi
  storageClassName: juicefs-sc

5、创建测试Pod–测试Pod创建容器内是否挂载成功

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-run
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: harbor.t3caic.com/nginxdemos/hello:plain-text
          ports:
            - containerPort: 80
          volumeMounts:
            - mountPath: /config
              name: web-data
      volumes:
        - name: web-data
          persistentVolumeClaim:
            claimName: juicefs-test-pvc

ERR注意：如果测试Pod创建卡死，查看kube-system命名空间的juicefs-XXX-pvc容器起来了吗
如因为拉取不下镜像，请在此容器所在节点手动解决Pod拉取 docker pull
如果镜像拉取下来Pod一直在重启可以考虑一下是否是OBS连接不上，如果OBS需要做域名解析可以在CoreDNS里配置

kubectl edit configmap coredns -n kube-system
添加
 hosts {
          100.125.32.189 juicefs.obs.cn-east-263.t3caic.com
          fallthrough
        }
风语者！平时喜欢研究各种技术，目前在从事后端开发工作，热爱生活、热爱工作。

站长推荐