k8s中删除处于Terminating状态的namespace

每当删除 namespace 或 pod 等一些 Kubernetes 资源时，有时资源状态会卡在 Terminating，很长时间无法删除，甚至有时增加 --force grace-period=0 之后还是无法正常删除。这时就需要 edit 该资源，或者将该资源导出为 json（通过调用原生接口进行删除）, 将 finalizers 字段设置为 []，之后 Kubernetes 资源就正常删除了。

查看 ns 状态

[root@master1 ~]# kubectl get ns
NAME                          STATUS        AGE
cattle-fleet-system           Active        7d19h
cattle-impersonation-system   Active        7d19h
cattle-system                 Active        7d19h
default                       Active        8d
kube-node-lease               Active        8d
kube-public                   Active        8d
kube-system                   Active        8d
local                         Active        7d19h
monitoring                    Terminating   7d19h
test                          Active        23h

可以看到 monitoring 这个 namespace 一直处于Terminating状态，一般情况下强删是删不掉的，强删的方法如下：这个方法在这里会卡主，也是删不掉的

[root@master1 ~]# kubectl delete ns monitoring --force --grace-period=0

如果删不掉，就参考下面的方法

获取 namespace 的 json 文件

[root@master1 ~]# kubectl get ns monitoring -o json > /tmp/monitoring.json

查看monitoring.json的内容

{
  "kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
    "name": "monitoring",
    "uid": "4db2f7a3-5fb2-434a-a2d2-59c70453c1b7",
    "resourceVersion": "2896381",
    "creationTimestamp": "2023-05-17T06:51:39Z",
    "deletionTimestamp": "2023-05-24T09:08:47Z",
    "deletionGracePeriodSeconds": 0,
    "labels": {
      "field.cattle.io/projectId": "p-8bqcb",
      "kubernetes.io/metadata.name": "monitoring"
    },
    "annotations": {
      "cattle.io/status": "{"Conditions":[{"Type":"ResourceQuotaInit","Status":"True","Message":"","LastUpdateTime":"2023-05-17T06:51:40Z"},{"Type":"InitialRolesPopulated","Status":"True","Message":"","LastUpdateTime":"2023-05-17T06:51:41Z"}]}",
      "field.cattle.io/projectId": "c-m-bxmlx5bv:p-8bqcb",
      "kubectl.kubernetes.io/last-applied-configuration": "{"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"name":"monitoring"}}
"},
  "spec": {
      "finalizers": [
          "kubernetes"
      ]
  },
  "status": {
      "phase": "Terminating"
  }
}

修改此monitoring.json文件内容为：

{
"kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
    "name": "monitoring",
    "uid": "4db2f7a3-5fb2-434a-a2d2-59c70453c1b7",
    "resourceVersion": "2896381",
    "creationTimestamp": "2023-05-17T06:51:39Z",
    "deletionTimestamp": "2023-05-24T09:08:47Z",
    "deletionGracePeriodSeconds": 0,
    "labels": {
      "field.cattle.io/projectId": "p-8bqcb",
      "kubernetes.io/metadata.name": "monitoring"
    },
    "annotations": {
      "cattle.io/status": "{"Conditions":[{"Type":"ResourceQuotaInit","Status":"True","Message":"","LastUpdateTime":"2023-05-17T06:51:40Z"},{"Type":"InitialRolesPopulated","Status":"True","Message":"","LastUpdateTime":"2023-05-17T06:51:41Z"}]}",
      "field.cattle.io/projectId": "c-m-bxmlx5bv:p-8bqcb",
      "kubectl.kubernetes.io/last-applied-configuration": "{"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"name":"monitoring"}}
"},
  "spec": {
  },
  "status": {
      "phase": "Terminating"
  }
}

调用 api-server 接口进行删除

打开一个新的终端，或者把下面的命令放到后台执行

[root@master1 prometheus]# kubectl proxy
Starting to serve on 127.0.0.1:8001

调用接口删除

[root@master1 ~]# curl -k -H "Content-Type: application/json" -X PUT --data-binary @monitoring.json http://127.0.0.1:8001/api/v1/namespaces/monitoring/finalize
{
  "kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
    "name": "monitoring",
    "uid": "4db2f7a3-5fb2-434a-a2d2-59c70453c1b7",
    "resourceVersion": "2896381",
    "creationTimestamp": "2023-05-17T06:51:39Z",
    "deletionTimestamp": "2023-05-24T09:08:47Z",
    "deletionGracePeriodSeconds": 0,
    "labels": {
      "field.cattle.io/projectId": "p-8bqcb",
      "kubernetes.io/metadata.name": "monitoring"
  },
  "spec": {
  },
  "status": {
    "phase": "Terminating"
  }
}

输出以上内容表示删除成功。

注: 如果kubectl get ns monitoring -o json的结果中"spec": {}中为空，则需要看下metadata部分是否有finalizers字段，如下以cattle-system所示：

{
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "annotations": {
      "cattle.io/status": "{"Conditions":[{"Type":"ResourceQuotaInit","Status":"True","Message":"","LastUpdateTime":"2023-05-17T06:51:40Z"},{"Type":"InitialRolesPopulated","Status":"True","Message":"","LastUpdateTime":"2023-05-17T06:51:41Z"}]}",
      "field.cattle.io/projectId": "c-m-bxmlx5bv:p-8bqcb",
      "kubectl.kubernetes.io/last-applied-configuration": "{"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"name":"monitoring"}}
",
      "lifecycle.cattle.io/create.namespace-auth": "true"
    },
    "creationTimestamp": "2023-05-17T06:51:39Z",
    "deletionTimestamp": "2023-05-24T09:08:47Z",
    "deletionGracePeriodSeconds": 0,
        "finalizers": [
            "controller.cattle.io/namespace-auth"
        ],
        "labels": {
            "field.cattle.io/projectId": "p-wfknh"
        },
        "name": "cattle-system",
        "resourceVersion": "165368",
        "selfLink": "/api/v1/namespaces/cattle-system",
        "uid": "223ad163-507c-4efe-b3a3-d3bc4b7a5211"
    },
    "spec": {},
    "status": {
        "phase": "Terminating"
    }
}

这里"spec": {},结果为空，无论我们怎么执行，此时此 ns 都不会被删除，此时想弄清楚这个问题，需要先了解下finalizers这个的含义 Finalizers 字段属于 Kubernetes GC 垃圾收集器，是一种删除拦截机制，能够让控制器实现异步的删除前（Pre-delete）回调。其存在于任何一个资源对象的 Meta 中，在 k8s 源码中声明为 []string，该 Slice 的内容为需要执行的拦截器名称。
对带有 Finalizer 的对象的第一个删除请求会为其 metadata.deletionTimestamp 设置一个值，但不会真的删除对象。一旦此值被设置，finalizers 列表中的值就只能被移除。
当 metadata.deletionTimestamp 字段被设置时，负责监测该对象的各个控制器会通过轮询对该对象的更新请求来执行它们所要处理的所有 Finalizer。 当所有 Finalizer 都被执行过，资源被删除。
metadata.deletionGracePeriodSeconds 的取值控制对更新的轮询周期。
每个控制器要负责将其 Finalizer 从列表中去除。
每执行完一个就从 finalizers 中移除一个，直到 finalizers 为空，之后其宿主资源才会被真正的删除。
看到这里有 finalizers 时，需要在导出的json文件中将finalizers的内容也删除，并再次调用接口删除。方法如下：

"finalizers": [
        ],

[root@master1 ~]# curl -k -H "Content-Type: application/json" -X PUT --data-binary @monitoring.json http://127.0.0.1:8001/api/v1/namespaces/monitoring/finalize