K8S快速入门-3基于智能114项目的自己实战

k8s快速入门-3基于智能114项目的自己实战

1. 项目设计结构

ttt

2023.1.26：h9 h7 h1部署后的整体图

image-20230126210032378

2. 阿里云容器镜像服务

https://blog.csdn.net/qq_23034755/article/details/126646493

2.1 创建镜像仓库

image-20230125230901447

image-20230125215819092

2.1 操作指南（注意，一个服务搞到一个里面去）

image-20230125215853559

2.2 按照操作指南进行操作

2.2.1 redis推送到Registry

  • docker images查看redis
1
2
REPOSITORY                                                        TAG        IMAGE ID       CREATED         SIZE
114_redis                                                         v1         614e3bc05a87   8 weeks ago     169MB
  • 将镜像推送到Registry
1
2
3
$ docker login --username=curiousliu registry.cn-hangzhou.aliyuncs.com
$ docker tag 614e3bc05a87 registry.cn-hangzhou.aliyuncs.com/curious/114_redis:v1
$ docker push registry.cn-hangzhou.aliyuncs.com/curious/114_redis:v1
  • 推送成功后查看一下

image-20230125222919312

2.2.2 nginx推送到Registry

注意，这里需要预定一下后端的一个clusterIP，很多配置可能要根据这里的clusterIP来进行配置

之前redis的clusterIP定义为10.109.179.231，于是这里后端（nginx、uwsgi+django的clusterIP定义为10.109.179.232）

注意nginx配置文件中相应条目的修改 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
server {
        listen       8082;
        server_name  10.109.179.232;

        #charset koi8-r;

        #access_log  logs/host.access.log  main;
        proxy_buffer_size 64k;
        proxy_buffers 4 32k;
        proxy_busy_buffers_size 64k;
        proxy_temp_file_write_size 64k;

        location / {
            include uwsgi_params;
            uwsgi_pass 10.109.179.232:5002;  # 用这个表示
            # uwsgi_pass unix:/tmp/_114.sock;
            # uwsgi_send_timeout 600;
            # uwsgi_read_timeout 600;
            # uwsgi_connect_timeout 600;
            # uwsgi_buffer_size 32k;
            # uwsgi_buffers 4 32k;
            # uwsgi_busy_buffers_size 64k;
            # uwsgi_temp_file_write_size 64k;
        }

        #error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }

重新build一遍（请参考以前的内容）

  • docker images查看nginx
1
2
bupt@h9:~$ sudo docker images | grep nginx
114_nginx                                                         v1         ea5c03878541   3 minutes ago   198MB
  • 将镜像推送到Registry
1
2
3
$ docker login --username=curiousliu registry.cn-hangzhou.aliyuncs.com
$ docker tag ea5c03878541 registry.cn-hangzhou.aliyuncs.com/curious/114_nginx:v1
$ docker push registry.cn-hangzhou.aliyuncs.com/curious/114_nginx:v1

WARNING! Your password will be stored unencrypted in /root/.docker/config.json.

2.2.3 django+uwsgi推送到Registry

注意，这里需要预定一下后端的一个clusterIP，很多配置可能要根据这里的clusterIP来进行配置

之前redis的clusterIP定义为10.109.179.231，于是这里后端（nginx、uwsgi+django的clusterIP定义为10.109.179.232）

注意好几个地方相应条目的修改

  • redis（views.py，Entrance.py）

image-20230126001239642

  • uwsgi.ini
1
socket = 10.109.179.231:5002
  • docker images查看114_uwsgi_django
1
2
bupt@h9:~$ sudo docker images | grep 114_uwsgi_django
114_uwsgi_django                                                  v1         64992d7a2790   About a minute ago   8.62GB
  • 将镜像推送到Registry
1
2
3
$ docker login --username=curiousliu registry.cn-hangzhou.aliyuncs.com
$ docker tag 64992d7a2790 registry.cn-hangzhou.aliyuncs.com/curious/114_uwsgi_django:v1
$ docker push registry.cn-hangzhou.aliyuncs.com/curious/114_uwsgi_django:v1

3. 缓存部分（redis）

先不要忘记创建namespace（114dev）

1
2
bupt@h9:~/k8s$ sudo kubectl create ns 114dev
namespace/114dev created

3.1 deployment（注意私有镜像的权限问题）

  • 私有镜像权限问题

https://blog.csdn.net/winterfeng123/article/details/113752887

1
2
3
4
5
6
7
kubectl create secret docker-registry 114devsecret \
    --docker-server=registry.cn-hangzhou.aliyuncs.com \
    --docker-username=curiousliu \
    --docker-password=<passwd，需要根据自己的来输入> \
    --docker-email= \
    -n 114dev
> secret/114devsecret created
  • deploy-114_redis.yaml（注：yaml文件不能用下划线_）
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
apiVersion: apps/v1
kind: Deployment
metadata:
  name: 114-redis
  namespace: 114dev
spec:
  replicas: 3
  selector:
    matchLabels:
      run: 114-redis
  template:
    metadata:
      labels:
        run: 114-redis
    spec:
      containers:
      - image: registry.cn-hangzhou.aliyuncs.com/curious/114_redis:v1
        name: 114-redis
        ports:
        - containerPort: 6380
          protocol: TCP
      imagePullSecrets:
      - name: 114devsecret
  • 创建&验证（等待一段时间，查看）
1
2
3
4
5
bupt@h9:~/k8s$ sudo kubectl create -f deploy-114_redis.yaml
deployment.apps/114-redis created
bupt@h9:~/k8s$ sudo kubectl get deploy -n 114dev
NAME        READY   UP-TO-DATE   AVAILABLE   AGE
114-redis   3/3     3            3           17s
  • 删除
1
2
bupt@h9:~/k8s$ sudo kubectl delete -f deploy-114_redis.yaml
deployment.apps "114-redis" deleted
  • 迟迟不redis的调试报错方法
1
2
3
4
5
6
7
8
9
10
# deploy状态不ok
bupt@h9:~/k8s$ sudo kubectl get deploy -n 114dev
NAME        READY   UP-TO-DATE   AVAILABLE   AGE
114-redis   0/3     3            0           17m
# pod状态显示ImagePullBackOff
bupt@h9:~/k8s$ sudo kubectl get pod -o wide -n 114dev
NAME                         READY   STATUS             RESTARTS   AGE   IP           NODE   NOMINATED NODE   READINESS GATES
114-redis-6f6dc67f59-c4qzm   0/1     ImagePullBackOff   0          17m   10.244.1.5   h7     <none>           <none>
114-redis-6f6dc67f59-cw2zk   0/1     ImagePullBackOff   0          17m   10.244.1.6   h7     <none>           <none>
114-redis-6f6dc67f59-stpqv   0/1     ImagePullBackOff   0          17m   10.244.1.4   h7     <none>           <none>

3.2 service（附debug过程和解决，主要是私有镜像库的权限问题）

svc-114_redis.yaml（注：这里NodePort是向外部暴露端口的方式，然后clusterIP就随便写一个内部的IP）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
apiVersion: v1
kind: Service
metadata:
  name: svc-114-redis
  namespace: 114dev
spec:
  clusterIP: 10.109.179.231
  ports:
  - port: 6380
    protocol: TCP
    targetPort: 6380
  selector:
    run: 114-redis
  type: NodePort
  • 创建&验证
1
2
3
4
5
bupt@h9:~/k8s$ sudo kubectl create -f svc-114_redis.yaml 
service/svc-114-redis created
bupt@h9:~/k8s$ sudo kubectl get svc svc-114-redis -n 114dev -o wide
NAME            TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE   SELECTOR
svc-114-redis   NodePort   10.109.179.231   <none>        6380:31190/TCP   9s    run=114-redis
  • 删除
1
2
bupt@h9:~/k8s$ sudo kubectl delete -f svc-114_redis.yaml
service "svc-114-redis" deleted

3.3 使用curl命令进行redis的测试

好像有些问题，因为这里redis也有些鉴权操作：https://stackoverflow.com/questions/33243121/abuse-curl-to-communicate-with-redis

1
2
3
4
5
bupt@h9:~/k8s$ curl --http0.9 -w "\n" -i -X GET http://10.109.246.xxx:31190
-ERR wrong number of arguments for 'get' command
-NOAUTH Authentication required.
-ERR unknown command `User-Agent:`, with args beginning with: `curl/7.68.0`, 
-ERR unknown command `Accept:`, with args beginning with: `*/*`,

至少不是普通的报错了，说明这里还是有些内容的

3.4 整体查看

image-20230125234120241

4. 后端部分（nginx、uwsgi+django）

首先对于nginx、uwsgi+django这个部分，需要记住clusterIP的一个自定义的配置，之后会在主图上有体现

4.1 deployment

  • deploy-114_backend.yaml（注：yaml文件不能用下划线_）
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
apiVersion: apps/v1
kind: Deployment
metadata:
  name: 114-backend
  namespace: 114dev
spec:
  replicas: 3
  selector:
    matchLabels:
      run: 114-backend
  template:
    metadata:
      labels:
        run: 114-backend
    spec:
      containers:
      - image: registry.cn-hangzhou.aliyuncs.com/curious/114_uwsgi_django:v1
        name: 114-uwsgi-django
        ports:
        - containerPort: 5002
          protocol: TCP
      - image: registry.cn-hangzhou.aliyuncs.com/curious/114_nginx:v1
        name: 114-nginx
        ports:
        - containerPort: 8082
          protocol: TCP
      imagePullSecrets:
      - name: 114devsecret
  • 创建&验证（等待一段时间，查看）
1
2
3
4
5
6
7
bupt@h9:~/k8s$ sudo kubectl create -f deploy-114_backend.yaml
deployment.apps/114-backend created
bupt@h9:~/k8s$ sudo kubectl get deploy -n 114dev
NAME          READY   UP-TO-DATE   AVAILABLE   AGE
114-backend   3/3     3            3           14m
114-redis     3/3     3            3           94m
bupt@h9:~/k8s$ sudo kubectl describe pod -n 114dev
  • 删除
1
bupt@h9:~/k8s$ sudo kubectl delete -f deploy-114_backend.yaml

4.2 service

svc-114_backend.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
apiVersion: v1
kind: Service
metadata:
  name: svc-114-backend
  namespace: 114dev
spec:
  clusterIP: 10.109.179.232
  ports:
  - name: nginx
    port: 8082
    targetPort: 8082
    protocol: TCP
  - name: uwsgi
    port: 5002
    targetPort: 5002
    protocol: TCP
  selector:
    run: 114-backend
  type: NodePort
  • 创建&验证
1
2
3
bupt@h9:~/k8s$ sudo kubectl create -f svc-114_backend.yaml
service/svc-114-backend created
bupt@h9:~/k8s$ sudo kubectl get svc svc-114-backend -n 114dev -o wide
  • 删除
1
2
bupt@h9:~/k8s$ sudo kubectl delete -f svc-114_backend.yaml
service "svc-114_backend" deleted

5. 整体查看+curl测试

5.1 整体查看

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
bupt@h9:~/k8s$ sudo kubectl get pod,deploy,svc -n 114dev -o wide
NAME                             READY   STATUS    RESTARTS   AGE   IP            NODE   NOMINATED NODE   READINESS GATES
pod/114-backend-78f69c78-2vlgq   2/2     Running   0          9h    10.244.1.10   h7     <none>           <none>
pod/114-backend-78f69c78-hw5h7   2/2     Running   0          9h    10.244.1.12   h7     <none>           <none>
pod/114-backend-78f69c78-lz6ds   2/2     Running   0          9h    10.244.1.11   h7     <none>           <none>
pod/114-redis-fffd7f8b-282hn     1/1     Running   0          10h   10.244.1.9    h7     <none>           <none>
pod/114-redis-fffd7f8b-59hpw     1/1     Running   0          10h   10.244.1.7    h7     <none>           <none>
pod/114-redis-fffd7f8b-6gf7d     1/1     Running   0          10h   10.244.1.8    h7     <none>           <none>

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS                   IMAGES                                                                                                                 SELECTOR
deployment.apps/114-backend   3/3     3            3           9h    114-uwsgi-django,114-nginx   registry.cn-hangzhou.aliyuncs.com/curious/114_uwsgi_django:v1,registry.cn-hangzhou.aliyuncs.com/curious/114_nginx:v1   run=114-backend
deployment.apps/114-redis     3/3     3            3           10h   114-redis                    registry.cn-hangzhou.aliyuncs.com/curious/114_redis:v1                                                                 run=114-redis

NAME                      TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE   SELECTOR
service/svc-114-backend   NodePort   10.109.179.232   <none>        8082:30860/TCP,5002:31894/TCP   9h    run=114-backend
service/svc-114-redis     NodePort   10.109.179.231   <none>        6380:31137/TCP                  9h    run=114-redis

5.2 curl测试

1
2
3
4
5
bupt@h9:~/k8s$ curl --location --request POST 'http://10.109.246.xxx(h9):30860/_114/' --header 'Content-Type: text/plain' --data '{ "commonVersion": "1.0", "taskId": 1, "callId": "20220126_0055", "dialog": "begin", "message": "上海科技大学。", "called": "12345678", "origcall": "87654321" }'
{"resultTag": "确认查号结果", "resultInfo": {"name": "上海科技大学", "number": "test12345678"}}
bupt@h9:~/k8s$ 
bupt@h9:~/k8s$ curl --location --request POST 'http://10.109.246.yyy(h7):30860/_114/' --header 'Content-Type: text/plain' --data '{ "commonVersion": "1.0", "taskId": 1, "callId": "20220126_0055", "dialog": "continue", "message": "可以。", "called": "12345678", "origcall": "87654321" }'
{"resultTag": "播放查号结果", "resultInfo": {"name": "上海科技大学", "number": "test12345678"}

6. 挂载&升级（挂载还有些问题）

原来的114_uwsgi_django需要挂载这些内容：

1
2
3
4
5
6
7
8
9
10
docker run \
    -p 5002:5002 \
    -v /data/sdb2/lyx/mount-files-202211270941/_114config.json:/home/lyx/_114/zfx-114-query-system/_114django/_114config.json \
    -v /data/sdb2/lyx/mount-files-202211270941/Configs:/home/lyx/_114/zfx-114-query-system/_114django/Configs \
    -v /data/sdb2/lyx/mount-files-202211270941/debug-logs:/home/lyx/_114/zfx-114-query-system/_114django/debug-logs \
    -v /data/sdb2/lyx/mount-files-202211270941/logs:/home/lyx/_114/zfx-114-query-system/_114django/logs \
    -v /data/sdb2/lyx/mount-files-202211270941/wsgi_logs:/home/lyx/_114/zfx-114-query-system/_114django/wsgi_logs \
    -v /data/sdb2/lyx/mount-files-202211270941/stopwords.txt:/usr/local/lib/python3.6/site-packages/pyhanlp/static/data/dictionary/stopwords.txt \
    -v /data/sdb2/lyx/mount-files-202211270941/CustomDictionary1.txt:/usr/local/lib/python3.6/site-packages/pyhanlp/static/data/dictionary/custom/CustomDictionary1.txt \
    -d 114_uwsgi_django:v1

现在要想办法把这些在k8s的方式下进行挂载，需要创建configMap，并修改：deploy-114_backend.yaml

尝试使用configMap和直接mount的方式，感觉都有各种各样奇怪的报错，包括下面chatGPT给出的解释：

Q：k8s挂载包含文件夹的文件夹会有问题吗

chatGPT A：在 Kubernetes 中使用 ConfigMap 或者 Secrets 挂载包含文件夹的文件夹时，可能会出现问题。ConfigMap 和 Secrets 都是将文件夹中的文件映射到容器中的文件系统中，如果文件夹中包含文件夹，可能会导致文件夹结构不完整或者文件丢失。为了避免这些问题，建议只在文件夹中挂载单个文件。

6.1 使用configMap挂载所有配置文件（外部->内部）

https://www.cnblogs.com/lizexiong/p/14990537.html

  • /data/sdb2/lyx/mount-files-202211270941/_114config.json（单独文件就用configmap）
1
sudo kubectl create configmap 114config --from-file=/home/bupt/k8s/mount-files-202211270941/_114config.json -n 114dev
  • /data/sdb2/lyx/mount-files-202211270941/stopwords.txt
1
sudo kubectl create configmap stopword --from-file=/home/bupt/k8s/mount-files-202211270941/stopwords.txt -n 114dev
  • /data/sdb2/lyx/mount-files-202211270941/CustomDictionary1.txt
1
sudo kubectl create configmap customdictionary1 --from-file=/home/bupt/k8s/mount-files-202211270941/CustomDictionary1.txt -n 114dev

  • Configs目录下的内容

    • /home/bupt/k8s/mount-files-202211270941/Configs/COMMON_QUERY_WORD.json
    1
    sudo kubectl create configmap commonqueryword --from-file=/home/bupt/k8s/mount-files-202211270941/Configs/COMMON_QUERY_WORD.json -n 114dev
    • /home/bupt/k8s/mount-files-202211270941/Configs/INTENT_CLASSIFICATION_CONFIGS.json
    1
    sudo kubectl create configmap intentconfig --from-file=/home/bupt/k8s/mount-files-202211270941/Configs/INTENT_CLASSIFICATION_CONFIGS.json -n 114dev
    • /home/bupt/k8s/mount-files-202211270941/Configs/QUERY_PHONENUMBER_CONFIGS.json
    1
    sudo kubectl create configmap queryconfig --from-file=/home/bupt/k8s/mount-files-202211270941/Configs/QUERY_PHONENUMBER_CONFIGS.json -n 114dev
    • /home/bupt/k8s/mount-files-202211270941/Configs/HeteromorphicCharacters.txt
    1
    sudo kubectl create configmap heteromorphic --from-file=/home/bupt/k8s/mount-files-202211270941/Configs/HeteromorphicCharacters.txt -n 114dev
    • /home/bupt/k8s/mount-files-202211270941/Configs/IntentClassificationKeywords/doctor.txt
    1
    sudo kubectl create configmap doctor --from-file=/home/bupt/k8s/mount-files-202211270941/Configs/IntentClassificationKeywords/doctor.txt -n 114dev
    • /home/bupt/k8s/mount-files-202211270941/Configs/IntentClassificationKeywords/move.txt
    1
    sudo kubectl create configmap move --from-file=/home/bupt/k8s/mount-files-202211270941/Configs/IntentClassificationKeywords/move.txt -n 114dev
    • /home/bupt/k8s/mount-files-202211270941/Configs/IntentClassificationKeywords/priority.txt
    1
    sudo kubectl create configmap priority --from-file=/home/bupt/k8s/mount-files-202211270941/Configs/IntentClassificationKeywords/priority.txt -n 114dev

  • 更新

1
2
3
bupt@h9:~/k8s$ sudo kubectl create configmap queryconfig --from-file /home/bupt/k8s/mount-files-202211270941/Configs/QUERY_PHONENUMBER_CONFIGS.json -o yaml --dry-run -n 114dev | sudo kubectl replace -f -
W0127 01:20:11.550767 2883207 helpers.go:555] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/queryconfig replaced
  • 展示
1
2
3
4
5
6
7
8
bupt@h9:~/k8s$ sudo kubectl get configmap -n 114dev
NAME                           DATA   AGE
114config                      1      7h28m
configs                        4      6h20m
customdictionary1              1      7h28m
intentclassificationkeywords   3      3s
kube-root-ca.crt               1      23h
stopword                       1      7h28m

6.2 使用PV和PVC挂载日志输出（内部->外部）

TODO 待尝试

6.3 修改deploy（加入了热更新configMap）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
apiVersion: apps/v1
kind: Deployment
metadata:
  name: 114-backend
  namespace: 114dev
spec:
  replicas: 5
  selector:
    matchLabels:
      run: 114-backend
  template:
    metadata:
      labels:
        run: 114-backend
      annotations:
        reloader.stakater.com/auto: "true"
    spec:
      containers:
      - image: registry.cn-hangzhou.aliyuncs.com/curious/114_uwsgi_django:v1
        name: 114-uwsgi-django
        volumeMounts:
        # 文件夹类型的挂载
        # TODO
        # 文件类型的挂载
        - mountPath: /home/lyx/_114/zfx-114-query-system/_114django/_114config.json
          name: 114config
          subPath: _114config.json
        - mountPath: /usr/local/lib/python3.6/site-packages/pyhanlp/static/data/dictionary/stopwords.txt
          name: stopword
          subPath: stopwords.txt
        - mountPath: /usr/local/lib/python3.6/site-packages/pyhanlp/static/data/dictionary/custom/CustomDictionary1.txt
          name: customdictionary1
          subPath: CustomDictionary1.txt
        - mountPath: /home/lyx/_114/zfx-114-query-system/_114django/Configs/COMMON_QUERY_WORD.json
          name: commonqueryword
          subPath: COMMON_QUERY_WORD.json
        - mountPath: /home/lyx/_114/zfx-114-query-system/_114django/Configs/INTENT_CLASSIFICATION_CONFIGS.json
          name: intentconfig
          subPath: INTENT_CLASSIFICATION_CONFIGS.json
        - mountPath: /home/lyx/_114/zfx-114-query-system/_114django/Configs/QUERY_PHONENUMBER_CONFIGS.json
          name: queryconfig
          subPath: QUERY_PHONENUMBER_CONFIGS.json
        - mountPath: /home/lyx/_114/zfx-114-query-system/_114django/Configs/HeteromorphicCharacters.txt
          name: heteromorphic
          subPath: HeteromorphicCharacters.txt
        - mountPath: /home/lyx/_114/zfx-114-query-system/_114django/Configs/IntentClassificationKeywords/doctor.txt
          name: doctor
          subPath: doctor.txt
        - mountPath: /home/lyx/_114/zfx-114-query-system/_114django/Configs/IntentClassificationKeywords/move.txt
          name: move
          subPath: move.txt
        - mountPath: /home/lyx/_114/zfx-114-query-system/_114django/Configs/IntentClassificationKeywords/priority.txt
          name: priority
          subPath: priority.txt
        ports:
        - containerPort: 5002
          protocol: TCP
      - image: registry.cn-hangzhou.aliyuncs.com/curious/114_nginx:v1
        name: 114-nginx
        ports:
        - containerPort: 8082
          protocol: TCP
      volumes:
      # 文件夹类型的挂载
      # TODO
      # 文件类型的挂载
      - name: 114config
        configMap:
          name: 114config
      - name: stopword
        configMap:
          name: stopword
      - name: customdictionary1
        configMap:
          name: customdictionary1
      - name: commonqueryword
        configMap:
          name: commonqueryword
      - name: intentconfig
        configMap:
          name: intentconfig
      - name: queryconfig
        configMap:
          name: queryconfig
      - name: heteromorphic
        configMap:
          name: heteromorphic
      - name: doctor
        configMap:
          name: doctor
      - name: move
        configMap:
          name: move
      - name: priority
        configMap:
          name: priority
      imagePullSecrets:
      - name: 114devsecret
  • apply
1
sudo kubectl apply -f deploy-114_backend.yaml
  • 查看
1
sudo kubectl get pod,deploy,svc -n 114dev -o wide

6.4 热更新（注意deploy中的配置）

/home/bupt/k8s/reloader.yaml替换掉其中的default为自己的这个namespace（114dev）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
---
# Source: reloader/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    meta.helm.sh/release-namespace: "114dev"
    meta.helm.sh/release-name: "reloader"
  labels:
    app: reloader-reloader
    chart: "reloader-v1.0.2"
    release: "reloader"
    heritage: "Helm"
    app.kubernetes.io/managed-by: "Helm"
  name: reloader-reloader
  namespace: 114dev
---
# Source: reloader/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole
metadata:
  annotations:
    meta.helm.sh/release-namespace: "114dev"
    meta.helm.sh/release-name: "reloader"
  labels:
    app: reloader-reloader
    chart: "reloader-v1.0.2"
    release: "reloader"
    heritage: "Helm"
    app.kubernetes.io/managed-by: "Helm"
  name: reloader-reloader-role
  namespace: 114dev
rules:
  - apiGroups:
      - ""
    resources:
      - secrets
      - configmaps
    verbs:
      - list
      - get
      - watch
  - apiGroups:
      - "apps"
    resources:
      - deployments
      - daemonsets
      - statefulsets
    verbs:
      - list
      - get
      - update
      - patch
  - apiGroups:
      - "extensions"
    resources:
      - deployments
      - daemonsets
    verbs:
      - list
      - get
      - update
      - patch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
---
# Source: reloader/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding
metadata:
  annotations:
    meta.helm.sh/release-namespace: "114dev"
    meta.helm.sh/release-name: "reloader"
  labels:
    app: reloader-reloader
    chart: "reloader-v1.0.2"
    release: "reloader"
    heritage: "Helm"
    app.kubernetes.io/managed-by: "Helm"
  name: reloader-reloader-role-binding
  namespace: 114dev
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: reloader-reloader-role
subjects:
  - kind: ServiceAccount
    name: reloader-reloader
    namespace: 114dev
---
# Source: reloader/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    meta.helm.sh/release-namespace: "114dev"
    meta.helm.sh/release-name: "reloader"
  labels:
    app: reloader-reloader
    chart: "reloader-v1.0.2"
    release: "reloader"
    heritage: "Helm"
    app.kubernetes.io/managed-by: "Helm"
    group: com.stakater.platform
    provider: stakater
    version: v1.0.2
  name: reloader-reloader
  namespace: 114dev
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: reloader-reloader
      release: "reloader"
  template:
    metadata:
      labels:
        app: reloader-reloader
        chart: "reloader-v1.0.2"
        release: "reloader"
        heritage: "Helm"
        app.kubernetes.io/managed-by: "Helm"
        group: com.stakater.platform
        provider: stakater
        version: v1.0.2
    spec:
      containers:
      - image: "stakater/reloader:v1.0.2"
        imagePullPolicy: IfNotPresent
        name: reloader-reloader

        ports:
        - name: http
          containerPort: 9091
        - name: metrics
          containerPort: 9090
        livenessProbe:
          httpGet:
            path: /live
            port: http
          timeoutSeconds: 5
          failureThreshold: 5
          periodSeconds: 10
          successThreshold: 1
        readinessProbe:
          httpGet:
            path: /metrics
            port: metrics
          timeoutSeconds: 5
          failureThreshold: 5
          periodSeconds: 10
          successThreshold: 1
      securityContext: 
        runAsNonRoot: true
        runAsUser: 65534
      serviceAccountName: reloader-reloader
1
2
3
4
5
6
7
sudo kubectl apply -f /home/bupt/k8s/reloader.yaml
sudo kubectl get pod,deploy,svc -n 114dev -o wide

# 更新了配置文件后，用这个更新configmap，建议扩容缩容操作不要和这个一起干，会自动生成pods来滚动更新
bupt@h9:~/k8s$ sudo kubectl create configmap queryconfig --from-file /home/bupt/k8s/mount-files-202211270941/Configs/QUERY_PHONENUMBER_CONFIGS.json -o yaml --dry-run -n 114dev | sudo kubectl replace -f -
W0127 01:20:11.550767 2883207 helpers.go:555] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/queryconfig replaced

7. 横向扩展（把h1加进来，变成1master2node）

7.1 操作

按照之前的博客先把h1的k8s环境配置好，然后

1
2
3
4
bupt@h9:~/k8s$ kubeadm token generate
fxh9yt.a2j49jxw31a9lt1x
bupt@h9:~/k8s$ sudo kubeadm token create fxh9yt.a2j49jxw31a9lt1x --print-join-command --ttl=0
kubeadm join 10.109.246.xxx:6443 --token fxh9yt.a2j49jxw31a9lt1x --discovery-token-ca-cert-hash sha256:f49123e158401fae72d6419f19942cd2b5919f0786e74de3702989c264046b17

查看：

1
2
3
4
5
bupt@h9:~/k8s$ sudo kubectl get nodes
NAME   STATUS   ROLES                  AGE     VERSION
h1     Ready    <none>                 5m22s   v1.22.15
h7     Ready    <none>                 42h     v1.22.15
h9     Ready    control-plane,master   43h     v1.22.15

把deploy中的replicas从3改成4，会发现自动在h1上创建中

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
bupt@h9:~/k8s$ sudo kubectl get pod,deploy,svc -n 114dev -o wide
NAME                               READY   STATUS              RESTARTS   AGE   IP            NODE   NOMINATED NODE   READINESS GATES
pod/114-backend-6d5c4bc967-9mtrt   2/2     Running             0          78m   10.244.1.38   h7     <none>           <none>
pod/114-backend-6d5c4bc967-fxnzb   2/2     Running             0          78m   10.244.1.37   h7     <none>           <none>
pod/114-backend-6d5c4bc967-t98mn   0/2     ContainerCreating   0          3s    <none>        h1     <none>           <none>
pod/114-backend-6d5c4bc967-vgm8h   2/2     Running             0          78m   10.244.1.39   h7     <none>           <none>
pod/114-redis-fffd7f8b-282hn       1/1     Running             0          18h   10.244.1.9    h7     <none>           <none>
pod/114-redis-fffd7f8b-59hpw       1/1     Running             0          18h   10.244.1.7    h7     <none>           <none>
pod/114-redis-fffd7f8b-6gf7d       1/1     Running             0          18h   10.244.1.8    h7     <none>           <none>

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS                   IMAGES                                                                                                                 SELECTOR
deployment.apps/114-backend   3/4     4            3           16h   114-uwsgi-django,114-nginx   registry.cn-hangzhou.aliyuncs.com/curious/114_uwsgi_django:v1,registry.cn-hangzhou.aliyuncs.com/curious/114_nginx:v1   run=114-backend
deployment.apps/114-redis     3/3     3            3           18h   114-redis                    registry.cn-hangzhou.aliyuncs.com/curious/114_redis:v1                                                                 run=114-redis

NAME                      TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE   SELECTOR
service/svc-114-backend   NodePort   10.109.179.232   <none>        8082:30860/TCP,5002:31894/TCP   16h   run=114-backend
service/svc-114-redis     NodePort   10.109.179.231   <none>        6380:31137/TCP                  16h   run=114-redis

image-20230126194131659

7.2 h1的空间不足问题，对照如下解决

https://blog.csdn.net/weixin_39865737/article/details/126717900

8. 常用命令

8.1 查看所有的状态

1
bupt@h9:~/k8s$ sudo kubectl get pod,deploy,svc -n 114dev -o wide

8.2 DEBUG查看（可以重定向）

1
sudo kubectl describe pod -n 114dev

8.3 批量删除k8s中的异常pod信息（Evicted等）

https://blog.csdn.net/Victory_Lei/article/details/126946508

  • Evicted可使用kubectl中的强制删除命令
1
2
3
4
5
6
7
8
9
# 打印指定命名空间下的所有Evicted  的 pod
kubectl get pods -n  namespace | grep Evicted | awk '{print $1}'

## 发现有很多 “Evicted” 状态的pod

## 问题原因：这是由于node节点的磁盘或内存资源不够后将现有的pod进行了驱赶。

# 批量删掉pod
kubectl get pods -n  namespace  | grep Evicted | awk '{print $1}' | xargs kubectl delete pods  -n  namespace

批量删除所有”Evicted“状态的pod即可

1
2
# 执行完成后，检查是否删除所有Evicted状态的pod
kubectl   get pods   -n  namespace
  • Terminating可使用kubectl中的强制删除命令
1
2
3
4
5
# 删除POD
kubectl delete pod PODNAME  -n  namespace --force --grace-period=0 
 
# 删除NAMESPACE
kubectl delete namespace NAMESPACENAME --force --grace-period=0
develop > k8s
#k8s
K8S快速入门-3基于智能114项目的自己实战
http://example.com/2023/01/25/develop/k8s/K8S快速入门-3基于智能114项目的自己实战/
作者
Curious;
发布于
2023年1月25日
许可协议