Kubernetes基础 ( 9 ) - 示例

一、概述

前面章节中都会有一些Demo,但不够整体,这里从运维角度看看该如何配置日常服务,后面想找各个资源对象的Yaml文件示例写法,看这里应该就够了。

k8s环境: Mac下Docker Desktop启用Kubernetes

二、PHP + Nginx

2.1 环境说明

PHPPythonGo这几种语言中,PHP的部署算是最麻烦的了,他需要依赖NginxPHPNginx之间还需要文件共享,静态页面由Nginx处理,PHP页面交给php-fpm解析,所以要配置PHP+Nginx需要先理一理:

PHPNginx的交互方式,大概有两种方式可供选择:

这里选择在同一个Pod中PHPNginx容器都需要能够读取到源代码文件,同一个Pod中挂载的目录各个容器都可以读到,我们可以直接挂个空目录,应用镜像只打代码文件,然后在PodinitController容器里将代码都拷贝到容器去。

另外,常规项目配置上的要求:

如果配置文件打到镜像中,则修改后需要重新构建镜像,如果通过ConfigMap管理配置文件,则需要将配置在运行时挂载到容器中。这里选择通过ConfigMap来控制配置文件

还有就是日志文件的问题,我们先通过hostPath的方式实现挂载Nginx日志。通过ingress实现7层代理。数据库这个场景我们先暂时不配置,可以使用本机的mysql

上面就是配置PHP环境的需求,接下来看看怎么配置:

2.2 配置镜像

我们会使用到3个镜像,分别是PHP镜像、Nginx镜像以及代码镜像。

我们模拟一个简单的项目,包含以下文件:

$ ls
Dockerfile api.php    config.php index.php

代码文件:index.php

<?php phpinfo();

代码文件:api.php,引用了配置文件。

<?php
include "config.php";
echo json_encode($config);

代码文件:config.php

<?php
$config = [
    "host" => "127.0.0.1",
    "env" => "uat"
];

为简单起见,只设置了这么3个文件,config.php配置文件需要通过ConfigMap注入。接下来写Dockerfile

FROM busybox:1.32.0

WORKDIR /src
COPY . /src

接下来在目录中创建镜像:

$ docker build -t pengbotao/project-php:v1 .

这样子一个简单的镜像就创建好了,代码镜像里只有纯代码,无法直接运行应用。没有设置.dockerignore,配置文件config.php也写入到镜像中了,后面我们会用线上配置文件覆盖掉,也可以打包的时候就忽略掉。

2.3 创建ConfigMap

通过正式的配置文件创建config.php

$ kubectl create configmap phpdemo-config --from-file=config.php
configmap/phpdemo-config created

$ kubectl describe cm phpdemo-config
Name:         phpdemo-config
Namespace:    default
Labels:       <none>
Annotations:  <none>

Data
====
config.php:
----
<?php
$config = [
    "host" => "0.0.0.0",
    "env" => "prod"
];

Events:  <none>

创建Nginx配置文件,我们也可以用同样的方法创建Nginx.confPHPphp.iniWeb服务的配置文件。

$ kubectl create configmap phpdemo-nginx --from-file=phpdemo.local.com.conf
configmap/phpdemo-nginx created

$ kubectl describe cm phpdemo-nginx
Name:         phpdemo-nginx
Namespace:    default
Labels:       <none>
Annotations:  <none>

Data
====
phpdemo.local.com.conf:
----
server {
    listen       80;
    listen  [::]:80;
    server_name  phpdemo.local.com;
    index index.html index.php;
    root /data/www;
    charset utf-8;

    location ~ \.php$ {
        fastcgi_pass   127.0.0.1:9000;
        fastcgi_index  index.php;
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
        include        fastcgi_params;
    }
}
Events:  <none>

同一个Pod内,所以php可以直接设置为本地9000端口。域名暂定为:phpdemo.local.com

2.4 创建 Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: phpdemo
  labels:
    project: phpdemo
    env: prod
spec:
  replicas: 2
  selector:
    matchLabels:
      project: phpdemo
      env: prod
  template:
    metadata:
      labels:
        project: phpdemo
        env: prod
    spec:
      initContainers:
      - name: init-phpdemo-src
        image: pengbotao/project-php:v2
        imagePullPolicy: IfNotPresent
        command: ['sh', '-c', "cp -rf /src/* /src-www && cp /src-config/* /src-www/"]
        volumeMounts:
        - name: wwwroot
          mountPath: /src-www
        - name: phpdemo-config
          mountPath: /src-config
      containers:
      - name: php
        image: pengbotao/php:7.4.8-fpm-alpine
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            memory: "64Mi"
            cpu: "250m"
          requests:
            memory: "64Mi"
            cpu: "250m"
        volumeMounts:
        - name: wwwroot
          mountPath: /data/www
        ports:
        - containerPort: 9000
      - name: nginx
        image: nginx:1.19.2-alpine
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
          requests:
            memory: "128Mi"
            cpu: "500m"
        volumeMounts:
        - name: wwwroot
          mountPath: /data/www
        - name: phpdemo-nginx
          mountPath: /etc/nginx/conf.d
        - name: nginx-log-path
          mountPath: /var/log/nginx
        ports:
        - containerPort: 80
      volumes:
      - name: wwwroot
        emptyDir: {}
      - name: phpdemo-config
        configMap:
          name: phpdemo-config
      - name: phpdemo-nginx
        configMap:
          name: phpdemo-nginx
      - name: nginx-log-path
        hostPath: 
          path: /Users/peng/k8s/logs

说明:

执行之后我们可以进容器看看代码文件是否正常,如果执行正常容器里应该可以看到源代码和线上的config.php

2.5 创建 Service

apiVersion: v1
kind: Service
metadata:
  name: phpdemo-svc
  labels:
    project: phpdemo
    env: prod
spec:
  selector:
    project: phpdemo
    env: prod
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
  clusterIP: None

创建成功后kubectl describe svc phpdemo-svc应该可以看到Endpoints已经关联上了Pod

2.6 创建Ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: phpdemo.local.com
spec:
  rules:
  - host: phpdemo.local.com
    http:
      paths:
      - path: /
        backend:
          serviceName: phpdemo-nginx-svc
          servicePort: 80

通过下面命令可以看到,当前ingress暴露的是宿主机80端口,但80已经使用了,把ingress-nginx绑定的端口调整为30080

$ kubectl get svc -n ingress-nginx
NAME                                 TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.109.107.221   localhost     80:31526/TCP,443:30328/TCP   7d9h

$ kubectl edit svc ingress-nginx-controller -n ingress-nginx

  ports:
  - name: http
    nodePort: 31526
    port: 30080
    protocol: TCP
    targetPort: http
    
    
$ kubectl get svc -n ingress-nginx
NAME                                 TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
ingress-nginx-controller             LoadBalancer   10.109.107.221   localhost     30080:31526/TCP,443:30328/TCP   7d9h

然后在宿主机hosts绑定127.0.0.1 phpdemo.local.com后访问 http://phpdemo.local.com:30080/api.php 就可以看到输出了,输出的是我们线上配置的config.php

{
    "host": "0.0.0.0",
    "env": "prod"
}

本地挂载的日志目录也可以看到Nginx日志,到这里配置就基本完成了,接下来就是跟后期日常维护相关的操作。

2.7 版本更新

调整index.php文件内容模拟更新版本

<?php print_r($_SERVER);

重新打镜像

$ docker build -t pengbotao/project-php:v2 .

更新镜像和回滚只需要指定镜像版本即可。

kubectl set image deployment phpdemo init-phpdemo-src=pengbotao/project-php:v2

如果前面我们通过不同的Pod来组合PHP环境,NginxPHP里都有代码文件,镜像更新则需要执行2个Pod的更新:

$ kubectl set image deployment phpdemo init-phpdemo-src=pengbotao/project-php:v2
$ kubectl set image deployment phpdemo-nginx init-phpdemo-src=pengbotao/project-php:v2

2.8 增加/修改配置文件

前面我们是直接通过kubectl create configmap命令来创建,如果要增加文件则相对麻烦,我们可以调整为通过Yaml文件来创建,

apiVersion: v1
kind: ConfigMap
metadata:
  name: phpdemo-config
  namespace: default
data:
  config.php: |
    <?php
    $config = [
        "host" => "0.0.0.0",
        "env" => "prod"
    ];
  database.php: |
    <?php
    $database = [
        "host" => "127.0.0.1",
        "port" => 3306,
    ];

这样子就增加了database.phpKey,更新Deployment后就会看到源代码目录增加了database.php文件。

2.9 重启服务

比如像上面场景更新了配置文件想重启Pod,或者某些情况下尝试重启Pod。如果Deployment没变更的话,重新kubectl apply不会触发滚动更新。手动删除Pod会重建,但一个个去删除也太累了。我们可以这么操作:

$ kubectl rollout restart deploy phpdemo

kubectl rollout包含以下功能:

$ kubectl rollout -h
Manage the rollout of a resource.

 Valid resource types include:

  *  deployments
  *  daemonsets
  *  statefulsets

Examples:
  # Rollback to the previous deployment
  kubectl rollout undo deployment/abc

  # Check the rollout status of a daemonset
  kubectl rollout status daemonset/foo

Available Commands:
  history     显示 rollout 历史
  pause       标记提供的 resource 为中止状态
  restart     Restart a resource
  resume      继续一个停止的 resource
  status      显示 rollout 的状态
  undo        撤销上一次的 rollout

可以通过undo做回滚操作,比如回退到前一版本:

# 设置为v1版本
$ kubectl set image deployment phpdemo init-phpdemo-src=pengbotao/project-php:v1
# 升级为v2版本
$ kubectl set image deployment phpdemo init-phpdemo-src=pengbotao/project-php:v2
# 回滚到前一版本,即v1版本
$ kubectl rollout undo deploy phpdemo

也可以指定回滚的版本:kubectl rollout undo deploy phpdemo --to-revision=1,可以通过查看rollout查看历史记录:

$ kubectl rollout history deploy phpdemo
deployment.apps/phpdemo
REVISION  CHANGE-CAUSE
1         <none>
3         <none>
4         <none>
5         <none>
8         <none>
11        <none>
12        <none>

但这个记录前面看过,基本看不太出差别,所以感觉直接更新镜像版本或者回退到上一版本会更实用些(也有可能是没找到CHANGE-CAUSE列的用法)。

2.10 小结

这个环境里实现了:

Pod层级还有就绪检测、存活检测可以做一做,接下来在Python的环境中加上这两项看看。

三、Python + Nginx

3.1 环境说明

这里主要还是出于演示目的,尽量体现出每个Demo的差异化,Python环境这边想这么做:

3.2 配置镜像

一个简单的Gunicorn+Flask应用,Github地址:https://github.com/pengbotao/k8s-py-demo。也可以通过docker pull pengbotao/k8s-py-demo:v1拉取。源代码已经打到Python镜像中了,默认启动是5000端口,拿到镜像就可以运行起来。接下来来部署这个镜像:

3.3 创建ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: flask-config
  namespace: default
data:
  gunicorn.py: |
    import gevent.monkey

    gevent.monkey.patch_all()
    debug = True
    loglevel = 'debug'
    bind = '0.0.0.0:80'

    workers = 1
    threads = 2
    worker_class = 'gunicorn.workers.ggevent.GeventWorker'
    daemon = False

    pidfile = './logs/gunicorn.pid'
    logfile = './logs/debug.log'
    accesslog = './logs/gunicorn_access.log'
    errorlog = './logs/gunicorn_error.log'

    x_forwarded_for_header = 'X-FORWARDED-FOR'

---

apiVersion: v1
kind: ConfigMap
metadata:
  name: flask-nginx
  namespace: default
data:
  pydemo.local.com.conf: |
    server {
        listen       80;
        listen  [::]:80;
        server_name  pydemo.local.com;

      location / {
            proxy_pass http://flask-svc;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    }

容器里默认使用的5000端口,这里为做演示将容器端口调整为80,配置站点。Nginxproxy_pass关联的是flask-svc

3.4 创建PersistentVolume

apiVersion: v1
kind: PersistentVolume
metadata:
  name: flask-pv001
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  storageClassName: flask-pv
  hostPath:
    path: /Users/peng/k8s/pv-data/pv001
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: flask-pv002
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  storageClassName: flask-pv
  hostPath:
    path: /Users/peng/k8s/pv-data/pv002

创建了2个PV,用来存储项目产生的日志文件。

3.5 创建StatefulSet

apiVersion: v1
kind: Service
metadata:
  name: flask-svc
spec:
  selector:
    app: flask-pod
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
  clusterIP: None


---

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: flask-sts
spec:
  replicas: 2
  selector:
    matchLabels:
      app: flask-pod
  serviceName: flask-svc
  template:
    metadata:
      labels:
        app: flask-pod
    spec:
      containers:
      - name: flask
        image: pengbotao/k8s-py-demo:v1
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
        volumeMounts:
        - name: flask-pvc
          mountPath: /data/www/logs
        - name: flask-config
          mountPath: /data/www/gunicorn.py
          subPath: gunicorn.py
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 10
          periodSeconds: 5
          timeoutSeconds: 3
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 60
          periodSeconds: 10
      volumes:
      - name: flask-config
        configMap:
          name: flask-config
          items:
          - key: gunicorn.py
            path: gunicorn.py
  volumeClaimTemplates:
  - metadata:
      name: flask-pvc
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: flask-pv
      resources:
        requests:
          storage: 1Gi

配置Pod,增加就绪检测、存活检测。同时将配置文件覆盖已经存在的gunicorn.py

3.6 创建Nginx

Nginx并不是必须的,Ingress可以直接关联上面的SVCflask-svc,这配之后倒产生了一个新的问题,后面再说。

apiVersion: v1
kind: Service
metadata:
  name: flask-nginx-svc
spec:
  selector:
    app: flask-nginx-pod
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
  clusterIP: None

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: flask-nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: flask-nginx-pod
  template:
    metadata:
      labels:
        app: flask-nginx-pod
    spec:
      containers:
      - name: nginx
        image: nginx:1.19.2-alpine
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
        volumeMounts:
        - name: flask-nginx
          mountPath: /etc/nginx/conf.d
      volumes:
      - name: flask-nginx
        configMap:
          name: flask-nginx

3.7 创建Ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: pydemo.local.com
spec:
  rules:
  - host: pydemo.local.com
    http:
      paths:
      - path: /
        backend:
          serviceName: flask-svc
          servicePort: 80

创建Ingress,配置Host后就可以访问到了:

$ curl http://pydemo.local.com:30080/
{
    "ClientIP": "192.168.65.3, 10.1.2.196",
    "Host": "flask-sts-1",
    "ServerIP": "10.1.3.88",
    "Time": "2020-09-30 02:31:01",
    "Version": "v1"
}

这里的访问流程:Ingress -> Nginx Depolyment -> Flask StatefulSet -> Pod。正常访问是没有问题,但当sts做更新的时候会存在问题,更新流程:

但如果不创建Nginx Depolyment,直接用Ingress关联flask-svc,则不存在这个问题,重启Nginx可以解决。问题产生的原因是Nginxproxy_pass为域名时,会做DNS缓存,因为容器的IP变化了,通过旧的IP就会访问失败了。

第一种解决方法

修改ConfigMap中的nginx增加resolver配置,指定DNS(通过kubectl get svc -n kube-system可以看到IP

    server {
        listen       80;
        listen  [::]:80;
        server_name  pydemo.local.com;
        resolver 10.96.0.10 valid=3s;
        resolver_timeout 5s;
        set $upstream flask-svc.default.svc.cluster.local;
        location / {
            proxy_pass http://$upstream;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    }

注:这里配置的地址是headless service的地址,用svc的名称没走通

第二种解决方法

由于StatefulSet需要用到Headless ServiceServiceIP变化感知很快,可以在建立一个Service设置相同的Label/SelectorNginx里在设置VIP的地址即可。

两种方法感觉都一般,这个应用可以直接走Ingress -> Flask StatefulSet,从而去掉Nginx

3.8 小结

这个环境更多的是为了演示,区分DeployStatefulSet,了解下STS的使用。


-- EOF --
发表于: 2020-11-01 17:15
标签: Kubernetes 容器化