Kubernetes容器化 - Kafka

除最新的2.8不需要ZooKeeper,这之前的版本都需要用到ZooKeeper,可以参考前一篇Kubernetes容器化 - Zookeeper安装ZooKeeper。ZooKeeper与Kafka版本之间貌似有一些对应关系,未找到原出处。在容器中安装Kafka首先要看有没有合适的镜像,主要看到的有2个镜像:

都不是官方镜像,两个都做过尝试,最终选择的是第二个镜像。

第一步,创建pv,这里和前面ZooKeeper的操作一样,找个合适的存储创建pv即可。

第二步,创建Service

apiVersion: v1
kind: Service
metadata:
  name: kafka
  namespace: default
  labels:
    project: kafka
spec:
  selector:
    project: kafka
  ports:
  - name: kafka
    port: 9092
    targetPort: 9092
    protocol: TCP
  clusterIP: None

第三步,创建StatefulSet,这里测试的版本不新,但这个镜像最近有更新,理论上和其他版本安装差不多,镜像使用说明:

1、通过StatefulSet来部署集群,通过BROKER_ID_COMMAND环境变量可以实现broker.id的动态设置

2、类似KAFKA_LOG_RETENTION_HOURS和KAFKA_DELETE_TOPIC_ENABLE,配置文件中的变量可以以类似方式注入到配置文件

3、目前集群内部访问正常,外部访问Broker失败,应该是Pod内部IP无法直接与外部通信。可以排查KAFKA_ADVERTISED_LISTENERS的设置。

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
  namespace: default
  labels:
    project: kafka
spec:
  serviceName: kafka
  replicas: 3
  selector:
    matchLabels:
      project: kafka
  template:
    metadata:
      labels:
        project: kafka
    spec:
      restartPolicy: Always
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 80
            podAffinityTerm:
              topologyKey: kubernetes.io/hostname
              labelSelector:
                matchExpressions:
                - key: project
                  operator: In
                  values: 
                  - kafka
      containers:
      - name: kafka
        image: "wurstmeister/kafka:2.11-0.11.0.3"
        imagePullPolicy: "IfNotPresent"
        ports:
        - name: kafka
          containerPort: 9092
        env:
        - name: BROKER_ID_COMMAND
          value: "hostname | awk -F '-' '{print $NF}'"
        - name: KAFKA_PORT
          value: "9092"
        - name: KAFKA_LISTENERS
          value: "PLAINTEXT://:9092"
        - name: KAFKA_ADVERTISED_PORT
          value: "9092"
        - name: KAFKA_ADVERTISED_LISTENERS
          value: "PLAINTEXT://:9092"
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: "zookeeper.default.svc.cluster.local:2181"
        - name: KAFKA_LOG_RETENTION_HOURS
          value: "168"
        - name: KAFKA_DELETE_TOPIC_ENABLE
          value: "false"
        - name: KAFKA_DEFAULT_REPLICATION_FACTOR
          value: "2"
        volumeMounts:
        - name: kafka-pvc
          mountPath: /kafka
  volumeClaimTemplates:
  - metadata:
      name: kafka-pvc
    spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 50Gi
        selector:
          matchLabels:
            project: kafka-pv


-- EOF --
最后更新于: 2021-09-15 08:27
发表于: 2021-06-10 19:46
标签: Kafka Kubernetes 容器化