无法在Kubernetes上生成Kafka消息。

huangapple 未分类评论46阅读模式
英文:

Cannot produce kafka message on kubernetes

问题

我在Kafka上遇到错误:[2020-05-04 12:46:59,477] ERROR [KafkaApi-1001] 存活的代理数量 '0' 未满足偏移量主题的所需复制因子 '1'(通过 'offsets.topic.replication.factor' 配置)。如果集群正在启动并且并非所有代理都已启动,则可以忽略此错误。 (kafka.server.KafkaApis)

在尝试生成消息时出现错误:2020-05-04 12:47:45.221 WARN 1 ---

无法在Kubernetes上生成Kafka消息。
org.apache.kafka.clients.NetworkClient : [Producer clientId=producer-1] 获取关联 ID 10 的元数据时出错: {activate-user=LEADER_NOT_AVAILABLE}

在使用docker-compose时一切正常,但我也尝试将其移植到Kubernetes。我使用 kompose convert 工具开始了这个过程并修改了输出。

以下是docker-compose的片段:

zookeeper:
  container_name: zookeeper
  image: wurstmeister/zookeeper:latest
  environment:
    ZOOKEEPER_CLIENT_PORT: 2181
  ports:
    - "2181:2181"
mail-sender-kafka:
  container_name: mail-sender-kafka
  image: wurstmeister/kafka:2.12-2.2.1
  environment:
    KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    KAFKA_ADVERTISED_HOST_NAME: mail-sender-kafka
    KAFKA_CREATE_TOPICS: "activate-user:1:1"
  ports:
    - 9092:9092
  depends_on:
    - zookeeper
account-service:
  image: szastarek/food-delivery-account-service:${TAG}
  container_name: account-service
  environment:
    - KAFKA_URI=mail-sender-kafka:9092
  depends_on:
    - config-server
    - account-service-db
mail-sender:
  image: szastarek/food-delivery-mail-sender:${TAG}
  container_name: mail-sender
  environment:
    - KAFKA_URI=mail-sender-kafka:9092
  depends_on:
    - config-server

转换为Kubernetes后,我得到了 zookeeper-deploymentzookeeper-servicemail-sender-deploymentmail-sender-kafka-deploymentmail-sender-kafka-service

我还尝试添加了一些环境变量,目前看起来是这样的:

spec:
  containers:
  - env:
    - name: KAFKA_ADVERTISED_HOST_NAME
      value: mail-sender-kafka
    - name: KAFKA_ADVERTISED_PORT
      value: '9092'
    - name: ADVERTISED_LISTENERS
      value: PLAINTEXT://mail-sender-kafka:9092
    - name: KAFKA_CREATE_TOPICS
      value: activate-user:1:1
    - name: KAFKA_ZOOKEEPER_CONNECT
      value: zookeeper:2181

我找到了一个可能与问题相关的事项。当我在Docker上运行 ping mail-sender-kafka 时,我可以连接到自己。但是,如果我连接到Kubernetes的 mail-sender-kafka Pod,则无法ping自己。

在更新了主机文件后,问题得到解决。之前是:

172.18.0.24     mail-sender-kafka-xxxxxxx

我将其更改为:

172.18.0.24     mail-sender-kafka

有关如何修复此问题的任何提示?

英文:

I'm getting error on kafka: [2020-05-04 12:46:59,477] ERROR [KafkaApi-1001] Number of alive brokers '0' does not meet the required replication factor '1' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)

And error when I'm trying to produce message: 2020-05-04 12:47:45.221 WARN 1 ---

无法在Kubernetes上生成Kafka消息。
org.apache.kafka.clients.NetworkClient : [Producer clientId=producer-1] Error while fetching metadata with correlation id 10 : {activate-user=LEADER_NOT_AVAILABLE}

Using docker-compose everything works fine but I'm trying to move it also to k8s. I started that process with kompose convert tool and modify the output.

Here is a fragment of the docker-compse:

  zookeeper:
    container_name: zookeeper
    image: wurstmeister/zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
    ports:
      - "2181:2181"
  mail-sender-kafka:
    container_name: mail-sender-kafka
    image: wurstmeister/kafka:2.12-2.2.1
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_HOST_NAME: mail-sender-kafka
      KAFKA_CREATE_TOPICS: "activate-user:1:1"
    ports:
      - 9092:9092
    depends_on:
      - zookeeper
  account-service:
    image: szastarek/food-delivery-account-service:${TAG}
    container_name: account-service
    environment:
      - KAFKA_URI=mail-sender-kafka:9092
    depends_on:
      - config-server
      - account-service-db
  mail-sender:
    image: szastarek/food-delivery-mail-sender:${TAG}
    container_name: mail-sender
    environment:
      - KAFKA_URI=mail-sender-kafka:9092
    depends_on:
      - config-server

After converting it to k8s I've got zookeeper-deployment, zookeeper-service, mail-sender-deployment, mail-sender-kafka-deployment, mail-sender-kafka-service.

I've also tried to add some env variables and for now, it looks like that:

 spec:
      containers:
      - env:
        - name: KAFKA_ADVERTISED_HOST_NAME
          value: mail-sender-kafka
        - name: KAFKA_ADVERTISED_PORT
          value: '9092'
        - name: ADVERTISED_LISTENERS
          value: PLAINTEXT://mail-sender-kafka:9092
        - name: KAFKA_CREATE_TOPICS
          value: activate-user:1:1
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: zookeeper:2181

I've found one thing that probably is connected with the problem.
When I run ping mail-sender-kafka on docker I can reach myself. But if I connect to kubernetes mail-sender-kafka pod then I cannot ping myself.

After update the hosts file it works. There was something like:

172.18.0.24     mail-sender-kafka-xxxxxxx

And I changed it to

172.18.0.24     mail-sender-kafka

Any tips about how should I fix it?

huangapple
  • 本文由 发表于 2020年5月4日 21:06:05
  • 转载请务必保留本文链接:https://java.coder-hub.com/61592899.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定