这是一个创建于 2120 天前的主题，其中的信息可能已经有所发展或是发生改变。
// 问题:
Golang 通过 sarama 连接到 kafka,没有指定 group,连接的 topic 只有一个 partition,通过 2 台机器上一样的程序连上 kafka 上, 这 2 个程序都能消费这一个 partition 的数据.
那么这个 2 台机器上的应用程序是属于同一个 group 的,而且是同一个 consumer? 要知道一个 partition 只能被每个 group 的一个 consumer 消费的!!!
// kafka 版本:
root@ubuntu18:/usr/local/kafka_2.11-2.0.0/bin# ./kafka-topics.sh --version 2.0.0 (Commit:3402a8361b734732)
// topic 只有一个 partition
root@ubuntu18:/usr/local/kafka_2.11-2.0.0/bin# kafka-topics.sh --describe --zookeeper 127.0.0.1:2181 --topic topic.XXX Topic:topic.skygram PartitionCount:1 ReplicationFactor:1 Configs: Topic: topic.XXX Partition: 0 Leader: 0 Replicas: 0 Isr: 0
// 只有一个 group,看不到其他的 group 和 consumer 的消息
root@ubuntu18:/usr/local/kafka_2.11-2.0.0/bin# ./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
KMOffsetCache-ubuntu18
// 确实是两台机器连上的
root@ubuntu18:/usr/local/kafka_2.11-2.0.0/bin# netstat -antop|grep 9092|grep ESTABLISHED tcp 0 0 192.168.198.145:50396 192.168.198.145:9092 ESTABLISHED 18330/./main_linux keepalive (1.00/0/0) tcp 0 0 192.168.198.145:50394 192.168.198.145:9092 ESTABLISHED 18330/./main_linux keepalive (4.07/0/0) tcp 0 0 192.168.198.145:50392 192.168.198.145:9092 ESTABLISHED 18330/./main_linux keepalive (4.07/0/0) tcp6 0 0 192.168.198.145:9092 192.168.198.145:50394 ESTABLISHED 4367/java keepalive (3630.28/0/0) tcp6 0 0 192.168.198.145:9092 192.168.198.1:54181 ESTABLISHED 4367/java keepalive (1688.40/0/0) tcp6 0 0 192.168.198.145:9092 192.168.198.145:43006 ESTABLISHED 4367/java keepalive (4265.90/0/0) tcp6 0 0 192.168.198.145:9092 192.168.198.1:54183 ESTABLISHED 4367/java keepalive (1688.40/0/0) tcp6 0 0 192.168.198.145:43012 192.168.198.145:9092 ESTABLISHED 4699/java keepalive (4266.62/0/0) tcp6 0 0 192.168.198.145:9092 192.168.198.1:54182 ESTABLISHED 4367/java keepalive (1688.40/0/0) tcp6 0 0 192.168.198.145:9092 192.168.198.145:50396 ESTABLISHED 4367/java keepalive (3742.75/0/0) tcp6 0 0 192.168.198.145:43006 192.168.198.145:9092 ESTABLISHED 4699/java keepalive (4265.90/0/0) tcp6 0 0 192.168.198.145:9092 192.168.198.145:50392 ESTABLISHED 4367/java keepalive (3630.28/0/0) tcp6 0 0 192.168.198.145:9092 192.168.198.145:43012 ESTABLISHED 4367/java keepalive (4266.62/0/0
6 条回复 2020-01-16 14:54:44 +08:00
 | 1 znood 2020-01-16 12:52:35 +08:00 没有指定 group，那 group 可能是随机的，去 zookeeper 里可以看到有哪些 comsumer group |
 | 2 hijoker OP @znood zk 里面是空的,很是令我困惑 [zk: localhost:2181(CONNECTED) 2] ls / [cluster, controller_epoch, controller, brokers, zookeeper, kafka-manager, admin, isr_change_notification, consumers, log_dir_event_notification, latest_producer_id_block, config] [zk: localhost:2181(CONNECTED) 3] ls /consumers [] |
| 3 hijoker OP 我也偏向认为这是 2 个 group,但是怎么从 kafka 的后台来证明这是 2 个 group 呢 还有,好像新版本的 kafka 的 consumer 的信息已经不在 zk 里了 是用这个命令来看 ./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list |
| 4 znood 2020-01-16 13:51:22 +08:00 那 group 有可能是空的，KMOffsetCache-ubuntu18 可能是命令行测试生成的 |
| 6 znood 2020-01-16 14:54:44 +08:00 不管 KMOffsetCache-ubuntu18 怎么来的，我的意思是你的两个程序可能都使用的空 groupID，我搜了下，groupID 只是一个 label 标识，默认为空，也就是可以为空 Consumers label themselves with a consumer group name, and each record published to a topic is delivered to one consumer instance within each subscribing consumer group. Consumer instances can be in separate processes or on separate machines. https://stackoverflow.com/questions/43118671/kafka-consumer-default-group-id https://cwiki.apache.org/confluence/display/KAFKA/KIP-289%3A+Improve+the+default+group+id+behavior+in+KafkaConsumer |
Select Language