英文:
How to Query kafka for latest offset / partition /timestamp for latest message for given key in a topic
问题
你好！以下是你要翻译的内容：
如何查询 Kafka 并获取给定主题和消息键的最新消息的时间戳、偏移量和分区？
我们的应用程序使用的是 Golang 和 Java。但是任何解决方案都可以。
谢谢！
英文:
How can I query kafka and get the timestamp/offset/partition for the latest, most recent message given the topic and message key?
Our apps are in golang and java. However any solution is welcome.
Thanks in advance.
答案1
得分: 0
消费者无法对主题进行过滤或仅接收某些消息。消费者将按顺序获取消息。
最好的方法可能是应用一些过滤器来确定特定的分区。但在该分区内，您将需要出队以获取坐标。
如果您正在寻找特定的键，您将需要出队消息，然后在客户端应用过滤器。对于ConsumerRecord，您可以获取偏移量、分区等信息。
一个标准的方法是让消费者将键和消息的坐标写入表中，这也是您可能要考虑的事情。
英文:
A Consumer cannot filter or only receive some messages from topics. The consumer will fetch messages in order.
At best we may be able to apply some filter to determine a specific partition. But within that partition you will have to dequeue to get the co-ordinates
If you are looking for a specific key , you will have to dequeue the message and then apply a filter on the customer side. For the ConsumerRecord you can then get the offset , partition etc.
A standard approach is to have the consumer write to the table for the key and the co-ordinates of the message , again something you may want to think about.
答案2
得分: 0
除非您在Kafka Streams / KsqlDB中构建KTable并使用交互式查询，否则无法通过键查询Kafka。
否则，您可以计算主题键的分区，并使用Kafka二进制文件中包含的GetOffsetShell Java CLI工具获取该分区的最新偏移量，或者在将消费者分配给该分区后，通过代码在那里进行消费者的查找。
如果您需要频繁地从Kafka进行键查找，最好将数据消费到实际的键值存储或数据库中。
英文:
You can't query Kafka by a key unless you build a KTable and use Interactive Queries in Kafka Streams / KsqlDB.
Otherwise, you can compute a partition for a key of a topic, and get the latest offset for that partition using GetOffsetShell java CLI tool included with Kafka binaries, or seek a consumer there via code after assigning the consumer to that partition.
If you need frequent key lookups from kafka, it would be best to consume the data into an actual key-value store or database
专注分享java语言的经验与见解，让所有开发者获益！
评论