标题翻译
Kafka streams, custom key-specific time-window
问题
给定：
处理大量消息的Kafka Streams应用程序（最高达每秒500,000条）。
消息键是动态且高度分布的（金融数据）。
我们现在需要在时间窗口内聚合数据。
然而，时间窗口定义只能从键中派生。例如：
- 键A -> TimeWindows.of(Duration.ofMinutes(1))
- 键B -> TimeWindows.of(Duration.ofMinutes(5))
- ...
在Kafka Streams中是否有任何插件点，可以添加自定义的窗口实现？
显然，windowedBy函数仅接受Windows或SessionWindows的实现，它们对键一无所知。
英文翻译
Given:
Kafka-streams application that processes a high amount of messages (up to 500k/sec).
The message keys are dynamic and highly distributed (financial data).
We now have a need to aggregate the data within time-windows.
However time-window definition could only be derived from a key. E.g.
- Key A -> TimeWindows.of(Duration.ofMinutes(1))
- Key B -> TimeWindows.of(Duration.ofMinutes(5))
- ...
Is there any plug-in point in Kafka-streams, where one can add custom windowing implementation?
Obviously, windowedBy function accepts only implementation of Windows or SessionWindows, which know nothing about the Key.
专注分享java语言的经验与见解，让所有开发者获益！
评论