Repository navigation
Introduce Stream API #23
Description
Activity
@bsideup Thanks for your comment.
The
Stream APIhere is non-reactive, not as same as Reactive Streams.The reactive-streams provides
Publisher, Subscriber, Subscription, ProcessorAPIs. I think we might consider providing reactive APIs in an individual module. These APIs can be used as a high level abstraction to process messages in the message queues.@bsideup
TheStreamAPI here, like Spark's DStream, Flink's DataStream or Kafka's KStream, is just stream abstraction including basic stream operations(filter,flatMap,map,reduce,groupBy,join,count,max,min,window, etc.). This is the different point from reactive-streams's common reactive APIs(Publisher, Subscriber, Subscription, Processor). reactive-streams provides these operations depending on 'RxJava, Akka Streams, Reactor or any other reactive framework'. And another point is that it's easy and convenient to translate sql toStreamrelated operations when we consider supporting sql(e.g. flink-sql, kafka-ksql).@vesense how do you think about if we want to support the Stream API base on the existing atomic APIs :-)
@vongosling Sorry for the delay because of my busy work. I will write a google doc for the Stream API design after completing the RocketMQ-Beam integration. Thanks for your patience.
Currently, there are
Producer API,Consumer APIfor basic IO operations on message queues.For many scenes, users like to process messages in Streams.
The
Stream APIis lightweight, not as same as the distributed streaming systems like Apache Storm, Flink, Spark-Streaming. And its implementation can be embeded in any java applications.It provides the most common stream operations like
filter,flatMap,map,reduce,groupBy,join,count,max,min,window, etc.It's different from the existing
StreamingConsumer, which is a low level Consumer API for processing consumer positions, etc. (IMHO, the class name ofStreamingConsumershould be renamed to a more reasonable one, the current might be confused for users.)