User:What-a-good-jungle/sandbox

Apache RocketMQ
Developer(s)Apache Software Foundation
Initial release2012; 12 years ago (2012)
Stable release
4.2.0 / April 27, 2018; 6 years ago (2018-04-27)
Written inJava
Operating systemCross-platform
TypeStream processing, Message broker
LicenseApache License 2.0
Websiterocketmq.apache.org

RocketMQ[1] is a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity and flexible scalability. It is the third generation distributed messaging middleware open sourced by Alibaba in 2012. Through several years of technical improvement, RocketMQ is now capable of transferring trillions of concurrent online messages as of Alibaba's Nov. 11th Shopping Festival. On November 21, 2016, Alibaba donated RocketMQ to the Apache Software Foundation. Next year, on February 20, the Apache Software Foundation announced Apache RocketMQ as a Top-Level Project. Nowadays, RocketMQ is widely used by thousands of companies, and research and educational institutions around the world.

History

edit

RocketMQ's evolution can be divided into three stages.

The first generation adopts the push mode in which data are stored in relational database. Low latency message delivery and distributed transactions meet Alibaba's demands and contribute to RocketMQ's wide adoption.

The second generation adopts the pull mode in which data storage is managed by file systems. Its performance on log collection is comparable to Kafka. Taking account of Alibaba's application scenarios, especially the requirement for high reliability during transaction processes, the second-generation RocketMQ does not blindly seek the throughput but also stresses stability and reliability. The long pulling mechanism makes the response time of the system no slower than that in the first generation.

The third generation, RocketMQ mainly uses the Pull mode along with some Push operations. Largely optimized, it has low latency and high performance, especially in high concurrency and massive amounts of data scenarios. Since 2012, RocketMQ has challenged the task of supporting Alibaba's annual Nov. 11th Global Shopping Festival multiple times. During the 2016 Festival, RocketMQ robustly provided stable infrastructure with a transfer throughput of more than one trillion messages.

Features

edit

Much comparison have been made between all kinds of messaging solutions, there is a table demonstrates the comparison between RocketMQ, ActiveMQ and Kafka (Apache’s most popular messaging solutions according to awesome-java[2]) up to December 17, 2016:[3]

Messaging Product Client SDK Protocol and Specification Ordered Message Scheduled Message Batched Message BroadCast Message Message Filter Server Triggered Redelivery Message Storage Message Retroactive Message Priority High Availability and Failover Message Track Configuration
ActiveMQ Java, .NET, C++ etc. Push model, support OpenWire, STOMP, AMQP, MQTT, JMS Exclusive Consumer or Exclusive Queues can ensure ordering Supported Not Supported Supported Supported Not Supported Supports very fast persistence using JDBC along with a high performance journal,such as levelDB, kahaDB Supported Supported Supported, depending on storage,if using kahadb it requires a ZooKeeper server Not Supported The default configuration is low level, user need to optimize the configuration parameters
Kafka Java, Scala etc. Pull model, support TCP Ensure ordering of messages within a partition Not Supported Supported, with async producer Not Supported Supported, you can use Kafka Streams to filter messages Not Supported High performance file storage Supported offset indicate Not Supported Supported, requires a ZooKeeper server Not Supported Kafka uses key-value pairs format for configuration. These values can be supplied either from a file or programmatically.
RocketMQ Java, C++, Go Pull model, support TCP, JMS, OpenMessaging Ensure strict ordering of messages,and can scale out gracefully Supported Supported, with sync mode to avoid message loss Supported Supported, property filter expressions based on SQL92 Supported High performance and low latency file storage Supported timestamp and offset two indicates Not Supported Supported, Master-Slave model, without another kit Supported Work out of box,user only need to pay attention to a few configurations

Architecture

edit
 

RocketMQ consists of four parts: name servers, brokers, producers and consumers. Each of them can be horizontally extended without a single point of Failure. As shown in image left.[4]

NameServer Cluster

edit

Name Servers provide lightweight service discovery and routing. Each Name Server records full routing information, provides corresponding reading and writing service, and supports fast storage expansion.

Broker Cluster

edit

Brokers take care of message storage by providing lightweight TOPIC and QUEUE mechanisms. They support the Push and Pull model, contains fault tolerance mechanism (2 copies or 3 copies), and provides strong padding of peaks and capacity of accumulating hundreds of billion messages in their original time order. In addition, Brokers provide disaster recovery, rich metrics statistics, and alert mechanisms, all of which are lacking in traditional messaging systems.

Producer Cluster

edit

Producers support distributed deployment. Distributed Producers send messages to the Broker cluster through multiple load balancing modes. The sending processes support fast failure and have low latency.

Consumer Cluster

edit

Consumers support distributed deployment in the Push and Pull model as well. It also supports cluster consumption and message broadcasting. It provides real-time message subscription mechanism and can meet most consumer requirements. RocketMQ’s website provides a simple quick-start guide to interested users.

Applications

edit

There are at least five aspects Apache RocketMQ could relate to:[5]

  • The integration of heterogeneous systems;
  • Decoupling between applications;
  • Backbone in the event-driven mechanism and complex event architecture model;
  • Data replication channels;
  • Integration with flow calculation engine.

Open Source and Community Maintenance

edit

The RocketMQ team have done much to active the community. and the team has adopted a strategy similar to Apache's other top-level projects as the team devotes itself to the high quality of its products. A series of standardized software development processes, such as milestone discussions, features design, code testing, peer reviews, integration tests, release discussions, and release announcements have been conducted in the RocketMQ version planning. On the other hand, the team interacts with the open source community through activities like line Meet-ups, Workshops, ApacheCon and Code Marathon, to attract new contributors and committers.[6]

The team also produces a commercial distribution called Aliware MQ. It supports the TCP, HTTP and MQTT protocols, enhances the maintenance tools of ecological integration (including visualization of the trajectory, resource reporting statistics monitoring and alerts, as well as Kafka integration). Designed for enterprise, it has multi-region deployment with high availability disaster recovery characteristics on the public cloud and private cloud.

Rewards

edit

2016 China’s most popular open source software award

2017 China’s most popular open source software award

12th CJK(China-Japan-South Korea) open source software outstanding technology award

Enterprises that use RocketMQ

edit

Those use RocketMQ:

See also

edit

References

edit
  1. ^ "apache/rocketmq". GitHub. Retrieved 2018-05-25.
  2. ^ "akullpp/awesome-java". GitHub. Retrieved 2018-05-25.
  3. ^ "Motivation". Apache RocketMQ. 2018-03-05. Retrieved 2018-06-14.
  4. ^ "RocketMQ Architecture". Apache RocketMQ. 2018-03-05. Retrieved 2018-05-25.
  5. ^ "Apache RocketMQ背后的设计思路与最佳实践". 阿里中间件团队博客 (in Simplified Chinese). Retrieved 2018-05-25.
  6. ^ "From Alibaba to Apache: RocketMQ's Past, Present, and Future". InfoQ. Retrieved 2018-05-29.
edit
Apache RocketMQ
Developer(s)Apache Software Foundation
Initial release2012; 12 years ago (2012)
Stable release
4.2.0 / April 27, 2018; 6 years ago (2018-04-27)
Written inJava
Operating systemCross-platform
TypeStream processing, Message broker
LicenseApache License 2.0
Websiterocketmq.apache.org

RocketMQ[1] is a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity and flexible scalability. It is the third generation distributed messaging middleware open sourced by Alibaba in 2012. On November 21, 2016, Alibaba donated RocketMQ to the Apache Software Foundation. Next year, on February 20, the Apache Software Foundation announced Apache RocketMQ as a Top-Level Project. Nowadays, RocketMQ is widely used by thousands of companies, and research and educational institutions around the world.

History

edit

The development of RocketMQ can be divided into three stages.[2]

The first generation uses the push mode in data transportation, and relational database in data storage. It shows low latency in message delivery and meets the command of a typical E-commerce platform[3] with distributed transactions. Then many industry began to adopt it.

The second generation uses the pull mode in data transportation, and file system in data storage. It paid more attention to stability and reliability, and shows a comparable performance to the first generation in response time and Kafka on log collection.

The third generation combines the Pull mode with some Push operations. It inherits the advantages of the first and second generation, and shows high performance in concurrency and massive amounts of data scenarios. It's the key for the success of 2016 Alibaba's Nov. 11th Shopping Festival.

Features

edit

Much comparison have been made between all kinds of messaging solutions, especially between Kafka and RocketMQ,[4] there is a table demonstrates the comparison between RocketMQ, ActiveMQ and Kafka(the three most popular messaging solutions[5]) up to December 17, 2016:[6]

Messaging Product Client SDK Protocol and Specification Ordered Message Scheduled Message Batched Message BroadCast Message Message Filter Server Triggered Redelivery Message Storage Message Retroactive Message Priority High Availability and Failover Message Track Configuration
ActiveMQ Java, .NET, C++ etc. Push model, support OpenWire, STOMP, AMQP, MQTT, JMS Exclusive Consumer or Exclusive Queues can ensure ordering Supported Not Supported Supported Supported Not Supported Supports very fast persistence using JDBC along with a high performance journal,such as levelDB, kahaDB Supported Supported Supported, depending on storage,if using kahadb it requires a ZooKeeper server Not Supported The default configuration is low level, user need to optimize the configuration parameters
Kafka Java, Scala etc. Pull model, support TCP Ensure ordering of messages within a partition Not Supported Supported, with async producer Not Supported Supported, you can use Kafka Streams to filter messages Not Supported High performance file storage Supported offset indicate Not Supported Supported, requires a ZooKeeper server Not Supported Kafka uses key-value pairs format for configuration. These values can be supplied either from a file or programmatically.
RocketMQ Java, C++, Go Pull model, support TCP, JMS, OpenMessaging Ensure strict ordering of messages,and can scale out gracefully Supported Supported, with sync mode to avoid message loss Supported Supported, property filter expressions based on SQL92 Supported High performance and low latency file storage Supported timestamp and offset two indicates Not Supported Supported, Master-Slave model, without another kit Supported Work out of box,user only need to pay attention to a few configurations

Because the characteristics of high performance, high reliability and high real time ability, more and more efforts have been make to the combination of RocketMQ and other protocol component in all kinds of messaging scenarios.[7]

Architecture

edit
 

RocketMQ consists of four parts: name servers, brokers, producers and consumers. Each of them can be horizontally extended without a single point of Failure.[8] As shown in image left.

NameServer Cluster

edit

The lightweight component for service discovery and they can be used to read and write routing information. Each one records global information, and supports fast storage expansion.

Broker Cluster

edit

They use lightweight TOPIC and QUEUE mechanisms to manage data storage. To realize fault tolerance, 2 copies or 3 copies of data are provided. And Client can get message in Push and Pull model. In addition, disaster recovery and rich metrics statistics are also supported.

Producer Cluster

edit

Producers can be distributed deployed, and messages from producers to brokers can be balanced through multi-path. In addition, fast failure and low latency are supported.

Consumer Cluster

edit

Consumers can also be distributed deployed in the push and pull model,[9] and they can subscribe message real-time, consume message in the unit of cluster. Message broadcasting[10] is also supported.

Applications

edit

There are at least five aspects Apache RocketMQ could relate to:[11]

  • The integration of heterogeneous systems;
  • Decoupling between applications;
  • Backbone in the event-driven mechanism and complex event architecture model;
  • Data replication channels;
  • Integration with flow calculation engine.

Community Maintenance

edit

The RocketMQ team have done much to active the community. Meetups, Workshops, ApacheCon and Code Marathon are regular held in BeiJing, ShenZhen, HangZhou to to attract new contributors and committers. As for version management, a series of standardized software development processes are adopted. The latest version is 4.2.0, and 4.3.0 is on the way. More information can be reached in here.

Papers

edit

Rewards

edit

Enterprises that use RocketMQ

edit

Those use RocketMQ:

See also

edit

References

edit
  1. ^ "apache/rocketmq". GitHub. Retrieved 2018-05-25.
  2. ^ "From Alibaba to Apache: RocketMQ's Past, Present, and Future". InfoQ. Retrieved 2018-06-26.
  3. ^ "Toward a General Distributed Messaging Framework for Online Transaction Processing Applications - IEEE Journals & Magazine". ieeexplore.ieee.org. Retrieved 2018-06-26.
  4. ^ "RocketMQ与Kafka对比(18项差异) - BYRans - 博客园". www.cnblogs.com (in Chinese (China)). Retrieved 2018-06-26.
  5. ^ "akullpp/awesome-java". GitHub. Retrieved 2018-06-26.
  6. ^ "Motivation". Apache RocketMQ. 2018-03-05. Retrieved 2018-06-14.
  7. ^ "A MQTT Protocol Message Push Server Based on RocketMQ - IEEE Conference Publication". ieeexplore.ieee.org. Retrieved 2018-06-26.
  8. ^ Lam, Jaskey. "RocketMQ——水平扩展及负载均衡详解 - 薛定谔的风口猪". jaskey.github.io. Retrieved 2018-06-26.
  9. ^ "RocketMQ 消息订阅Subscribe—— Push & Pull 模式 - xxggy的个人页面 - 开源中国". my.oschina.net. Retrieved 2018-06-26.
  10. ^ "RocketMQ(五)——消费模式 - CSDN博客". blog.csdn.net (in Chinese (China)). Retrieved 2018-06-26.
  11. ^ "Apache RocketMQ背后的设计思路与最佳实践". 阿里中间件团队博客 (in Simplified Chinese). Retrieved 2018-05-25.
edit
Apache RocketMQ
Developer(s)Apache Software Foundation
Initial release2012; 12 years ago (2012)
Stable release
4.2.0 / April 27, 2018; 6 years ago (2018-04-27)
Written inJava
Operating systemCross-platform
TypeStream processing, Message broker
LicenseApache License 2.0
Websiterocketmq.apache.org

RocketMQ[1] is a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity and flexible scalability. It is the third generation distributed messaging middleware open sourced by Alibaba in 2012. On November 21, 2016, Alibaba donated RocketMQ to the Apache Software Foundation. Next year, on February 20, the Apache Software Foundation announced Apache RocketMQ as a Top-Level Project.

History

edit

The development of RocketMQ can be divided into three stages.[2]

The first generation uses the push mode in data transportation, and relational database in data storage. It shows low latency in message delivery and meets the command of a typical E-commerce platform[3] with distributed transactions.

The second generation uses the pull mode in data transportation, and file system in data storage. It paid more attention to stability and reliability, and shows a comparable performance to the first generation in response time and Kafka on log collection.

The third generation combines the Pull mode with some Push operations. It inherits the advantages of the first and second generation, and shows high performance in concurrency and massive amounts of data scenarios.

Features

edit

Much comparison have been made between all kinds of messaging solutions, and it is widely known that when the number of topics increases dramatically , the throughput of RocketMQ dropped much less than Kafka.[4]

Because the characteristics of high performance, high reliability and high real time ability, more and more efforts have been make to the combination of RocketMQ and other protocol component in all kinds of messaging scenarios such as MQTT.[5] It is

Client SDK Protocol and Specification Ordered Message Scheduled Message Batched Message BroadCast Message Message Filter Server Triggered Redelivery Message Storage Message Retroactive Message Priority High Availability and Failover Message Track Configuration
Java, C++, Go Pull model, support TCP, JMS, OpenMessaging Ensure strict ordering of messages,and can scale out gracefully Supported Supported, with sync mode to avoid message loss Supported Supported, property filter expressions based on SQL92 Supported High performance and low latency file storage Supported timestamp and offset two indicates Not Supported Supported, Master-Slave model, without another kit Supported Work out of box,user only need to pay attention to a few configurations

Architecture

edit
 

RocketMQ consists of four parts: name servers, brokers, producers and consumers. Each of them can be horizontally extended without a single point of Failure. As shown in image left.

NameServer Cluster

edit

The lightweight component for service discovery and they can be used to read and write routing information. Each one records global information, and supports fast storage expansion.

Broker Cluster

edit

They use lightweight TOPIC and QUEUE mechanisms to manage data storage. To realize fault tolerance, 2 copies or 3 copies of data are provided. And Client can get message in Push and Pull model. In addition, disaster recovery and rich metrics statistics are also supported.

Producer Cluster

edit

Producers can be distributed deployed, and messages from producers to brokers can be balanced through multi-path. In addition, fast failure and low latency are supported.

Consumer Cluster

edit

Consumers can also be distributed deployed in the push and pull model, and they can subscribe message real-time, consume message in the unit of cluster. Message broadcasting is also supported.

Applications

edit

There are at least five aspects Apache RocketMQ could relate to:

  • The integration of heterogeneous systems;
  • Decoupling between applications;
  • Backbone in the event-driven mechanism and complex event architecture model;
  • Data replication channels;
  • Integration with flow calculation engine.

Community Maintenance

edit

The RocketMQ team have done much to active the community. Meetups, Workshops, ApacheCon and Code Marathon are regular held in BeiJing, ShenZhen, HangZhou to to attract new contributors and committers. The OpenMessaging benchmarking suites are currently available for the RocketMQ and it makes RocketMQ keep the pace with the global standard for distributed messaging.[6] As for version management, a series of standardized software development processes are adopted. The latest version is 4.2.0, and 4.3.0 is on the way. More information can be reached in here.

Awards

edit

See also

edit

References

edit
  1. ^ "apache/rocketmq". GitHub. Retrieved 2018-05-25.
  2. ^ "From Alibaba to Apache: RocketMQ's Past, Present, and Future". InfoQ. Retrieved 2018-06-26.
  3. ^ "Toward a General Distributed Messaging Framework for Online Transaction Processing Applications - IEEE Journals & Magazine". ieeexplore.ieee.org. Retrieved 2018-06-26.
  4. ^ Cloud, Alibaba (2018-01-04). "Kafka vs. RocketMQ- Multiple Topic Stress Test Results". Medium. Retrieved 2018-07-08.
  5. ^ "A MQTT Protocol Message Push Server Based on RocketMQ - IEEE Conference Publication". ieeexplore.ieee.org. Retrieved 2018-06-26.
  6. ^ "The OpenMessaging Benchmark Framework". openmessaging.cloud. Retrieved 2018-07-08.
edit