This post provides an in-depth overview of various data streaming protocols including Kafka, RabbitMQ, MQTT, and CoAP, detailing their history, implementation, and use cases. The discussion highlights the differences between push and pull mechanisms, particularly in the context of Kafka and RabbitMQ. Detailed scenarios and examples, including IoT applications, are used to illustrate the practicality and utility of these protocols. Additionally, a practical example using Docker, Apache Spark, and various other tools is provided to demonstrate a comprehensive streaming architecture.
Table of contents
The streaming bridges — A Kafka, RabbitMQ, MQTT, and CoAP example1. Introduction1.1. MQTT1.2. AMQP History1.3. RabbitMQ1.4. CoAP1.5. Kafka1.6 Goals2. Requirements3. Scenario3.1. Central Server3.2. Train Server3.3. Bridge Server4. Implementation4.1. Before continuing4.2. Setting up RabbitMQ on the central streaming server4.3. Setting RabbitMQ, MQTT, and CoAP on bridge-server4.4. Setting RabbitMQ and Kafka on the central-server5. Database ER model (PostgreSQL)6. Database model (Cassandra NoSQL)7. Data Collectors implementation7.1. Collectors Implementation8. Demo implementation9. A spy story10. ConclusionClosing quotesReferencesSort: