Best of Metadata — December 2024

1. 1
   Article

   Metadata·1y

   Stream Processing

   Batch processes can delay business operations, so stream processing is used to handle events immediately as they occur. Stream processing involves systems notifying consumers of new events, often through message brokers like RabbitMQ or log-based brokers like Kafka. Dual writes can lead to errors and inconsistencies, so Change Data Capture (CDC) allows for consistent data replication across systems. Event sourcing records all changes immutably, aiding in auditability, recovery, and analytics. Stream processing can be used in various applications, including fraud detection, trading systems, and manufacturing, and relies on techniques like microbatching and checkpointing for fault tolerance.

   24
2. 2
   Article

   Metadata·1y

   Use of Time in Distributed Databases (part 1)

   Distributed systems require coordination among nodes for event ordering and state coherence despite having no shared state or common clock. Timestamping using logical and vector clocks is essential but comes with drawbacks. Synchronized clocks, especially with the advances in technology, provide a more reliable and precise means for coordination, significantly reducing uncertainty. The integration of physical clocks with logical time in Hybrid Logical Clocks (HLC) and the development of tightly synchronized clocks enhance performance in distributed databases. Looking forward, the increasing adoption of synchronized clocks in systems promises greater efficiency and accuracy.

   18