Hacker News·2yThe Architecture Behind A One-Person Tech Startup
The post discusses the architecture and tools used in a one-person tech startup, including Kubernetes on AWS, automatic DNS and SSL setup, load balancing, automated rollouts and rollbacks, horizontal autoscaling, caching, app administration, scheduled jobs, loggin and monitoring, and more.
ByteByteGo·2yHow PayPal Scaled Kafka to 1.3 Trillion Daily Messages
PayPal scaled Kafka to handle an enormous volume of 1.3 trillion messages per day. They use Kafka for various use cases, such as tracking, database synchronization, and risk detection. PayPal implemented improvements in cluster management to reduce operational overhead.
Towards Data Science·2yHow I Dockerized Apache Flink, Kafka, and PostgreSQL for Real-Time Data Streaming
Achieve seamless integration of Apache Flink, Kafka, and PostgreSQL using Docker-Compose, leveraging pyFlink for real-time data processing. This guide provides practical tips, configures Flink in session mode, and demonstrates how to create custom Docker images for pyFlink to ensure Python jobs run smoothly. Additionally, the post covers setting up Kafka topics, creating Postgres tables, and handling sensor data streams. Follow the step-by-step guide to build and experiment with a streaming pipeline that efficiently processes and stores data.
Community Picks·2yKafka Migration and Event Streaming
Apache Kafka is an open-source distributed event and stream-processing platform known for its scalability and high throughput. This tutorial guides you through expanding a Kafka cluster by adding a new node and migrating topic partitions for optimal resource utilization using both manual scripts and Kafka Cruise Control. It also covers aggregating event data with ksqlDB, a database that operates on top of Kafka topics using SQL-like syntax. The tutorial includes Docker Compose configurations, command line instructions, and detailed steps for setting up and verifying the expanded Kafka cluster and ksqlDB integration.
Medium·2yStream changes from a PostgreSQL Database to a Vector Store
This post explains how to keep a vector store up to date using Change Data Capture (CDC), Python, and Redpanda. It provides a step-by-step guide to building a CDC-powered indexing pipeline that streams changes from a PostgreSQL database to a vector store. Using a prototype application, the post demonstrates how to use Docker, Quix Streams, and other tools to continuously ingest and update vectors for real-time search result optimization. Detailed instructions are provided for setting up and running the pipeline, as well as for understanding the underlying code and architecture.