Community Picks·1ycloudlena/s3manager: A Web GUI for your S3 buckets
s3manager is a Web GUI written in Go for managing S3 buckets from any provider. It allows users to list buckets, create new ones, upload and download objects, and delete objects. The application is configurable via environment variables and can be run locally, with Docker, or deployed to a Kubernetes cluster. It supports SSL, various signature types, IAM roles, and server-side encryption.
Materialized View·1yS3 Is the New SFTP
Fintech companies handle diverse data processing tasks, including shuffling files between vendors and partners, often using SFTP. Transitioning to modern data lakehouses using S3, Apache Iceberg, and Apache Parquet can centralize and streamline this process. This new method allows ease of access and management while maintaining advantages such as fast transfers and central access control. Although challenges like schema evolution remain, adopting data lakehouses can benefit companies seeking efficient and scalable data solutions. The trend is supported by rising customer demand and the involvement of startups providing innovative data export platforms.
Lobsters·1yBuilding a distributed log using S3 (under 150 lines of Go)
The post describes how to implement a durable, distributed, and highly available log using AWS S3 in less than 150 lines of Go. Key highlights include the structure of the log interface, the implementation of the Append and Read operations, handling concurrent writes with S3 conditional writes, and failover/crash recovery mechanisms. The open-source project includes code and tests, with several open issues for further improvements.