Arc Notes Weekly #81: Rise
This week, explore Netflix’s scalable data systems, DoorDash’s self-serve Kafka platform, ngrok’s open-source data pipeline, plus AI-powered drones for search and rescue.
This week, discover Netflix’s scalable solution for managing petabytes of temporal data with millisecond latency, delve into DoorDash’s Kafka self-serve platform that boosts engineering efficiency, and explore how ngrok built a data platform with open-source tools to handle high-volume pipelines. Plus, learn about innovative AI-powered drones for search and rescue, and StarbaseDB, an edge database with scale-to-zero capabilities using Cloudflare Durable Objects.
Enjoy this week’s insightful reads!
— Mahdi Yusuf (@myusuf3) or LinkedIn
👋🏾 You are reading Architecture Notes - Your Sunday newsletter, which curates best system design and architecture news from around the web. We would appreciate you sharing it with like-minded people.
Articles
Smolderingly Fast B-Trees
This article delves into the performance of B-trees compared to hashmaps, exploring benchmarks, cache behavior, and considerations around key comparison efficiency. It provides a technical analysis of when B-trees may or may not outperform hashmaps in various scenarios, particularly regarding lookup speed and cache optimization.
How we built ngrok's data platform
ngrok’s journey in building a robust data platform, focusing on their transition from AWS-centric architecture to an open-source stack. It covers technical insights on challenges with schema management, stream processing, and leveraging tools like Airbyte, Apache Flink, and dbt to handle high-volume data pipelines efficiently.DoorDash Empowers Engineers with Kafka Self-Serve
DoorDash implemented a self-serve Kafka platform to streamline resource management and reduce dependencies on infrastructure teams. By automating topic creation, configuration, and access control, the platform empowers engineers to manage Kafka resources independently, significantly improving efficiency and scalability.Netflix’s TimeSeries Data Abstraction Layer
Netflix has introduced a specialized TimeSeries Abstraction Layer to efficiently handle vast amounts of temporal event data at millisecond latency. This scalable solution, designed for high-throughput and cost-effective storage, supports Netflix’s diverse needs in Video on Demand and Gaming, complementing their existing data architecture while focusing on immutable temporal data rather than general time-series analytics.Automated Architecture Documentation for Streamlined System Design
Multiplayer auto-documents your system, from the high-level logical architecture down to the individual components, APIs, dependencies, and environments. Perfect for teams looking to streamline system design and documentation management without the manual overhead.
Optimizing Postgres Table Layout
This technical guide explores how rearranging column order in Postgres tables can reduce disk space usage and improve efficiency. It provides a detailed look into data alignment, showing how even minor adjustments in column positioning can significantly impact storage and performance.Projects
sq
Explore sq, an open-source tool for powerful data wrangling that simplifies importing, querying, and exporting data across various formats like SQL, Excel, JSON, and CSV. The platform provides a lightweight solution for tasks ranging from table diffing to database exports, all via streamlined one-liner commands.StarbaseDB
StarbaseDB provides a scale-to-zero SQLite database on the edge, built using Cloudflare Durable Objects. With features like HTTPS endpoints, WebSocket connections, and REST API support, it enables efficient data storage and retrieval for scalable, low-latency applications.
This Homemade Drone Software Finds People When Search and Rescue Teams Can’t
Innovative AI-powered drone software developed to assist in search and rescue operations. It’s designed to autonomously locate missing persons in challenging terrains, outperforming traditional methods in locating individuals faster and more efficiently.