Distributed Key-Value Store

Overview

This project implements a distributed key-value store that can tolerate node failures and network partitions. The store uses the Raft consensus algorithm to ensure that data is consistently replicated across multiple nodes. The implementation includes leader election, log replication, and snapshot mechanisms. The system can handle network splits gracefully and continues to serve requests as long as a majority of nodes remain connected.

Architecture

The system uses a client-server architecture with multiple replicated servers. Each server maintains a state machine (the KV store), a replicated log of commands, and Raft-specific state (current term, voted-for, etc.). Servers communicate via TCP using Protobuf for message serialization. The Tokio async runtime handles concurrent connections and enables efficient network I/O. Persistence is achieved through log files stored on disk, with periodic snapshots to avoid unbounded log growth.

Key Challenges

• •Ensuring atomicity of state transitions under concurrent requests
• •Handling split-brain scenarios and network partitions correctly
• •Optimizing Raft to provide reasonable latency for client operations
• •Managing memory efficiently with streaming log replay on startup

What I Learned

• •Deep knowledge of the Raft consensus algorithm and its practical implementation details
• •Understanding of distributed systems trade-offs between consistency, availability, and partition tolerance
• •Rust ownership system and how it provides safety guarantees for concurrent code
• •Performance analysis and optimization techniques for distributed systems

Future Improvements

• •Add range query support (prefix scan)
• •Implement more sophisticated log compaction strategies
• •Add automatic cluster membership changes
• •Create benchmarking suite against other KV stores