I am Excited About a Google Product?
what is google rapid storage and why is it cool?
Google Cloud Next is going on right now and so far nothing has caught my eye… EXCEPT Rapid Storage (all the IOPS always)!
Google's recent announcement of Rapid Storage caught my attention because it doesn't just iterate on existing technology - it fundamentally reimagines how storage should work in the cloud native era.
Let me break down why Google's Colossus-powered Rapid Storage might just be the game-changer we've been waiting for.
The Storage Landscape Today
To appreciate what makes Rapid Storage special, we need to understand where today's storage options fall short:
Hard Disk Drives (HDDs)
Remember the spinny guys? Funny enough, HDDs still power massive portions of enterprise storage. Why? They're dirt cheap for capacity. But with access times measured in milliseconds (an eternity in computing), they're painfully slow for random access. Plus, when was the last time you worried about physically parking disk heads? It's 2025, folks.
Solid State Drives (SSDs)
SSDs were revolutionary when they hit the mainstream. Suddenly, random I/O didn't kill performance, and everything felt snappier. But even today's best local SSDs face bottlenecks from interfaces (SATA, SAS) and protocols that weren't designed for flash media.
Local NVMe
NVMe fixed many of SSD's protocol limitations by leveraging PCIe lanes directly. This slashed latency and boosted throughput dramatically for local storage. Direct-attached NVMe drives deliver impressive performance, but they're still islands of storage—tied to specific servers and lacking the flexibility of networked options.
Traditional SAN/NAS
Enterprise storage arrays try to give you the best of both worlds: performant storage with network flexibility. But they're complex beasts that require specialized hardware, finicky configuration, and often come with eye-watering price tags. Plus, their architectures fundamentally limit scale-out capabilities.
Object Storage (S3)
Amazon's S3 revolutionized cloud storage with its simple REST API and seemingly infinite scalability. Every cloud now offers S3-compatible storage, and it's become the default for cloud-native applications. But traditional object storage makes significant tradeoffs—higher latency and limited operations (no appends or modifications) in exchange for scalability and durability.
Enter Google's Rapid Storage and Colossus
Google's Rapid Storage takes a completely different approach by building on Colossus, their internal zonal cluster-level file system. Instead of incrementally improving existing paradigms, they've reimagined storage architecture from first principles.
Here's why this matters:
1. The Handle-Based Stateful Protocol is Changing the Game
The secret sauce in Rapid Storage is its stateful protocol using handles. When a client opens a file, it receives a handle containing complete information about where and how the file is stored. With this handle, clients can communicate directly with storage disks using an optimized RDMA-like network protocol.
This dramatically reduces latency compared to:
HDDs: We're talking sub-millisecond vs. 5-10ms seek times. That's at least 10x faster!
Traditional Object Storage: S3-style storage typically has latencies in the tens or hundreds of milliseconds due to the stateless HTTP protocol overhead.
SAN/NAS: Enterprise arrays typically add multiple milliseconds of latency through their controller software.
The handle approach creates a near-direct connection between applications and storage, removing the unnecessary hops and protocol translations that slow down conventional networked storage.
2. Performance That Puts Local Storage to Shame
Google claims Rapid Storage can support 20 million requests per second in a single bucket with sub-millisecond latency. Let that sink in.
To put that in perspective:
A high-end local NVMe drive might handle 1 million IOPS
Enterprise all-flash arrays advertise a few million IOPS (shared across all clients)
S3-compatible storage doesn't even try to compete on IOPS
And unlike local NVMe, which can only serve the server it's installed in, Rapid Storage delivers this performance across an entire zone, accessible to any compute resource.
3. Append Operations: What Object Storage Was Missing
One of the most innovative aspects of Rapid Storage is its handling of append operations. Traditional object storage (like S3) requires you to upload an entirely new object if you want to add even a single byte. It's an architectural limitation that has forced countless workarounds in cloud-native applications.
Rapid Storage's stateful protocol enables unlimited appends to objects with sub-millisecond durability. This is transformative for:
Database transaction logs
Stream processing
Time-series data
Log analytics
Event-driven applications
This capability alone bridges a massive gap between file and object storage paradigms. You get the scale and durability of object storage with the operational flexibility of a file system.
4. Failure Handling That Doesn't Lose Your Data
Rapid Storage implements a clever transactional ownership model that handles client failures gracefully. If a client crashes while appending to an object, a new client can take over ownership through a transactional process that prevents data corruption.
Compare this to:
Local storage: If your server dies, that NVMe drive becomes inaccessible
SAN/NAS: Controller failures often cause I/O pauses or worst-case data corruption
S3: Simpler model, but the stateless design limits what you can do during failures
Google's approach combines the reliability of cloud storage with sophisticated failure handling that preserves data integrity—even during partial system failures.
When to Choose What
Despite Rapid Storage's advantages, other storage technologies still have their place:
Stick with HDDs when:
Cost is your only concern
You're storing truly cold data that's rarely accessed (think plex server)
Local SSDs make sense when:
You're on a tight budget but need better-than-HDD performance
Simple workloads don't justify more sophisticated solutions
Local NVMe remains viable when:
You need the absolute lowest latency for a single server
Your workload is perfectly suited to direct-attached storage
SAN/NAS systems work best when:
You need rich file system semantics across multiple servers
Your existing workflows depend on traditional file protocols
S3 and similar object storage is ideal when:
You need extreme durability and virtually unlimited scale
Your access patterns are primarily write-once, read-many
(Future) Choose Rapid Storage when:
Your workloads demand both massive scale and low latency
You need efficient append operations without full rewrites
You're running AI/ML pipelines, especially with random access patterns
You want object storage-like scalability with file-like operations
You're tired of compromising between performance and flexibility
Look, AI is sucking up all the compute and storage. This new product may only be helpful to that industry.
Still, I'm genuinely impressed by what Google has built here. Let’s see where this goes!
Cheers,
Joe
This blog post is based on the April 10, 2025 announcement of Google Rapid Storage.