Tiering data is challenging because it often requires tradeoffs between costs and performance. The secret sauce that powers this 10x elasticity is Confluent Cloud Intelligent Storage, which uses multiple layers of cloud storage and workload heuristics to make data rebalancing and movement faster than ever before. The quicker data can move within the system, the quicker it can scale. Our dedicated offering scales to 20 GBps of throughput with the ability to expand and shrink clusters with the click of a button or the call of an API. Our Basic and Standard offerings allow instant auto-scaling up to 100 MBps of throughput and can scale down to zero with no pre-configured infrastructure you have to manage. With Confluent Cloud, scaling is 10x faster than with Apache Kafka-and doesn’t involve the capacity planning, data rebalancing, or any other typical operational burden that goes into scaling data infrastructure. Delivering 10x elasticity with Confluent Cloud On a 10 gigabit network, that would take 43 hours using vanilla Apache Kafka (assuming the full bandwidth was available for replication-in practice it would take even longer). Assuming perfect data balancing, this means the new broker will be storing 194.4 TB (777.6 TB / 4), which will need to be read from the existing brokers. Now when we scale the cluster by one broker to add some more capacity, what happens? Now there are four brokers and we need to bring the new machine up to speed so it can start accepting reads and writes. ![]() Given a replication factor of three, the cluster will retain up to 777.6 TB of storage, or about 259.2 TB per broker with equal balance. The cluster retains messages for 30 days and receives 100 MBps on average from its producers. Let’s take an example with a cluster with three brokers. To understand how Confluent Cloud is 10x more elastic than Kafka, it’s important to understand how Apache Kafka scales. But how exactly did we achieve that? Uncovering opportunities to go beyond Apache Kafka ![]() To meet the demands of these modern data pipelines, we took Apache Kafka ®’s horizontal scalability to the next level-in fact, we took it up multiple levels-and made Confluent Cloud 10x more elastic. Our customers look for a truly cloud-native data streaming platform that scales transparently across data centers around the world, while abstracting away the tedious implementation details. At Confluent, we serve thousands of customers-and they expect a lot more from their data infrastructure than ever before. Its customer-facing applications, backend systems, and business processes were all put to the test as millions of people turned to Instacart as an essential service to safely get the groceries and goods they needed.Įlasticity is the cornerstone of cloud-native computing, and it’s what allows a business like Instacart to scale quickly, add resiliency to a system, and make its products cost effective. At the onset of the COVID-19 pandemic in early 2020, Instacart experienced its highest customer demand in history.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |