Vsync A library to help you build distributed systems.

The Vsync project at Cornell is a long-term effort to create a new option for cloud computing that can enable reliable, secure replication of data even in the highly elastic first-tier of the cloud. Today’s cloud platforms tend to adhere to an approach called CAP in which one relaxes consistency in order to get better availability and fault-tolerance. With Vsync we’ve focused on creating new options: strong forms of consistency (e.g. updates are applied to all replicas, and in the same order), but in ways that work around the limitations CAP highlights.

Vsync is our most recent platform in a long history dating back more than 25 years. Cornell’s work in this area started with an earlier system, called the Isis Toolkit, with many of the same ideas that Vsync embodies. The Isis Toolkit was ultimately commercialized and became the basis for reliability in settings that included the New York and Swiss Stock Exchanges (Isis ran the NYSE for more than a decade without a single disruptive outage), the French Air Traffic Control system, the US Navy AEGIS warship and many others. Spin-off technologies are still to be found today in products from a wide variety of vendors in the distributed systems and cloud management spaces.

The Isis Toolkit inspired all sorts of follow-on systems both here at Cornell and elsewhere. Cornell’s systems included Horus, Ensemble, JGroups (later commercialized by JBoss) and Quicksilver. Each of these specialized in different aspects of the broader reliability problem: security, raw speed, elegance of structure, leveraging formal methods to achieve strong correctness proofs, embedding into Java. A collaboration with the Transis project at Hebrew University helped us understand how to adapt these techniques to maximize availability during network partitioning failures. You’ll find many, many papers on these systems here: http://www.cs.cornell.edu/projects/quicksilver/pubs.html.

Vsync draws on ideas from all of these ancestors, and was originally called Isis2. The system looks like a library and one uses it from programs written in languages like C# (our favorite), IronPython, C\\ and so forth. In fact we can work with any language available in the .NET framework (about 40 in total) and over time we expect to offer a similarly wide range of options for Linux users. Today, our Linux support is via a cross-compiler called Mono, and we can support any Linux application that can link with a Mono library. In theory this is already quite a long list of options.

Vsync has built-in data replication, key-value storage and security solutions, among others.

To enable these kinds of powerful functionalities all you need to do as the user is to configure your group appropriately. For example, if an application replicates data over some set of nodes, a single call to g.SetSecure() will activate the group security framework, encrypting all group-related communication so that nosy onlookers will learn… nothing.

(Do I) Need Paxos? Virtual Synchrony?

Vsync uses a novel new “dynamic reliability” model that integrates the two best known replication technologies into a single easily-used solution. So, if you’ve read about Leslie Lamport’s Paxos protocol, which implements State Machine Replication, you’ve come to the right place: the Vsync SafeSend primitive implements exactly that technology! If you want a faster but slightly less fault-tolerant option, consider the virtually synchronous Send primitive, or OrderedSend. Vsync is designed to let you justify your application design against even a tough critic who demands formal correctness models and proofs, even as you build solutions that can scale out while pegging the network at 100% loads and keeping even hundreds of thousands of server processes busy. If you own a data center you’ll want it working hard for you. With Vsync we can put those resources to work, and guarantee high assurance too.

Designed to be easy for object oriented programmers to master.

The programmer works in an object oriented style, but by associating application objects with Vsync groups, is able to achieve data replication, security, automated checkpointing and recovery, fault-tolerant parallel computing and a wide range of other behaviors. Our programming manual, at 100-pages and counting, illustrates the programming styles you’ll need to use. A collection of demo programs is planned for 2012 and will be downloadable examples that you can copy to your machine and customize to create new applications.

Newcomers to distributed computing might consider ordering a copy of Ken’s new textbook. While the textbook is much broader in coverage than just focusing on Vsync it does discuss the technology underlying the system and includes a number of chapters on designing and implementing complex distributed application using these kinds of tools.

Fast, scalable, elastic: a platform designed for the cloud

The entire goal of our project is to create a new option for consistent, secure cloud-scale computing. So this naturally raises basic questions: How fast is Vsync How big can a deployment be?

These are good questions, but in fact we just don’t know the answers yet. One reason is that Vsync is a moving target. We’re evolving the system almost day by day. Another is that the system is new and still a bit unstable, as one might expect for a new platform. Don’t use it if you need a professional product with professional Q/A and a 24x7 support telephone line. And don’t count on Vsync scaling to a half-million nodes anytime in the immediate future. But don’t bet that we won’t get there eventually, either!

In smaller configurations with IP multicast enabled, we’re running at about 1000 application-initiated multicasts per second (small messages), but the data rates sustainable by the system vary widely as we change the size of an application or the loads to which we subject it. We’re aiming for the fastest data rates possible, but of course we also want stability, flow control, seamless handling of failures and all of those can fight against super-high data rates. Performance will be less good if IP multicast isn’t enabled, forcing us to tunnel over TCP, or if strong security is used, or if membership of an application changes rapidly while the application is running.

In terms of deployment scale, for the initial release we’re recommending that applications use smaller numbers of replicas. For example, things seem very solid with low levels of replication (e.g. applications with 3-5 members in a group).

Right now our heaviest testing has been in the 25-500 member range, and we think Vsync is stable for most uses at that scale.

Early in 2013 we ran experiments with 1000+ members, and Vsync was quite stable. Still, by this size you must work with master/worker startups and the batched MultiJoin API. We're starting experiments now, in early summer 2013, at a scale of 2000-4000 members and are hoping to be at 10,000 by the end of 2013.

Why is it free? How can I get support when I find a way to crash it?

We believe that when public funding is used to develop a platform, the public should reap the benefits! So we’ve made a decision to release Vsync under a free BSD license. Our group at Cornell is providing support, and we’re setting up a user discussion forum where you can post questions, suggested improvements or even bug reports. Ken Birman is the main author of the code, and will be doing much of the support personally. We plan to do our best to make the experience a positive one for everyone. But do keep one thing in mind: you didn’t pay anything for this system, and it isn’t a product with a multibillion dollar company behind it. So do cut us a little slack in terms of your support expectations.

Last edited Nov 19, 2015 at 9:01 PM by birman, version 2