为何使用Curator和Exhibitor而不用Zookeeper

Apache ZooKeeper is an invaluable component for service-oriented architectures. It effectively solves the difficult problem of distributed coordination (i.e. locking, leader selection, etc.). However, it can be tricky to use correctly. Curator and Exhibitor were written to make using and operating ZooKeeper easier and less error prone.

Apache ZooKeeper is a client/server system for distributed coordination. On the client side, you use the client library (from Java, C/C++, etc.) to connect to the server. The client library exposes APIs that resemble a simple file system. You create/read/update/delete ZNodes via the API. The ZooKeeper documentation describes high level recipes that can be built using this API. On the server side, you maintain a cluster of ZooKeeper servers (called an ensemble). For most uses the ensemble will have 3 or 5 instances. Someone new to ZooKeeper will quickly discover a steep learning curve on both the client and server side. Users of the ZooKeeper client will have to:

• Manually deal with connection issues.
• Handle “recoverable” errors as described in the ZooKeeper documentation.
• Implement any needed “recipes”.
• Learn and understand numerous undocumented ZooKeeper edge cases and best practices.
• On the server side:
• ZooKeeper does not come bundled with any cluster wide tools.
• ZooKeeper does not support any kind of backup/restore.
• Once a ZooKeeper ensemble is running, there is no automatic way to change its configuration.
• Each ZooKeeper instance must be configured with the host/address of all the other instances in the ensemble.

Curator, A Better ZooKeeper Client

Curator is a Java library that enhances the built-in ZooKeeper client. Curator’s main benefits are:
• A simplified, more natural API.
• Automatic ZooKeeper connection management with retries.
• Complete, well-tested implementations of ZooKeeper recipes.
• A framework that makes writing new ZooKeeper recipes much easier.

Connection Management
New users of ZooKeeper are surprised to learn that a significant amount of connection management must be done manually. For example, when the ZooKeeper client connects to the ensemble it must negotiate a new session, etc. This takes some time. If you use a ZooKeeper client API before the connection process has completed, ZooKeeper will throw an exception. These types of exceptions are referred to as “recoverable” errors.

Curator automatically handles connection management for you, greatly simplifying client code. Instead of directly using the ZooKeeper APIs you use Curator APIs that internally check for connection completion and wrap each ZooKeeper API in a retry loop. Curator uses a retry mechanism to handle recoverable errors and automatically retry operations. The method of retry is customizable. Curator comes bundled with several implementations (ExponentialBackoffRetry, etc.) or you can write your own.

Recipes

The ZooKeeper documentation describes many possible uses for ZooKeeper calling each a “recipe”. While the distribution comes bundled with a few implementations of these recipes, most ZooKeeper users will need to manually implement one or more of the recipes.

Implementing a ZooKeeper recipe is not trivial. Besides the connection handling issues mentioned earlier, there are numerous edge cases that are not well documented that must be considered. For example, many recipes require that an ephemeral-sequential node be created. New users of ZooKeeper will not know that there is an edge case in ephemeral-sequential node creation that requires you to put a special “marker” in the node’s name so that you can search for the created node if an I/O failure occurs. This is but one of many edge cases that are not well documented. Curator provides close to two dozen (and growing) well-tested recipe implementations that incorporate workarounds/handling for all known edge cases. Most ZooKeeper users really want the recipes and don’t need to be bogged down with the internal workings of ZooKeeper. Curator provides recipes for: Leader Elections, Locks (mutex, semaphore, read/write, etc.), Barriers, Caches, Service Discovery, etc. Curator’s recipe implementations are straightforward to use. As an example, Listing 1 shows how to use a lock.

Listing 1. Curator Lock

CuratorFramework client = …

InterProcessMutex lock = new InterProcessMutex(client, lockPath);

lock.acquire();

try {

… do your work here …

} finally {

lock.release();

}

Curator comes bundled with an implementation of Service Discovery. In SOA/distributed systems, services need to find each other. i.e. a web service might need to find a caching service, etc. DNS can be used for this but it is nowhere near flexible enough for services that are constantly changing. A Service Discovery system provides a mechanism for:

• Services to register their availability.
• Locating a single instance of a particular service.
• Notifying when the instances of a service change.

Using Curator’s Service Discovery implementation you can quickly build a service oriented system whereby instances register themselves as they come up and other instances in the system can query for services of a desired type.

Building Your Own Recipes

Using Curator’s APIs you can quickly build recipes of your own. Curator wraps all of the low-level ZooKeeper APIs. As mentioned above, Curator’s versions automatically handle connection issues, retries, etc. Listing 2 shows an example of creating a node using Curator APIs.

Listing 2. Create a ZooKeeper Node

CuratorFramework client = …

client.create().forPath(path, data);

Testing
Writing unit tests for ZooKeeper-based applications can be tricky as it involves a client-server system. Curator comes bundled with a TestingServer and TestingCluster that are in-memory temporary ZooKeeper instances that can be used in unit tests. Curator’s own unit tests use these classes extensively.

Exhibitor, Easier ZooKeeper Ops

Operating a ZooKeeper ensemble presents some tricky problems:

• ZooKeeper is statically configured.
• Almost no tools are provided to manage the ensemble.
• Backup/restore is sometimes needed.
• A visualization tool is vital.
• Adding/removing instances to the ensemble is cumbersome and error prone.

Exhibitor is a ZooKeeper co-process for instance monitoring, backup/recovery, cleanup and visualization. Exhibitor runs as a separate application on each instance running ZooKeeper (see Figure 1). Exhibitor will continually monitor the ZooKeeper instance restarting it, etc. as needed.

Figure 1. Exhibitor as a separate application

Shared Configuration

Exhibitor presents a console (see Figure 2) for configuring all aspects of the ZooKeeper ensemble. Importantly, the configuration values set in Exhibitor are shared amongst all instances in the ensemble. So, you only need to configure your ensemble once. When new instances come online or existing instances need to be restarted they will use the shared configuration values.

Figure 2. Exhibitor Console

Instance Monitoring

Each Exhibitor instance monitors the ZooKeeper server running on the same server. If ZooKeeper is not running, Exhibitor will write the zoo.cfg file, etc. and start it. If ZooKeeper crashes for some reason, Exhibitor will restart it.

Rolling Ensemble Changes

Exhibitor can update the servers in the ensemble in a rolling fashion so that the ZooKeeper ensemble can stay up and in quorum while the changes are being made.

Visualizer

Exhibitor provides a graphical tree view of the ZooKeeper ZNode hierarchy (see Figure 3). From the visualizer you can also create/delete/modify nodes.

Figure 3. Exhibitor Visualizer

Integration With Curator
Exhibitor integrates with Curator for ensemble changes. There is no way default way for ZooKeeper clients to get notified when instances in the ensemble change. Curator and Exhibitor solve this by maintaining communication so that if instances come and go in the ensemble Curator clients are notified and stay in sync.

Automatic Instance Management
When an Exhibitor instance comes online it will automatically add itself to the ensemble. Note this is done in a rolling fashion to minimize disruption to the ensemble. This feature makes bringing up a ZooKeeper ensemble almost trivial. Once the shared configuration is set, instances can come and go without any human intervention.

Backup/Restore
Backups in a ZooKeeper ensemble are more complicated than for a traditional data store (e.g. an RDBMS). Generally, most of the data in ZooKeeper is ephemeral. It would be harmful to blindly restore an entire ZooKeeper data set. What is needed is selective restoration to prevent accidental damage to a subset of the data set. Exhibitor enables this. Exhibitor will periodically backup the ZooKeeper transaction files. Once backed up, you can index any of these transaction files. Once indexed, you can search for individual transactions and “replay” them to restore a given ZNode to ZooKeeper.

ZooKeeper at Netflix
ZooKeeper via Curator and Exhibitor is being used extensively at Netflix. Some of the uses are:

• InterProcessMutex used for ensuring unique values in various sequence ID generators.
• Cassandra Backups.
• TrackID Service.
• Our Chukwa collector uses the LeaderSelector for various housekeeping tasks.
• We make use of some third party services that allow only a limited number of concurrent users. The InterprocessSemaphore is used to manage this.
• Various Caches.
• Much more.

Accessing Curator and Exhibitor

The binaries are posted to Maven Central making accessing them very easy. Both projects are hosted at Github: https://github.com/Netflix/curator and https://github.com/Netflix/exhibitor.

Jordan Zimmerman @randgalt Jordan Zimmerman is a Senior Software Engineer on the Platform Team at Netflix. He has contributed to open source software throughout his career. At Netflix he’s opened Curator, Exhibitor and Governator and maintains the Netflix Open Source hub at http://netflix.github.com. In his spare time he plays drums (bebop and big band), listens to opera (mostly Puccini), drinks fine whisky (Laphroaig and Buffalo Creek) and watches soccer (ManU) and Epic Meal Time.