How to choose a shard key for MongoDB

https://bugsnag.com/blog/mongo-shard-key

TL/DR. Shard using an index on {_id: 'hashed'} or {projectId: 1, _id: 1}.

A few months ago, we sharded our MongoDB cluster to give us two replica sets.Last week we just added another new shard. Adding the first shard took a littlework, but we did it without downtime. Adding a new shard is now trivial.

How does sharding work?

Sharding lets you split your MongoDB database accross multiple servers or, inproduction, multiple replica sets. It's important to do this early becauseMongoDB is harder to shard once you have >512Gb ofdata. Vertical scaling can only get you so far.

To do this you tell MongoDB to use one of your indexes as a shard key. It thendivides your documents into chunks with similar shard keys. These chunks arethen spread out to your replica sets, in approximate shard key order.

As you can see, everything depends on choosing the right shard key.

What makes a good shard key?

MongoDB automatically ensures that each replica set contains the same number ofchunks, 2 in the image above, 6300 or so in the Bugsnag cluster. But that ispretty much the only guarantee.

The choice of shard key determines three important things:

1. The distribution of reads and writes

The most important of these is distribution of reads and writes. If you'realways writing to one machine, then that machine will have a high write-lock-%,and so writes to your cluster will be slow. It doesn't matter how many machinesyou have in total, as all the writes will go to the same place. This is why youshould never use the monotonically increasing _id or a timestamp as the shardkey, you'll always be adding things to the last replica set.

Similarly, if all of your reads are going to the same replica set, then you'dbetter hope that your working set fits into RAM on one machine. By splittingreads evenly across all replica sets, you can scale your working set sizelinearly with number of shards. You will be utilising RAM and disks equallyacross all machines.

2. The size of your chunks

Secondarily important is the chunk size. MongoDB will split large chunks intosmaller ones if, and only if, the shard keys are different. If you have toomany documents with the same shard key you end up with jumbo chunks. Jumbochunks are bad not only because they cause the data to be unevenly distributedbut also because once they grow too large you cannot move them between shardsat all.

3. The number of shards each query hits

Finally it's nice to ensure that most queries hit as few shards as possible.The latency of a query is directly dependant on the latency of the slowestserver it hits; so the fewer you hit, the faster queries are likely to run.This isn't a hard requirement, but it's nice to strive for. Because thedistribution of chunks onto shards is only approximately in order it can neverbe enforced strictly.

Good shard key schemes

With all of that knowledge, what makes a good shard key?

Hashed id

As a first approximation you can use a hash of the _id of your documents.

db.events.createIndex({_id: 'hashed'})

This will distribute reads and writes evenly, and it will ensure that eachdocument has a different shard key so chunks can be fine-grained and small.

It's not perfect, because queries for multiple documents will have to hit allshards, but it might be good enough.

Multi-tenant compound index

If you want to beat the hashed _id scheme, you need to come up with way ofgrouping related documents close together in the index. At Bugsnag we group thedocuments by project, because of the way our app works most queries are run inthe scope of a project. You will have to figure out a grouping that works foryour app.

We can't just use projectId as a shard key because that leads to jumbochunks, so we also include the _id to break large projects into multiplechunks. These chunks are still adjacent in the index, and so still most likelyto end up on the same shard.

db.events.createIndex({projectId: 1, _id: 1})

This works particularly well for us because the number of reads and writes fora project is mostly independent of the age of that project, and old projectsusually get deleted. If that wasn't the case we might see a slight imbalancetowards higher load on more modern projects.

To avoid this problem in the future, we will likely migrate to an index on{projectId: 'hashed', _id: 1} as soon as MongoDB supports compound indexeswith hashed values.(SERVER-10220).

In summary

Choosing a shard key is hard, but there are really only two options. If youcan't find a good grouping key for your application, hash the _id. If youcan, then go with that grouping key and add the _id to avoid jumbo chunks.Remeber that whichever grouping key you use, it needs to also distribute readsand writes evenly to get the most out of each node in your cluster.