12

It looks like database sharding is great if I have huge collections. What if I have lots of fairly sized collections ? Let's say that for 1 collection of 100 000 000 documents (not very big comments) sharding is effective. Is it also effective for 10 000 collections with 10 000 documents each ?

(I think this question is still valid for table oriented databases if you replace collections with tables and documents with rows. If possible I'd like to know the theoretical answer as well as the answer in the specific MongoDB scenario, if different from the theoretical answer.)

João Pinto Jerónimo
  • 293
  • 2
  • 6

2 Answers2

5

Is it also effective for 10 000 collections with 10 000 documents each ?

Most people have the "single large collection" problem and so the sharding is clearly useful for reducing headaches of balancing this data.

However, when you have 10 000 small collections, your headache is probably not "balancing the data". With this many small collections your problem is likely about tracking these collections. Depending on your document size, you may not even break the lower limit for sharding to actually happen.

For the really small collections, you can use the little-known movePrimary command to manage the location of your data.

Of course, the other way to look at this is why do you have 10k collections? A collection doesn't need homogeneous objects and with 10k collections most of them have to be generated. It's quite possible to store different "types" of data in the same collection, reduce the number of collections and then include the type as part of the shard key.

Gates VP
  • 184
  • 3
  • Thanks, I was exactly trying to know if the best I could do is get rid of these tons of collections and make a big one. I had tons of collections before because I heard a common belief: "Huge collections are bad for you because the indexes don't fit into RAM and it will be very slowly to query and update them". But I guess sharding was created to solve that problem... Thanks !! – João Pinto Jerónimo Sep 06 '11 at 09:49
  • Honestly, I find that you can often "cheat" on the indexes too. If you have two collections foo and bar with the same data structure, you can merge them into the baz collection and override the _ids (in code): { _id: "foo123" }, { _id: "bar123" }. You have a larger index, but you have only one index that includes the type. Not a requirement, just "food for thought". – Gates VP Sep 06 '11 at 18:35
4

MongoDB sharding works by splitting up a collection into smaller 'chunks' and distibuting them evenly across a number of machines. The default chunk size, which is generally the most efficient, is 200MB. So unless a collection grows much larger than 200MB it won't split into chunks, and therefore won't be eligible for sharding, so there'll be no benefits.

In the general case, sharding data over multiple machines is a very effective way of scaling reads, writes and queries. You get the benefits of multiple CPUs, hard disks, and memory storage, working in parallel to read, write and process data. Scaling out memory is particularly important for MongoDB, where high performance is very sensitive to data fitting in memory.

Chris Fulstow
  • 141
  • 3