What database platform better at scaling?

Question

I am planning a for a database that i expect will be very big in terms of number of records rather than tables and batabases. My Question is what product (i.e, SQL 2008, Cassandra, Azure) handles scaling better?

I expect that there will be 100k or so records will be added everyday, It should easily retrieve a single row and a batch of them.

The app will crawl the web and for each word in a dictionary will find the same word used in different sentences and record them to a database. The developers are mainly .net oriented but they are fluent in C/C++ as well and because this is a academic project anything is basically affordable.

One platform/product per answer please.

While I think SQLite and MSAccess will not work for you, databases like MySQL, PostgreSQL, Oracle *** and DB2 are proven to do the thing you want. Be more specific. What features do you need? Who will use the 100k records per day? Which systems do you need to connect? What programming languages do your developers speak? Can you do PL/SQL? What budget you have.. — Sebastian Roth, Jan 05 '11 at 14:37
Depending on what you're doing, I've also seen people break things up by year, and then use a UNION if they're gong to be searching across year boundriess. This is particularly useful for databases like mySQL where you don't have as fine control of how to spread your data out across spindles, or where you have (had?) limitations on table size (either number of records or total size on disk) — Joe, Jan 05 '11 at 15:16
Also, 100k records per day with 4-6 fields can be much different from 100k records with 100+ fields each. — Joe, Jan 05 '11 at 15:17
@Sevki "Has anyone tried Denali CTP" is a different question. Please try to phrase it as such on a new question, but make it a "correct" question please. http://dba.stackexchange.com/questions/how-to-ask — jcolebrand, Jan 06 '11 at 02:40

score 2 · Answer 1 · answered Jan 05 '11 at 21:45

Any of the big RDBMS systems or NoSQL systems can handle that sort of data load. Back in 2002 I had a database table with 1B records in it that got more than 100k records a day, and that was running SQL Server 2000. The systems today are much faster (both the hardware and the software) and should be able to handle that load going in with no problem. The big problem becomes getting the data out, and having the data indexed properly to find it based on how you want to search. And having enough memory to load the data into cache to find it faster.

score 0 · Answer 2 · answered Jan 05 '11 at 16:29

0

I don't know which is the best, but I was running some tests on 3 tables, using Guid, BigInt and Int as primary keys.

Each table have 1,000,000 entries. To retrieve a whole table it takes 12 seconds on my machine. To retrieve 80,000 it takes about a second.

SQL Server is surely very reliable.

There is this page on microsoft.com that has some stuff about performance and scalability, maybe you can find something else if you need.

answered Jan 05 '11 at 16:29

BrunoLM

3,473
7
27
22

A table with a guid as a primary key will become very fragmented, very quickly as data is put in, unless the guids are sequential. – mrdenny Jan 05 '11 at 21:40
@mrdenny So wouldn't you use a heap table and have a index to account for that? – jcolebrand Jan 06 '11 at 02:41

score 0 · Answer 3 · answered Jan 06 '11 at 18:02

0

You could consider GridSQL, which can be placed on top of PostgreSQL to enable intraquery parallelization. It is GPL.

answered Jan 06 '11 at 18:02

David

1,185
1
12
27

What database platform better at scaling?

3 Answers3