Warp wrote:
>   From the wikipedia page I get the impression that Oracle and IBM DB2 are
> approximately the only two RDBMS systems which scale up well to the hundreds
> of terabytes of data.

Cool.

>   I wonder how eg. MySQL or PostgreSQL would handle such vast amounts of
> data. Would they handle them ok, just maybe a bit slower, would they be
> significantly less efficient, or would they simply fail?

"""InnoDB has a limit of 1023 concurrent transactions that have created undo 
records by modifying data."""
I doubt you're going to get 70,000 transactions a second on a MySql 
installation, altho I don't see anything about individual table sizes listed 
in terms of number of rows.

The latest MySql seems to have a storage engine that talks to DB2 databases, 
so I'm not sure if there's any significant difference there.

However, given that looking something up on an index on a 6-million-row 
table took over a second on a 64-bit machine, I wouldn't want to put 
billions of rows, let alone trillions, into a MySQL table.

> Or is it a question
> of them not having such extensive support for the hardware required to
> handle that amount of data efficiently?

I would imagine that since they're portable C et al, there isn't a whole lot 
of support for (say) specialized hardware to run the disks. Mainframes have 
chips for I/O just like desktop systems have custom chips for 3D graphics, 
and if you don't take advantage of it, you get pretty average performance.

I don't know what kinds of things mainframes have nowadays, but I'd be 
surprised if you didn't get serious performance enhancement by (for example) 
coding up database tables to use raw I/O to the hardware. (Indeed, UNIX 
database systems used to prefer to work on /dev raw disk type partitions, 
rather than stuff in the file system.)  Watch your performance die when one 
of the top-level leaves in your busy B-tree winds up on a remapped sector on 
the disk.

I think it's also the case that the number of places that need transactional 
data that's actually that large is surprisingly small. I would guess that 
most new businesses that need tens-of-terabytes databases don't need them to 
be transactional. (As in, I expect you get very few race conditions trying 
to update climate data in the multi-petabyte databases.)

-- 
   Darren New, San Diego CA, USA (PST)
   There's no CD like OCD, there's no CD I knoooow!

Post a reply to this message