Warp wrote:
>   Btw, do database systems (such as Oracle) support any other interfaces
> than SQL? 

This is cool, btw:

http://www.businessintelligencelowdown.com/2007/02/top_10_largest_.html

I know the AT&T database runs SQL. When I was there 15 years ago, they had 
100 million lines of SQL.

-- 
   Darren New, San Diego CA, USA (PST)
   There's no CD like OCD, there's no CD I knoooow!

Post a reply to this message

Darren New <dne### [at] sanrrcom> wrote:
> http://www.businessintelligencelowdown.com/2007/02/top_10_largest_.html

  I can only imagine how expensive it is to backup all that information
(and I have hard time believing they don't backup it). Not only is the
backup medium expensive (because of the sheer amount required), but
physically *storing* it somewhere must also be expensive (after all,
physical storage space is never free). I also imagine that they want to
store the backups in a secure location (from accidents, eg. fires, and
robberies/vandalism), which can only make the storage costs several times
larger.

-- 
                                                          - Warp

Post a reply to this message

Warp <war### [at] tagpovrayorg> wrote:
> Darren New <dne### [at] sanrrcom> wrote:
> > http://www.businessintelligencelowdown.com/2007/02/top_10_largest_.html
>
>   I can only imagine how expensive it is to backup all that information
> (and I have hard time believing they don't backup it). Not only is the
> backup medium expensive (because of the sheer amount required), but
> physically *storing* it somewhere must also be expensive (after all,
> physical storage space is never free). I also imagine that they want to
> store the backups in a secure location (from accidents, eg. fires, and
> robberies/vandalism), which can only make the storage costs several times
> larger.
>
> --
>                                                           - Warp

The company I work for is no where near as large as ATT.  We have a small to
medium size mainframe operation, with about 15TB of online disk storage and
around 120TB of tape for backup and archival storage. We keep a minimum of
three backup copies of all critical data, one of those copies is always stored
offsite. For legal reasons some data we have to keep more copies for a longer
period of time. For non-critical data we normally keep only a couple of copies
locally.  No it is not cheap, but it is a lot cheaper than losing the data,
facing lawsuits, fines, and going out of business.

Isaac.

Post a reply to this message

Warp wrote:
>   I can only imagine how expensive it is to backup all that information
> (and I have hard time believing they don't backup it). 

Back in 1992 or so, I spent about a week locked in with EDS's credit card 
processing systems. (EDS handled at least the top four credit card banks at 
the time, and of course a bunch of other stuff.) A room maybe 50 meters on a 
side, two power feeds from two separate electric companies, full of 
mainframes, printers, etc. There were at least three people at any given 
time wheeling carts full of tapes between the tape changers and the back 
room. Plus two people on "wheel fresh paper to the 1600LPM line printers" 
duty. It was a pretty impressive sight.

As for AT&T, altho I never thought to ask about backup stuff, I do know they 
had at least 400 people full-time coding SQL to maintain it.

-- 
   Darren New, San Diego CA, USA (PST)
   There's no CD like OCD, there's no CD I knoooow!

Post a reply to this message

Darren New <dne### [at] sanrrcom> wrote:
> As for AT&T, altho I never thought to ask about backup stuff, I do know they 
> had at least 400 people full-time coding SQL to maintain it.

  Speaking of AT&T... 312 terabytes in one single database? That starts
requiring some efficiency from the part of the database software...

  I think it could perhaps be safely assumed they use Oracle? (I have read
Oracle is about the only database system out there which scales up so much.
Or at least that's what the Oracle company wants us to believe in any case.)

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:
>   Speaking of AT&T... 312 terabytes in one single database? That starts
> requiring some efficiency from the part of the database software...

Indeed!  Almost mind-boggling.  And that's just one of the 9 large databases 
they have.  This one sounds like PREMIS, which is all the call data, 
billing, etc. Basically, ever interaction with an end-user of the network.

The other giant one (which I thought used to be bigger, but might have 
shrunk with more fiber while PREMIS continues to grow) is TURKS. The Trunk 
Utilization and Resource Knowledge System tracks basically where all the 
wires go, what streets they go under, what the connections between cables 
are, each punch-down block, each manhole cover, the configurations of all 
the phone switches, which wires connect to which line cards, and basically 
everything that has ever been installed, who worked on it, what time, how 
long it took, etc. Given that the local loops alone in the USA used 58 
light-minutes of copper wire, you can imagine the kind of data you're 
talking about.

I'm pretty sure they roll operational stuff out of the main databases after 
six months or so, but I guess they still keep all the records somewhere.

>   I think it could perhaps be safely assumed they use Oracle? 

It's IBM mainframe DB2. Or was, 15 years ago. I don't imagine they changed 
that, tho.

 > I have read
> Oracle is about the only database system out there which scales up so much.

It probably has the best scale per dollar. I don't imagine it's going to 
take advantage of mainframe features like IOPs and knowing which sectors are 
on which cylinders of the hard disks and such. But mainframes still blow 
away I/O driven applications by hundreds to one.  Apparently, some people do 
things like put dozens of Linux VMs on one mainframe,

> Or at least that's what the Oracle company wants us to believe in any case.)

I think IBM has the same sort of advertising. You just see it less because 
it's not really targeted at minicomputer type installations.

-- 
   Darren New, San Diego CA, USA (PST)
   There's no CD like OCD, there's no CD I knoooow!

Post a reply to this message

Darren New <dne### [at] sanrrcom> wrote:
> >   I think it could perhaps be safely assumed they use Oracle? 

> It's IBM mainframe DB2. Or was, 15 years ago. I don't imagine they changed 
> that, tho.

  From the wikipedia page I get the impression that Oracle and IBM DB2 are
approximately the only two RDBMS systems which scale up well to the hundreds
of terabytes of data.

  I wonder how eg. MySQL or PostgreSQL would handle such vast amounts of
data. Would they handle them ok, just maybe a bit slower, would they be
significantly less efficient, or would they simply fail? Or is it a question
of them not having such extensive support for the hardware required to
handle that amount of data efficiently?

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:
>   From the wikipedia page I get the impression that Oracle and IBM DB2 are
> approximately the only two RDBMS systems which scale up well to the hundreds
> of terabytes of data.

Cool.

>   I wonder how eg. MySQL or PostgreSQL would handle such vast amounts of
> data. Would they handle them ok, just maybe a bit slower, would they be
> significantly less efficient, or would they simply fail?

"""InnoDB has a limit of 1023 concurrent transactions that have created undo 
records by modifying data."""
I doubt you're going to get 70,000 transactions a second on a MySql 
installation, altho I don't see anything about individual table sizes listed 
in terms of number of rows.

The latest MySql seems to have a storage engine that talks to DB2 databases, 
so I'm not sure if there's any significant difference there.

However, given that looking something up on an index on a 6-million-row 
table took over a second on a 64-bit machine, I wouldn't want to put 
billions of rows, let alone trillions, into a MySQL table.

> Or is it a question
> of them not having such extensive support for the hardware required to
> handle that amount of data efficiently?

I would imagine that since they're portable C et al, there isn't a whole lot 
of support for (say) specialized hardware to run the disks. Mainframes have 
chips for I/O just like desktop systems have custom chips for 3D graphics, 
and if you don't take advantage of it, you get pretty average performance.

I don't know what kinds of things mainframes have nowadays, but I'd be 
surprised if you didn't get serious performance enhancement by (for example) 
coding up database tables to use raw I/O to the hardware. (Indeed, UNIX 
database systems used to prefer to work on /dev raw disk type partitions, 
rather than stuff in the file system.)  Watch your performance die when one 
of the top-level leaves in your busy B-tree winds up on a remapped sector on 
the disk.

I think it's also the case that the number of places that need transactional 
data that's actually that large is surprisingly small. I would guess that 
most new businesses that need tens-of-terabytes databases don't need them to 
be transactional. (As in, I expect you get very few race conditions trying 
to update climate data in the multi-petabyte databases.)

-- 
   Darren New, San Diego CA, USA (PST)
   There's no CD like OCD, there's no CD I knoooow!

Post a reply to this message

Darren New <dne### [at] sanrrcom> wrote:
> > anyway, is it much of a hassle to use double to represent monetary values?
>
> Yes. It's not correct.  Add up a billion dollars in penny increments, and
> tell me you can guarantee it's right.

To my knowledge that's not even the main issue:

Science cares about high precision.

Business doesn't give a damn about high precision, as long as rounding rules are
precisely followed.

Ever seen a "pocket" calculator for business use? They have extra switches to
set the rounding mode, e.g. fixed two digits, fixed five digits, or
what-have-you.

I once had to hack up some programs to run plausibility checks on records in an
insurance company "database" (well, individual files, but what the heck...
those were the database for the "worst case" contracts, that had been the last
to be ported to some newer software that hadn't been able to handle them
earlier, due to a lot of exceptional parameters in them).

I guess there wasn't a single record in the database where the plausibility
check did compute a perfect match for the values in the database. Not because
my program would have been imprecise - but because for simplicity I had left
out a huge number of rounding rules that I would have had to add otherwise.

In COBOL, the rounding rules are automatically enforced by the data format
description in the WORKING STORAGE SECTION.

Post a reply to this message

clipka wrote:
> Science cares about high precision.

And, actually, science usually doesn't care about high precicion for that 
matter, either. Usually the accuracy is far less than the precision of a 
single float, if you're talking about any sort of control application. Or, 
in the words of one embedded control programmer I knew: "If you need more 
than six digits of precision and you're not getting feedback to correct 
errors, you've already blown up the wrong building."

Of course, there are *some* apps where having unlimited precision is a good 
thing (POV-Ray springs to mind), but it's always going to be limited by your 
input and output.

-- 
   Darren New, San Diego CA, USA (PST)
   There's no CD like OCD, there's no CD I knoooow!

Post a reply to this message