On 03/08/2011 04:35 PM, Darren New wrote:
> On 8/3/2011 7:44, Invisible wrote:
>> Perhaps if I worked at Google, managing 20,000 "servers" [which are
>> really
>> just commodity desktop PCs], having that many disks might be an issue.
>
>  From what I've read, closer to half a million servers, all with the
> disk attached directly. :-)

Sure, but not all in one place, and not with one guy in charge of it 
all. I'd imagine 20,000 machines in one datacenter is not an 
unreasonable guess.

And yes, from what I can tell, Google doesn't do SAN. They replicate at 
the filesystem level. Then again, all their machines run the same 
application. Business datacenters aren't usually like that. Even so, it 
seems "obvious" to me that the network should operate above the 
filesystem level, not below it.

> You generate 200G of incremental data a day? ow.

Nah, that's for a full backup. An incrimental backup is more like 10GB. 
Which is still going to take a while at 5 mbit/sec. (Once you add IP 
framing, TCP overhead, VPN encryption, latency, other WAN traffic...)

> Nothing wrong with spinning disk backup, especially if it's more
> reliable than the tape itself.

This is the thing, really. Disks spin constantly. Quite apart from the 
power that uses, it also means that at any instant they can break. 
Mechanical failure, electronics failure, software glitch, whatever. 
Tapes just sit there on the shelf, doing nothing. About the only thing 
you need to worry about is the tape demagnetising.

Did I mention that tape is cheaper?

>> So it's no good at all just mirroring what's on the server onto another
>> server somewhere else. The /history/ must be kept. Now, there are various
>> ways you might achieve that, but all of them unavoidably involve the
>> set of
>> backup disks being drastically larger than the total size of the working
>> disks.
>
> Not really. You don't actually change that much, I expect. Tapes don't
> have things like hard links and directories, but spinning disks do. Make
> a full backup, then an incremental once a day for a week, then a weekly
> incremental, etc, until you get up to monthly.

Note that for our purposes, security information, file modification 
times and so forth must also be reliably stored and retrieved.

>> something that's far less reliable.
>
> I'm not sure how you know spinning disk is less reliable.

Well, I don't /know/ for a fact. It just seems highly likely.

(We /know/ that disks have an annual failure rate of about 3% to 5%. We 
don't know what the rate is for tape.)

> Once it fills up, you disconnect it and put in a new one, and you put
> the old on on the shelf.

And I thought drives don't like power cycles...

>> Still, what I do know? Apparently not a lot.
>
> You know a lot. You should spend time writing this up, especially the
> bandwidth part, and send it to the people who would be able to evaluate
> this.

Heh, yeah, like I want to get yelled at.

> If nothing else, ask them what backup software they plan to use that
> will do incremental backups over a network and keep every old backup
> separately restorable.

I already know for a fact that they haven't decided how to implement 
this yet. They're currently looking at options. I will of course make 
sure they know what the hard requirements are.

Post a reply to this message