 |
|
|
|
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Hooookay, well that was an interesting meeting! o_O
First, it appears that our head office is replacing their current
arrangement with a big new SAN.
The concept of a SAN utterly baffles me. As far as I can tell, a "SAN"
basically means that instead of connecting your HDs to the server that
wants to use them, you connect them to a network instead, and then the
server accesses it over the network. This has the following benefits:
* Tens of thousands of times more expensive than a direct connection.
* Hundreds of thousands of times reduced performance compared to a
direct connection.
* Radically increased complexity compared to a direct connection.
* If the storage network fails, ALL servers fail, so you're adding a new
single point of failure.
* All the servers now have to share the limited bandwidth available. One
busy server can bring all the others to a crawl.
* Significantly more "enterprisey" than a direct connection.
Actually, wait... those are all disadvantages. Really, really /big/
disadvantages. Huh, OK. So why on Earth would any sane person embark on
this course of action??
Oh, wait, I just found a real advantage: If you want to move a disk from
one server to another, you just press a button, rather than having to
physically move the device.
Oh, well, that /totally/ makes it worth it. And, uh, when will you
*ever* need to perform this operation? I mean, seriously, I've been
working at the UK site for almost 10 years now, and I've never once
wanted to do this. And even if I did, if on one day in 10 years I have
to shut both servers down to physically move a drive, I think I can
probably live with that.
Perhaps if I worked at Google, managing 20,000 "servers" [which are
really just commodity desktop PCs], having that many disks might be an
issue. And with that many machines, perhaps the massive performance hit
would be acceptable. But anywhere else? WTF?
Fortunately, this only affects HQ, so in a sense I don't have to care
about this. Even so, it still baffles me.
Second, they're replacing our tape backup system with a disk-based
backup system. That's right, they're seriously talking about
transferring over 200 GB of data from our UK site to our USA
headquarters, every night, via the Internet.
Jesus, this meeting just gets better, doesn't it?
First of all, speed. The LTO-3 tape system we currently use has a
maximum transfer rate of 80 MB/sec. At that rate, 200 GB should
theoretically take about 41 minutes (which agrees with my actual backup
logs). But our Internet connection is a piffling 5 Mbit/sec. At that
speed, 200 GB should theoretically take... 3 days + 17 hours.
And you want to do this /nightly/???
Second, what is the purpose of a backup? The purpose of having a "backup
copy" of your data is so that if your working copy dies somehow, you can
use the backup copy to get your data back.
This only works if the backup copy is more reliable than the working
copy. If your working copy is on spinning disk, and you're stupid enough
to put your backup copy on spinning disk as well... then it becomes
equally likely that the /backup copy/ will die and you'll be left with
just the working copy.
Third, having access only to the most recent backup is no good. There
are scenarios where that would be perfectly OK, but in our line of
business, we sometimes have to recover data from /months/ ago. Data that
has been accidentally deleted. Data which got corrupted. Data which we
thought we didn't need any more but actually we do. And so forth.
So it's no good at all just mirroring what's on the server onto another
server somewhere else. The /history/ must be kept. Now, there are
various ways you might achieve that, but all of them unavoidably involve
the set of backup disks being drastically larger than the total size of
the working disks. And, if we're going to continue our current policy of
never deleting old backups, then the backup disk set must continue to
grow forever. Given the relative prices of tape (£17 for 400GB) verses
disk (£25 for 250GB), that's an expensive proposition for something
that's far less reliable.
And then there's the fact that you either need to keep all this disk
spinning (i.e., ever increasing power and cabling demands), or only keep
the recent backups spinning (i.e., managing powering off and powering on
drives, which supposedly shortens their lifetime).
In all, this seems like an extremely bad idea.
Still, what I do know? Apparently not a lot.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Invisible <voi### [at] dev null> wrote:
> Hooookay, well that was an interesting meeting! o_O
> ...
> Still, what I do know? Apparently not a lot.
Knowledge isn’t the issue here. The problem is that you’re just not thinking
like a Systems Administrator. (The capitalization *is* correct because the names
of deities always begin with a capital letter.)
You’re asking questions like: “What’s the most reasonable, efficient and
cost-effective way of meeting the firms actual needs?” You’ll never become an
Admin thinking like that. Instead, you need to learn to ask the sorts of
questions that are pertinent to the true concerns and goals of an Administrative
Professional. You should ask, for example, “How can I constantly initiate new
projects and changes to old ones at a sufficient rate such that my only job will
consist of giving my ‘expert’ opinions and instructions to others rather than
having to do any actual work myself?” In the absence of perpetual upheaval, the
necessity of employing someone at astronomical cost whose only function is to
mange the upheaval could be called into question.
Also, you need to have a broader perspective with regard to how issues which are
IT matters in one sense integrate into a more comprehensive view of world
affairs. Take the overall state of the global economy for example. One with a
truly Administrative mindset would think ahead and ask: “How can I make the
firm’s IT needs complicated and extensive enough so as to justify putting
numerous unemployed technically inept friends and relatives on the payroll as
Co-Administrators?”
An Administrator must be a “Team Player” who looks at “The Big Picture” while
simultaneously multi-tasking with regard to numerous additional trite clichés as
well. Dealing with trivialities like storing data securely and managing the
flow of information so as to facilitate genuinely productive tasks is a job for
the low-paid nerds.
I hope this clears things up... No! No!... I mean I hope this obfuscates matters
thoroughly while imparting an air of authority unto myself. (I’m trying to hone
my own Administrative “skills”.)
I’ll have my people send regards to your people,
Mike C.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 8/3/2011 7:44, Invisible wrote:
> Perhaps if I worked at Google, managing 20,000 "servers" [which are really
> just commodity desktop PCs], having that many disks might be an issue.
From what I've read, closer to half a million servers, all with the disk
attached directly. :-)
> Second, they're replacing our tape backup system with a disk-based backup
> system. That's right, they're seriously talking about transferring over 200
> GB of data from our UK site to our USA headquarters, every night, via the
> Internet.
You generate 200G of incremental data a day? ow.
> This only works if the backup copy is more reliable than the working copy.
> If your working copy is on spinning disk, and you're stupid enough to put
> your backup copy on spinning disk as well... then it becomes equally likely
> that the /backup copy/ will die and you'll be left with just the working copy.
That's why you have two spinning disk backups that you alternate. Nothing
wrong with spinning disk backup, especially if it's more reliable than the
tape itself.
> So it's no good at all just mirroring what's on the server onto another
> server somewhere else. The /history/ must be kept. Now, there are various
> ways you might achieve that, but all of them unavoidably involve the set of
> backup disks being drastically larger than the total size of the working
> disks.
Not really. You don't actually change that much, I expect. Tapes don't have
things like hard links and directories, but spinning disks do. Make a full
backup, then an incremental once a day for a week, then a weekly
incremental, etc, until you get up to monthly.
> something that's far less reliable.
I'm not sure how you know spinning disk is less reliable.
> And then there's the fact that you either need to keep all this disk
> spinning (i.e., ever increasing power and cabling demands), or only keep the
> recent backups spinning (i.e., managing powering off and powering on drives,
> which supposedly shortens their lifetime).
Once it fills up, you disconnect it and put in a new one, and you put the
old on on the shelf.
> Still, what I do know? Apparently not a lot.
You know a lot. You should spend time writing this up, especially the
bandwidth part, and send it to the people who would be able to evaluate this.
If nothing else, ask them what backup software they plan to use that will do
incremental backups over a network and keep every old backup separately
restorable.
--
Darren New, San Diego CA, USA (PST)
How come I never get only one kudo?
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 8/3/2011 7:44, Invisible wrote:
> having that many disks might be an issue.
Speaking of which, I recently read a quote about the early days of Google.
Levy had seen the racks where Google's storage was running and said "If you
can imagine a college freshman made out of gigabytes, this would be his dorm
room."
--
Darren New, San Diego CA, USA (PST)
How come I never get only one kudo?
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 03/08/2011 04:30 PM, Mike the Elder wrote:
> You’re asking questions like: “What’s the most reasonable, efficient and
> cost-effective way of meeting the firms actual needs?” You’ll never become an
> Admin thinking like that. Instead, you need to learn to ask the sorts of
> questions that are pertinent to the true concerns and goals of an Administrative
> Professional. You should ask, for example, “How can I constantly initiate new
> projects and changes to old ones at a sufficient rate such that my only job will
> consist of giving my ‘expert’ opinions and instructions to others rather than
> having to do any actual work myself?” In the absence of perpetual upheaval, the
> necessity of employing someone at astronomical cost whose only function is to
> mange the upheaval could be called into question.
Pfahahahahah!
Apparently you don't realise what they pay me. :-P
Mind you, the argument works from the point of view of the Director of IT...
> Also, you need to have a broader perspective with regard to how issues which are
> IT matters in one sense integrate into a more comprehensive view of world
> affairs. Take the overall state of the global economy for example. One with a
> truly Administrative mindset would think ahead and ask: “How can I make the
> firm’s IT needs complicated and extensive enough so as to justify putting
> numerous unemployed technically inept friends and relatives on the payroll as
> Co-Administrators?”
Hahah. That would be funny if it wasn't true. The number of
mission-critical applications here developed by somebody's nephew in
QBASIC is ridiculous.
> An Administrator must be a “Team Player” who looks at “The Big Picture” while
> simultaneously multi-tasking with regard to numerous additional trite clichés as
> well.
BULLSHIT!
Uh, I mean, BINGO!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 03/08/2011 04:35 PM, Darren New wrote:
> On 8/3/2011 7:44, Invisible wrote:
>> Perhaps if I worked at Google, managing 20,000 "servers" [which are
>> really
>> just commodity desktop PCs], having that many disks might be an issue.
>
> From what I've read, closer to half a million servers, all with the
> disk attached directly. :-)
Sure, but not all in one place, and not with one guy in charge of it
all. I'd imagine 20,000 machines in one datacenter is not an
unreasonable guess.
And yes, from what I can tell, Google doesn't do SAN. They replicate at
the filesystem level. Then again, all their machines run the same
application. Business datacenters aren't usually like that. Even so, it
seems "obvious" to me that the network should operate above the
filesystem level, not below it.
> You generate 200G of incremental data a day? ow.
Nah, that's for a full backup. An incrimental backup is more like 10GB.
Which is still going to take a while at 5 mbit/sec. (Once you add IP
framing, TCP overhead, VPN encryption, latency, other WAN traffic...)
> Nothing wrong with spinning disk backup, especially if it's more
> reliable than the tape itself.
This is the thing, really. Disks spin constantly. Quite apart from the
power that uses, it also means that at any instant they can break.
Mechanical failure, electronics failure, software glitch, whatever.
Tapes just sit there on the shelf, doing nothing. About the only thing
you need to worry about is the tape demagnetising.
Did I mention that tape is cheaper?
>> So it's no good at all just mirroring what's on the server onto another
>> server somewhere else. The /history/ must be kept. Now, there are various
>> ways you might achieve that, but all of them unavoidably involve the
>> set of
>> backup disks being drastically larger than the total size of the working
>> disks.
>
> Not really. You don't actually change that much, I expect. Tapes don't
> have things like hard links and directories, but spinning disks do. Make
> a full backup, then an incremental once a day for a week, then a weekly
> incremental, etc, until you get up to monthly.
Note that for our purposes, security information, file modification
times and so forth must also be reliably stored and retrieved.
>> something that's far less reliable.
>
> I'm not sure how you know spinning disk is less reliable.
Well, I don't /know/ for a fact. It just seems highly likely.
(We /know/ that disks have an annual failure rate of about 3% to 5%. We
don't know what the rate is for tape.)
> Once it fills up, you disconnect it and put in a new one, and you put
> the old on on the shelf.
And I thought drives don't like power cycles...
>> Still, what I do know? Apparently not a lot.
>
> You know a lot. You should spend time writing this up, especially the
> bandwidth part, and send it to the people who would be able to evaluate
> this.
Heh, yeah, like I want to get yelled at.
> If nothing else, ask them what backup software they plan to use that
> will do incremental backups over a network and keep every old backup
> separately restorable.
I already know for a fact that they haven't decided how to implement
this yet. They're currently looking at options. I will of course make
sure they know what the hard requirements are.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On Wed, 03 Aug 2011 08:35:42 -0700, Darren New wrote:
> On 8/3/2011 7:44, Invisible wrote:
>> Perhaps if I worked at Google, managing 20,000 "servers" [which are
>> really just commodity desktop PCs], having that many disks might be an
>> issue.
>
> From what I've read, closer to half a million servers, all with the
> disk
> attached directly. :-)
Most recently it was suggested the number was closer to 900,000. :)
Jim
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 8/3/2011 8:54, Invisible wrote:
> They replicate at the filesystem level.
Sorta. I guess if you call the separate library with separate semantics a
"filesystem", then yes. (And yes, I do, but many wouldn't, for some reason.)
> Then again, all their machines run the same application.
Huh? No they don't. That doesn't even make sense, since then the only
application they'd be running would be "the file system." Heck, even the
file system has three or four separate applications to run it.
> Business datacenters aren't usually like that. Even so, it seems "obvious"
> to me that the network should operate above the filesystem level, not below it.
Personally, I've never figured out the draw of paying more for a terabyte on
dedicated hardware that's not any more reliable or inexpensive than just
plugging a terabyte drive into a PC.
>> Nothing wrong with spinning disk backup, especially if it's more
>> reliable than the tape itself.
>
> This is the thing, really. Disks spin constantly.
Not backup disks. No more than your backup tapes from last year do.
> Did I mention that tape is cheaper?
This is true, yes. That's the one benefit.
> Note that for our purposes, security information, file modification times
> and so forth must also be reliably stored and retrieved.
Sure. You can still do that with disks and hard links. If the security
information or file modification times change, you add it to the backup.
Good backup software will keep a checksum and either not actually copy the
data if only the metadata has changed.
> (We /know/ that disks have an annual failure rate of about 3% to 5%. We
> don't know what the rate is for tape.)
Depends if they're spinning or not. I can guarantee that if you left a tape
moving like you say you want to leave disks moving, you're have failures
much more frequently.
>> Once it fills up, you disconnect it and put in a new one, and you put
>> the old on on the shelf.
>
> And I thought drives don't like power cycles...
How often are you going to power cycle it? You'll turn it off when it's
full, and not turn it on again until you're ready to use it.
I think it's not so much that disks don't like power cycles (especially
nowadays in the last 10 years or so), but that most failures happen when
powering them up or down.
In any case, spinning up the drive once a day and then turning it back off
again isn't going to significantly decrease its lifetime. Buy a disk
designed for that sort of use, rather than a "server" disk that's optimized
to not fail when it's always spinning.
> I already know for a fact that they haven't decided how to implement this
> yet. They're currently looking at options. I will of course make sure they
> know what the hard requirements are.
Well, that's all you can do, really. :-)
--
Darren New, San Diego CA, USA (PST)
How come I never get only one kudo?
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 8/3/2011 7:44 AM, Invisible wrote:
> * Tens of thousands of times more expensive than a direct connection.
>
> * Hundreds of thousands of times reduced performance compared to a
> direct connection.
>
> * Radically increased complexity compared to a direct connection.
>
> * If the storage network fails, ALL servers fail, so you're adding a new
> single point of failure.
>
> * All the servers now have to share the limited bandwidth available. One
> busy server can bring all the others to a crawl.
>
> * Significantly more "enterprisey" than a direct connection.
>
> Actually, wait... those are all disadvantages. Really, really /big/
> disadvantages. Huh, OK. So why on Earth would any sane person embark on
> this course of action??
>
>...
Hmm.. So, in tech, its the same as everything else now. Facts, reality,
and sane policy don't matter, just whether or not the idea sells? lol
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 04/08/2011 06:26 PM, Patrick Elliott wrote:
> Hmm.. So, in tech, its the same as everything else now. Facts, reality,
> and sane policy don't matter, just whether or not the idea sells? lol
What do you mean "now"? :-P
I read somewhere that way, waaaaay back in the days when people actually
bought line printers from IBM, and they cost hundreds of thousands of
dollars, there was a model that had a "field installable upgrade". For
something like 80,000 USD, an IBM engineer would come to your site,
remove one of the drive wheels, and install one with a different
diameter. This made the printer print twice as fast.
It's an engineering solution - once you realise that the goal is not to
make a better printer, but to bleed the customer dry.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Le 2011-08-03 10:44, Invisible a écrit :
> Hooookay, well that was an interesting meeting! o_O
>
>
>
> First, it appears that our head office is replacing their current
> arrangement with a big new SAN.
>
> The concept of a SAN utterly baffles me. As far as I can tell, a "SAN"
> basically means that instead of connecting your HDs to the server that
> wants to use them, you connect them to a network instead, and then the
> server accesses it over the network. This has the following benefits:
>
> * Tens of thousands of times more expensive than a direct connection.
>
For a small or remote office with a couple file-servers, I agree that it
doesn't make sense but for a data center, it's a different story. If
you have 200 servers connected to the SAN, the "unit cost" of the SAN
hardware becomes a lot more palatable, and you can leverage unused disk
space that nay servers will have.
> * Hundreds of thousands of times reduced performance compared to a
> direct connection.
>
Not necessarily. It depends a lot on the configuration of the locally
attached drives vs. SAN logical disks, and the types of IO requests that
are made (e.g. long sequential reads or writes vs. lots of short random
file accesses). SANs have all kinds of caching mechanisms that can make
the IO actually faster than a SATA connection.
> * Radically increased complexity compared to a direct connection.
>
That depends on the point of view. From the server administrator's
point of view, it's much simpler. She no longer has to worry about
spreading her disks across multiple controllers, mirroring, etc.... on
which partition should the indexes go? What level of RAID to use? Is
there enough power in the server's power supplies to drive an additional
pair of disks? It's all taken care of at the other end of a pair of
fibre connectors.
From the point of view of the backup administrator, it's much simpler
since he no longer has to worry about making sure that the backup agent
is running on all the servers and that their schedules don't actually
step over each other's toes, etc... it can all be done off-line in the
background.
You just had a very weird problem? Just clone the DB to an new set of
disks, and assign those on a lab server so that the developpers can work
on the issue all weekend.
The biggest added level of complexity is that all these people have to
talk to one another. Scary. I know.
> * If the storage network fails, ALL servers fail, so you're adding a new
> single point of failure.
>
True, but that rarely happens. In 15 years, I've seen it happen once,
and heard of one other occurance. Compared to a hundred of Compaq
Proliant drive arrays marking all disks in the RAID bad at once,
happening on a weekly basis for 6 months, until Compaq admitted they had
a faulty backplane desing on their units and agreed to replace them all...
> * All the servers now have to share the limited bandwidth available. One
> busy server can bring all the others to a crawl.
What limited bandwidth? Even a SAN that's been improperly designed, and
that has seen 5 years of unchecked growth will have more than enough
bandwitdh to accomodate all the servers. Have you seen the
specifications of the SAN your head office wants to implement and were
able to determine there will be bottlenecks?
> Oh, wait, I just found a real advantage: If you want to move a disk from
> one server to another, you just press a button, rather than having to
> physically move the device.
>
> Oh, well, that /totally/ makes it worth it. And, uh, when will you
> *ever* need to perform this operation? I mean, seriously, I've been
> working at the UK site for almost 10 years now, and I've never once
> wanted to do this. And even if I did, if on one day in 10 years I have
> to shut both servers down to physically move a drive, I think I can
> probably live with that.
My current customer has two data centers with 350 servers in each. "We
need more disk space on server XYZ" happens on a daily basis, in fact,
multiple times a day, in some cases (e.g. a fast growing log file due to
a change in user traffic, or the activation of a debbugging feature
during problem troubleshooting)
> First of all, speed. The LTO-3 tape system we currently use has a
> maximum transfer rate of 80 MB/sec. At that rate, 200 GB should
> theoretically take about 41 minutes (which agrees with my actual backup
> logs). But our Internet connection is a piffling 5 Mbit/sec. At that
> speed, 200 GB should theoretically take... 3 days + 17 hours.
>
> And you want to do this /nightly/???
>
> Second, what is the purpose of a backup? The purpose of having a "backup
> copy" of your data is so that if your working copy dies somehow, you can
> use the backup copy to get your data back.
>
> This only works if the backup copy is more reliable than the working
> copy. If your working copy is on spinning disk, and you're stupid enough
> to put your backup copy on spinning disk as well... then it becomes
> equally likely that the /backup copy/ will die and you'll be left with
> just the working copy.
As long as your working copy doesn't disappear before your can recreate
the backup copy, where's the problem? Are you saying that RAID is a
waste of time because there's an equal likelyhood that any of the disks
in the array will die?
> Third, having access only to the most recent backup is no good. There
> are scenarios where that would be perfectly OK, but in our line of
> business, we sometimes have to recover data from /months/ ago. Data that
> has been accidentally deleted. Data which got corrupted. Data which we
> thought we didn't need any more but actually we do. And so forth.
>
Have they actually stated that you would lose the ability of going back
to older copies of your backups?
If so, make a business case (or let your boss know) that this is
unacceptable. They may reconsider, or decide that it's not worth the
investment. Either way, you will have your "I told you so!" e-mail in
case a problem occurs.
> So it's no good at all just mirroring what's on the server onto another
> server somewhere else. The /history/ must be kept. Now, there are
> various ways you might achieve that, but all of them unavoidably involve
> the set of backup disks being drastically larger than the total size of
> the working disks. And, if we're going to continue our current policy of
> never deleting old backups, then the backup disk set must continue to
> grow forever. Given the relative prices of tape (£17 for 400GB) verses
> disk (£25 for 250GB), that's an expensive proposition for something
> that's far less reliable.
>
> And then there's the fact that you either need to keep all this disk
> spinning (i.e., ever increasing power and cabling demands), or only keep
> the recent backups spinning (i.e., managing powering off and powering on
> drives, which supposedly shortens their lifetime).
>
> In all, this seems like an extremely bad idea.
>
Have they actually stated that they would not be taking tape (or more
than likely DVD) backups of that data for archival?
In all likelyhood, that remote copy will probably be backed up to
offsite storage - if not, it should be, just for the sake of business
continuity, but in many cases, for legal reasons as well.
>
>
> Still, what I do know? Apparently not a lot.
No offense, but I thought this was an established fact. ;)
Don't get me wrong, you are extremely knowledgeable in many areas, but
you seem to be unable to admit that other people may have other reasons
to do something that you deem unnecessary because it doesn't apply to
your situation.
--
/*Francois Labreque*/#local a=x+y;#local b=x+a;#local c=a+b;#macro P(F//
/* flabreque */L)polygon{5,F,F+z,L+z,L,F pigment{rgb 9}}#end union
/* @ */{P(0,a)P(a,b)P(b,c)P(2*a,2*b)P(2*b,b+c)P(b+c,<2,3>)
/* gmail.com */}camera{orthographic location<6,1.25,-6>look_at a }
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> * Tens of thousands of times more expensive than a direct connection.
>
> For a small or remote office with a couple file-servers, I agree that it
> doesn't make sense but for a data center, it's a different story. If you
> have 200 servers connected to the SAN, the "unit cost" of the SAN
> hardware becomes a lot more palatable, and you can leverage unused disk
> space that nay servers will have.
AFAIK, our data centre has about 10 physical servers. (And a slightly
larger number of virtual servers.)
They're talking about increasing the number of virtual servers (possibly
making /every/ end-user server virtual), which makes being able to swap
disks around a more attractive proposition. If physical server X fails,
move all the disks to physical server Y and start all the VMs back up.
I'm still not convinced we need to spend 150,000 USD just to make it
easier to move disks though.
>> * Hundreds of thousands of times reduced performance compared to a
>> direct connection.
>
> Not necessarily. It depends a lot on the configuration of the locally
> attached drives vs. SAN logical disks, and the types of IO requests that
> are made (e.g. long sequential reads or writes vs. lots of short random
> file accesses). SANs have all kinds of caching mechanisms that can make
> the IO actually faster than a SATA connection.
SATA 3.0 connection: 6.0 GB/sec
Fast Ethernet connection: 0.1 GB/sec
That's, what, 6000x slower?
Sure, you can add caching, which might speed up write access. But it
can't speed up read access that misses the cache.
>> * Radically increased complexity compared to a direct connection.
>
> That depends on the point of view. From the server administrator's point
> of view, it's much simpler. She no longer has to worry about spreading
> her disks across multiple controllers, mirroring, etc.... on which
> partition should the indexes go? What level of RAID to use? Is there
> enough power in the server's power supplies to drive an additional pair
> of disks? It's all taken care of at the other end of a pair of fibre
> connectors.
I fail to see why RAID is no longer an issue. (Unless you mean that it's
now acceptable to lose data in the event of a hardware fault.)
I've yet to see any system where power consumption was even remotely
important. Servers seem to come with power supplies that can handle
*way* more current than you will ever need.
I've also never seen a sever with more than one disk controller, come to
mention it...
> From the point of view of the backup administrator, it's much simpler
> since he no longer has to worry about making sure that the backup agent
> is running on all the servers and that their schedules don't actually
> step over each other's toes, etc... it can all be done off-line in the
> background.
I don't see how that works.
> You just had a very weird problem? Just clone the DB to an new set of
> disks, and assign those on a lab server so that the developpers can work
> on the issue all weekend.
You can trivially do that without a SAN. Disk imaging tools have existed
for decades. Either way, you still have to take the DB server offline
while you do this.
Of course, if your DB server is a VM, the problem just became even more
trivial. Depending on your virtualisation technology, maybe you don't
even have to stop the server to copy the disk image. And you /still/
don't need a SAN.
> The biggest added level of complexity is that all these people have to
> talk to one another. Scary. I know.
Nah. Our entire IT function (including me) is about half a dozen humans.
>> * If the storage network fails, ALL servers fail, so you're adding a new
>> single point of failure.
>
> True, but that rarely happens.
OK, fair enough.
>> * All the servers now have to share the limited bandwidth available. One
>> busy server can bring all the others to a crawl.
>
> What limited bandwidth? Even a SAN that's been improperly designed, and
> that has seen 5 years of unchecked growth will have more than enough
> bandwitdh to accomodate all the servers. Have you seen the
> specifications of the SAN your head office wants to implement and were
> able to determine there will be bottlenecks?
My point is, if you have 10 servers, each with their own dedicated SATA
connection (or whatever), you have 10 lots of bandwidth. And if one
server decides to hog it all, it can't slow the others down.
If you have a SAN, you have /one/ access channel, which all the servers
have to share between them. If one server decides to hit it really hard
(or just crashes and starts sending gibberish), all the others get
clobbered.
>> Oh, wait, I just found a real advantage: If you want to move a disk from
>> one server to another, you just press a button, rather than having to
>> physically move the device.
>>
>> Oh, well, that /totally/ makes it worth it. And, uh, when will you
>> *ever* need to perform this operation? I mean, seriously, I've been
>> working at the UK site for almost 10 years now, and I've never once
>> wanted to do this. And even if I did, if on one day in 10 years I have
>> to shut both servers down to physically move a drive, I think I can
>> probably live with that.
>
> My current customer has two data centers with 350 servers in each. "We
> need more disk space on server XYZ" happens on a daily basis, in fact,
> multiple times a day, in some cases (e.g. a fast growing log file due to
> a change in user traffic, or the activation of a debbugging feature
> during problem troubleshooting)
OK, well, perhaps if you have hundreds of servers, the massive
performance hit /might/ be justified by the advantages of being able to
keep more systems up more of the time. But for our 10 servers or so,
which are almost all up almost all of the time, the advantage seems
vanishingly small.
>> Second, what is the purpose of a backup? The purpose of having a "backup
>> copy" of your data is so that if your working copy dies somehow, you can
>> use the backup copy to get your data back.
>>
>> This only works if the backup copy is more reliable than the working
>> copy. If your working copy is on spinning disk, and you're stupid enough
>> to put your backup copy on spinning disk as well... then it becomes
>> equally likely that the /backup copy/ will die and you'll be left with
>> just the working copy.
>
> As long as your working copy doesn't disappear before your can recreate
> the backup copy, where's the problem? Are you saying that RAID is a
> waste of time because there's an equal likelyhood that any of the disks
> in the array will die?
A backup copy is supposed to be an image of the system frozen at a
particular point in time. Once that data is gone, you can't get it back.
Because that moment in time has gone.
RAID isn't the same time; it's supposed to keep the system running in
the event of a fault, not preserve historical data.
>> Third, having access only to the most recent backup is no good. There
>> are scenarios where that would be perfectly OK, but in our line of
>> business, we sometimes have to recover data from /months/ ago. Data that
>> has been accidentally deleted. Data which got corrupted. Data which we
>> thought we didn't need any more but actually we do. And so forth.
>
> Have they actually stated that you would lose the ability of going back
> to older copies of your backups?
No.
I'm not saying we will lose this ability. I'm just noting that we need
to be careful that we don't lose this ability. Simply mirroring the
disks to the SAN at HQ or whatever is not sufficient. We are legally
required to be able to restore old data.
Hopefully the guys setting this up know that. As I say, they haven't
actually decided how this is going to work yet.
>> And then there's the fact that you either need to keep all this disk
>> spinning (i.e., ever increasing power and cabling demands), or only keep
>> the recent backups spinning (i.e., managing powering off and powering on
>> drives, which supposedly shortens their lifetime).
>>
>> In all, this seems like an extremely bad idea.
>
> Have they actually stated that they would not be taking tape (or more
> than likely DVD) backups of that data for archival?
Bizarro. Why would you back up to a DVD that holds a piffling 4.2 GB of
data when a single tape holds 400 GB? That makes no sense at all.
> In all likelyhood, that remote copy will probably be backed up to
> offsite storage - if not, it should be, just for the sake of business
> continuity, but in many cases, for legal reasons as well.
Nope. The idea, apparently, is that there will be two remote copies, at
two different sites. If one dies, we've still got the other site.
They're planning to have no tape at all, just spinning disk.
>> Still, what I do know? Apparently not a lot.
>
> No offense, but I thought this was an established fact. ;)
Hey, the greatest knowledge is in knowing that you know nothing.
Remember that...
> Don't get me wrong, you are extremely knowledgeable in many areas, but
> you seem to be unable to admit that other people may have other reasons
> to do something that you deem unnecessary because it doesn't apply to
> your situation.
I'm saying that from what I can understand about this company's
situation, the proposed course of action doesn't make sense. That's all.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Le 2011-08-12 10:41, Invisible a écrit :
>> Not necessarily. It depends a lot on the configuration of the locally
>> attached drives vs. SAN logical disks, and the types of IO requests that
>> are made (e.g. long sequential reads or writes vs. lots of short random
>> file accesses). SANs have all kinds of caching mechanisms that can make
>> the IO actually faster than a SATA connection.
>
> SATA 3.0 connection: 6.0 GB/sec
>
> Fast Ethernet connection: 0.1 GB/sec
>
> That's, what, 6000x slower?
No, that's 60x times slower.
Besides, SAN connections are NOT fast ethernet. Depending on the length
and type of fibre, the speed will vary from 100 MBps to 1.6 GBps, which
is still admittedly slower than a SATA 3 controller, but unless you are
copying one large sequential file, you will also need to worry about
drive seek time and file system layout to get acutal throughput
comparisons, and Google says it's a wash.
>
> Sure, you can add caching, which might speed up write access. But it
> can't speed up read access that misses the cache.
>
>>> * Radically increased complexity compared to a direct connection.
>>
>> That depends on the point of view. From the server administrator's point
>> of view, it's much simpler. She no longer has to worry about spreading
>> her disks across multiple controllers, mirroring, etc.... on which
>> partition should the indexes go? What level of RAID to use? Is there
>> enough power in the server's power supplies to drive an additional pair
>> of disks? It's all taken care of at the other end of a pair of fibre
>> connectors.
>
> I fail to see why RAID is no longer an issue. (Unless you mean that it's
> now acceptable to lose data in the event of a hardware fault.)
RAID is an issue for the SAN administrator, not the server
administrator. The server admin gets assigned one (or more) virtual
drive(s) and can safely assume that the data integrity will be
maintained. The SAN administrator will make sure that the logical drive
has been allocated out of properly configured pools of RAIDs.
>
> I've yet to see any system where power consumption was even remotely
> important. Servers seem to come with power supplies that can handle
> *way* more current than you will ever need.
>
Usually, yes, but not always.
> I've also never seen a sever with more than one disk controller, come to
> mention it...
>
Then your servers are far from optimal. In my work, It's not uncommon
to see servers with 3 or 4 controllers.
1 for the OS, with its own set of RAID drives.
1 for the apps with its own set of RAID drives. (you don't want log
file or library file I/O to interfere with DB performance)
1 or more for the database, with its own sets of drives. The indexes
and redo logs all on different drives than the tables, to make sure that
you don't cause bottlenecks by having the drive heads jumping from the
index to the table back to the index...
With the exception of the OS drive (You don't want to boot off the SAN),
all of those* can be sent to the SAN for where they will be spread
across many disks and controllers for an optimal performance.
*Don't run your domain controllers or DNSes from a SAN either.
>> From the point of view of the backup administrator, it's much simpler
>> since he no longer has to worry about making sure that the backup agent
>> is running on all the servers and that their schedules don't actually
>> step over each other's toes, etc... it can all be done off-line in the
>> background.
>
> I don't see how that works.
The DVD or tape drives have direct backplane connections to the disk
arrays so they can read the disks without impacting the network or
competing with the server's own IO requests.
Just like a hardware RAID controller can rebuild a failed drive without
the any noticeable performance degradation.
>
>> You just had a very weird problem? Just clone the DB to an new set of
>> disks, and assign those on a lab server so that the developpers can work
>> on the issue all weekend.
>
> You can trivially do that without a SAN. Disk imaging tools have existed
> for decades. Either way, you still have to take the DB server offline
> while you do this.
No. That's the point. The SAN administrator simply mirrors the disks
in the background, without the server even knowing about it.
>
> Of course, if your DB server is a VM, the problem just became even more
> trivial. Depending on your virtualisation technology, maybe you don't
> even have to stop the server to copy the disk image. And you /still/
> don't need a SAN.
>
>> The biggest added level of complexity is that all these people have to
>> talk to one another. Scary. I know.
>
> Nah. Our entire IT function (including me) is about half a dozen humans.
>
>>> * If the storage network fails, ALL servers fail, so you're adding a new
>>> single point of failure.
>>
>> True, but that rarely happens.
>
> OK, fair enough.
>
>>> * All the servers now have to share the limited bandwidth available. One
>>> busy server can bring all the others to a crawl.
>>
>> What limited bandwidth? Even a SAN that's been improperly designed, and
>> that has seen 5 years of unchecked growth will have more than enough
>> bandwitdh to accomodate all the servers. Have you seen the
>> specifications of the SAN your head office wants to implement and were
>> able to determine there will be bottlenecks?
>
> My point is, if you have 10 servers, each with their own dedicated SATA
> connection (or whatever), you have 10 lots of bandwidth. And if one
> server decides to hog it all, it can't slow the others down.
>
> If you have a SAN, you have /one/ access channel, which all the servers
> have to share between them. If one server decides to hit it really hard
> (or just crashes and starts sending gibberish), all the others get
> clobbered.
Again, wrong. Each server has dedicated access to the SAN fabric and
the SAN switches will dispatch I/O requests to the proper drive(s)
without impacting other servers' I/O requests.
>
>>> Oh, wait, I just found a real advantage: If you want to move a disk from
>>> one server to another, you just press a button, rather than having to
>>> physically move the device.
>>>
>>> Oh, well, that /totally/ makes it worth it. And, uh, when will you
>>> *ever* need to perform this operation? I mean, seriously, I've been
>>> working at the UK site for almost 10 years now, and I've never once
>>> wanted to do this. And even if I did, if on one day in 10 years I have
>>> to shut both servers down to physically move a drive, I think I can
>>> probably live with that.
>>
>> My current customer has two data centers with 350 servers in each. "We
>> need more disk space on server XYZ" happens on a daily basis, in fact,
>> multiple times a day, in some cases (e.g. a fast growing log file due to
>> a change in user traffic, or the activation of a debbugging feature
>> during problem troubleshooting)
>
> OK, well, perhaps if you have hundreds of servers, the massive
> performance hit /might/ be justified by the advantages of being able to
> keep more systems up more of the time. But for our 10 servers or so,
> which are almost all up almost all of the time, the advantage seems
> vanishingly small.
>
>>> Second, what is the purpose of a backup? The purpose of having a "backup
>>> copy" of your data is so that if your working copy dies somehow, you can
>>> use the backup copy to get your data back.
>>>
>>> This only works if the backup copy is more reliable than the working
>>> copy. If your working copy is on spinning disk, and you're stupid enough
>>> to put your backup copy on spinning disk as well... then it becomes
>>> equally likely that the /backup copy/ will die and you'll be left with
>>> just the working copy.
>>
>> As long as your working copy doesn't disappear before your can recreate
>> the backup copy, where's the problem? Are you saying that RAID is a
>> waste of time because there's an equal likelyhood that any of the disks
>> in the array will die?
>
> A backup copy is supposed to be an image of the system frozen at a
> particular point in time. Once that data is gone, you can't get it back.
> Because that moment in time has gone.
>
> RAID isn't the same time; it's supposed to keep the system running in
> the event of a fault, not preserve historical data.
>
>>> Third, having access only to the most recent backup is no good. There
>>> are scenarios where that would be perfectly OK, but in our line of
>>> business, we sometimes have to recover data from /months/ ago. Data that
>>> has been accidentally deleted. Data which got corrupted. Data which we
>>> thought we didn't need any more but actually we do. And so forth.
>>
>> Have they actually stated that you would lose the ability of going back
>> to older copies of your backups?
>
> No.
>
> I'm not saying we will lose this ability. I'm just noting that we need
> to be careful that we don't lose this ability. Simply mirroring the
> disks to the SAN at HQ or whatever is not sufficient. We are legally
> required to be able to restore old data.
>
> Hopefully the guys setting this up know that. As I say, they haven't
> actually decided how this is going to work yet.
>
>>> And then there's the fact that you either need to keep all this disk
>>> spinning (i.e., ever increasing power and cabling demands), or only keep
>>> the recent backups spinning (i.e., managing powering off and powering on
>>> drives, which supposedly shortens their lifetime).
>>>
>>> In all, this seems like an extremely bad idea.
>>
>> Have they actually stated that they would not be taking tape (or more
>> than likely DVD) backups of that data for archival?
>
> Bizarro. Why would you back up to a DVD that holds a piffling 4.2 GB of
> data when a single tape holds 400 GB? That makes no sense at all.
>
Tapes degrade over time faster than DVDs do.
I/O is sequential on a tape. If you need a specific file, you need to
read through the whole tape from the beginning.
If you only have a 10GB incremental per day, and need to sent it to
offsite storage every day (sometimes required by law), using 400GB tape
is a serious waste of money.
Each application has its preferred solution. Your mileage may vary.
>> In all likelyhood, that remote copy will probably be backed up to
>> offsite storage - if not, it should be, just for the sake of business
>> continuity, but in many cases, for legal reasons as well.
>
> Nope. The idea, apparently, is that there will be two remote copies, at
> two different sites. If one dies, we've still got the other site.
> They're planning to have no tape at all, just spinning disk.
>
That _is_ silly.
--
/*Francois Labreque*/#local a=x+y;#local b=x+a;#local c=a+b;#macro P(F//
/* flabreque */L)polygon{5,F,F+z,L+z,L,F pigment{rgb 9}}#end union
/* @ */{P(0,a)P(a,b)P(b,c)P(2*a,2*b)P(2*b,b+c)P(b+c,<2,3>)
/* gmail.com */}camera{orthographic location<6,1.25,-6>look_at a }
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> SATA 3.0 connection: 6.0 GB/sec
>>
>> Fast Ethernet connection: 0.1 GB/sec
>>
>> That's, what, 6000x slower?
>
> No, that's 60x times slower.
Epic math fail. :'{
> Besides, SAN connections are NOT fast ethernet. Depending on the length
> and type of fibre, the speed will vary from 100 MBps to 1.6 GBps, which
> is still admittedly slower than a SATA 3 controller, but unless you are
> copying one large sequential file, you will also need to worry about
> drive seek time and file system layout to get acutal throughput
> comparisons, and Google says it's a wash.
I rather doubt that drive seek time is longer than network latency. I
also rather doubt that you can't saturate a single network link with
half a dozen drives.
>> I fail to see why RAID is no longer an issue. (Unless you mean that it's
>> now acceptable to lose data in the event of a hardware fault.)
>
> RAID is an issue for the SAN administrator, not the server
> administrator.
Oh, I see.
This applies only if the SAN administrator and the server administrator
aren't the same person. In our setup, with a grand total of 4 IT staff,
I doubt this is relevant.
Like I said, increased complexity.
>> I've also never seen a sever with more than one disk controller, come to
>> mention it...
>
> Then your servers are far from optimal.
Well, I guess you can /always/ have more fault tolerance. But in 10
years of working here, I've only seen one controller fail. Come to think
of it, I've only seen about 6 drives fail (including desktop PCs and
laptops).
Currently, I have two servers. One has 2 disks, the other has 3 active
disks plus a hot spare. I doubt adding more controllers is worth the effort.
> With the exception of the OS drive (You don't want to boot off the SAN),
> *Don't run your domain controllers or DNSes from a SAN either.
Any specific reason?
>> I don't see how that works.
>
> The DVD or tape drives have direct backplane connections to the disk
> arrays so they can read the disks without impacting the network or
> competing with the server's own IO requests.
So how does it obtain a consistent image of the filesystem?
> Just like a hardware RAID controller can rebuild a failed drive without
> the any noticeable performance degradation.
[RAID does it by constantly resynchronising.]
>>> You just had a very weird problem? Just clone the DB to an new set of
>>> disks, and assign those on a lab server so that the developpers can work
>>> on the issue all weekend.
>>
>> You can trivially do that without a SAN. Disk imaging tools have existed
>> for decades. Either way, you still have to take the DB server offline
>> while you do this.
>
> No. That's the point. The SAN administrator simply mirrors the disks in
> the background, without the server even knowing about it.
Like I say, I don't see how you can obtain a consistent image of the
filesystem without cooperation from the server.
>> My point is, if you have 10 servers, each with their own dedicated SATA
>> connection (or whatever), you have 10 lots of bandwidth. And if one
>> server decides to hog it all, it can't slow the others down.
>>
>> If you have a SAN, you have /one/ access channel, which all the servers
>> have to share between them. If one server decides to hit it really hard
>> (or just crashes and starts sending gibberish), all the others get
>> clobbered.
>
> Again, wrong. Each server has dedicated access to the SAN fabric and the
> SAN switches will dispatch I/O requests to the proper drive(s) without
> impacting other servers' I/O requests.
There is one network switch. All data must pass through this single
device. If it only processes X bytes per second, than all the servers
between them cannot access more than X bytes per second. The end.
Unless every single server and several single disk has its own dedicated
channel to the switch (which is presumably infeasible), there will be
additional bottlenecks for components sharing channels too.
>>>> In all, this seems like an extremely bad idea.
>>>
>>> Have they actually stated that they would not be taking tape (or more
>>> than likely DVD) backups of that data for archival?
>>
>> Bizarro. Why would you back up to a DVD that holds a piffling 4.2 GB of
>> data when a single tape holds 400 GB? That makes no sense at all.
>
> Tapes degrade over time faster than DVDs do.
In which universe? I would have said the exact opposite was true. I've
seen CDs and DVDs burned 6 months ago where the ink has faded to the
point where the disk is utterly unusable. I've yet to see a tape fail to
read.
> I/O is sequential on a tape. If you need a specific file, you need to
> read through the whole tape from the beginning.
That's true though. Fortunately, file restoration is a rare event, so it
doesn't matter too much.
> If you only have a 10GB incremental per day, and need to sent it to
> offsite storage every day (sometimes required by law), using 400GB tape
> is a serious waste of money.
The way we do it is to store all the tapes off-site, except today's
tape. Rotate the tapes until they become full, then add in a new tape.
All 400GB of capacity gets used, and yet at any time the majority of
tapes are off-site.
> Each application has its preferred solution. Your mileage may vary.
Fair enough. I'm just saying, for our setup, this wouldn't be a good idea.
>>> In all likelyhood, that remote copy will probably be backed up to
>>> offsite storage - if not, it should be, just for the sake of business
>>> continuity, but in many cases, for legal reasons as well.
>>
>> Nope. The idea, apparently, is that there will be two remote copies, at
>> two different sites. If one dies, we've still got the other site.
>> They're planning to have no tape at all, just spinning disk.
>
> That _is_ silly.
...which is what I said a week ago. ;-)
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Le 2011-08-12 11:55, Invisible a écrit :
>>> I fail to see why RAID is no longer an issue. (Unless you mean that it's
>>> now acceptable to lose data in the event of a hardware fault.)
>>
>> RAID is an issue for the SAN administrator, not the server
>> administrator.
>
> Oh, I see.
>
> This applies only if the SAN administrator and the server administrator
> aren't the same person. In our setup, with a grand total of 4 IT staff,
> I doubt this is relevant.
This is why I said it would be silly to have a SAN for only a handful of
devices. In a data center with 200+ servers, you'll usually end up with
entirely different support teams for Mainframe, mid-range (AS/400,
Tandem, etc...), Unix (sometimes different groups for different flavours
of), Windows, and Netware (if there are still some of it), having a
group of 4 or 5 people dedicated to the SAN administration allows them
to specialize in their field, and hopefully become more competent at it,
and leaving the OS folks deal with patching, tuning processes, etc...
>>> I've also never seen a sever with more than one disk controller, come to
>>> mention it...
>>
>> Then your servers are far from optimal.
>
> Well, I guess you can /always/ have more fault tolerance. But in 10
> years of working here, I've only seen one controller fail. Come to think
> of it, I've only seen about 6 drives fail (including desktop PCs and
> laptops).
Not fault tolerance. PERFORMANCE. You know, that thing you were saying
would degrade with a SAN.
>> With the exception of the OS drive (You don't want to boot off the SAN),
>
>> *Don't run your domain controllers or DNSes from a SAN either.
>
> Any specific reason?
>
Because in the event that the SAN went away, you may end up in a
situation where the DNS or AD is actually required to allow support
personnel to work on the issue (think VPN access for remote support
personnel). As much as you want to, in an environment with hundreds of
servers, you're bound to end up with either having to rely on DNS
services or deal with updating hundreds of local HOSTS files, etc.
>>> I don't see how that works.
>>
>> The DVD or tape drives have direct backplane connections to the disk
>> arrays so they can read the disks without impacting the network or
>> competing with the server's own IO requests.
>
> So how does it obtain a consistent image of the filesystem?
>
The same way you do with a local backup agent. The SAN is not a magical
invention that removes all worldly constraints. In other words, you
schedule the backup at the time where its least likely there will be
changes to the file system, and pray to the deity of your choice.
In other words, in a data center for a multinational corporation where
planes are up in the air, or trading is going on in a stock exchange
somewhere 24 hrs a day, 7 days a week, you NEVER have a consistent copy.
You simply hope to have a GOOD ENOUGH copy.
For systems that require a "snapshot" of the state at specific times,
you schedule downtime at the application level and take the backup
during that time, for that application. For example, at one of my
former customers, the HR/Finance ERP system would switch to read-only
mode from 0h00 to 06h00, every day to allow consistent backups to be
taken. This allowed an accountant burning the midnight oil to be able
to still read the general ledger, while allowing for a clean backup to
be taken.
>>>
>>> If you have a SAN, you have /one/ access channel, which all the servers
>>> have to share between them. If one server decides to hit it really hard
>>> (or just crashes and starts sending gibberish), all the others get
>>> clobbered.
>>
>> Again, wrong. Each server has dedicated access to the SAN fabric and the
>> SAN switches will dispatch I/O requests to the proper drive(s) without
>> impacting other servers' I/O requests.
>
> There is one network switch. All data must pass through this single
> device. If it only processes X bytes per second, than all the servers
> between them cannot access more than X bytes per second. The end.
There are multiple SAN switches (you definitely DON'T want a single
point of failure). Each with enough backplane bandwidth, processor
power and ports to handle I/O requests at wire speed.
A typical set-up would have each server connected to two different SAN
switches (sometimes refered to as "front-end" switches), and these would
in turn be connected to each of the disk bays, on top of having
interconnections between the switches, to prevent a fibre cut from
shutting a part of the storage network down. Said typical set-up would
also probably have performance monitoring software available allowing
the SAN administrators to see where potential bottelnecks will be and
take approriate measure during a scheduled maintenance window.
>
> Unless every single server and several single disk has its own dedicated
> channel to the switch (which is presumably infeasible), there will be
> additional bottlenecks for components sharing channels too.
>
You have presumed wrong. This is exactly what happens.
>>>> Have they actually stated that they would not be taking tape (or more
>>>> than likely DVD) backups of that data for archival?
>>>
>>> Bizarro. Why would you back up to a DVD that holds a piffling 4.2 GB of
>>> data when a single tape holds 400 GB? That makes no sense at all.
>>
>> Tapes degrade over time faster than DVDs do.
>
> In which universe? I would have said the exact opposite was true. I've
> seen CDs and DVDs burned 6 months ago where the ink has faded to the
> point where the disk is utterly unusable. I've yet to see a tape fail to
> read.
Stop buying cheap DVDs then ;). I've seen tapes fail in drives.
<strike>I've seen attack ships fire off the shoulder of Orion</strike>,
I've seen tapes where someone had to actually change the plastic
leader thingie because it had become so mangled from repeated use that
it wouldn't engage in the drive any more. Etc...
>
>> I/O is sequential on a tape. If you need a specific file, you need to
>> read through the whole tape from the beginning.
>
> That's true though. Fortunately, file restoration is a rare event, so it
> doesn't matter too much.
Again that depends on the application. Most large corporations will
have data exchange that still takes place over physical media being
FEDEXed (interbank transfers, payroll data for the bank that issues the
cheques, airline schedule exchanges, government regulatory filings,
etc.) as in some cases, it is still cheaper than setting up dedicated
inter-entreprise connections. Some backups are only for business
recovery, but others are intended for use in the very near future.
>
>> If you only have a 10GB incremental per day, and need to sent it to
>> offsite storage every day (sometimes required by law), using 400GB tape
>> is a serious waste of money.
>
> The way we do it is to store all the tapes off-site, except today's
> tape. Rotate the tapes until they become full, then add in a new tape.
> All 400GB of capacity gets used, and yet at any time the majority of
> tapes are off-site.
>
See above. You can't always reuse the same tape. There may ver Very
Good(tm) reasons for that.
Also, I know that multiple backups can be combined on a tape. This is
exactly what we used to do for the file servers, when I did server
support 15 years ago. All incrementals go on rotating tapes, and the
weekly full backup is done on a different tape, which was then
duplicated and one copy was sent to the offsite vault. However, even in
this scenario, if one of the rotating tapes got damaged or lost, the
missing incrementals could still cause major legal woes for the company.
I've since moved away from server support, so I can't comment on what
the storage admins do nowadays. All I know if they have a really cool
robot that fetches tapes and DVDs from the racks instead of relying on
"tape monkeys" as we used to call them.
--
/*Francois Labreque*/#local a=x+y;#local b=x+a;#local c=a+b;#macro P(F//
/* flabreque */L)polygon{5,F,F+z,L+z,L,F pigment{rgb 9}}#end union
/* @ */{P(0,a)P(a,b)P(b,c)P(2*a,2*b)P(2*b,b+c)P(b+c,<2,3>)
/* gmail.com */}camera{orthographic location<6,1.25,-6>look_at a }
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> Oh, I see.
>>
>> This applies only if the SAN administrator and the server administrator
>> aren't the same person. In our setup, with a grand total of 4 IT staff,
>> I doubt this is relevant.
>
> This is why I said it would be silly to have a SAN for only a handful of
> devices.
Fair enough.
>> Well, I guess you can /always/ have more fault tolerance. But in 10
>> years of working here, I've only seen one controller fail. Come to think
>> of it, I've only seen about 6 drives fail (including desktop PCs and
>> laptops).
>
> Not fault tolerance. PERFORMANCE. You know, that thing you were saying
> would degrade with a SAN.
How does adding more controllers increase performance? The controller is
usually the fastest component in the system. The bottleneck is usually
either the bus to the CPU, or the connections to the HDs.
>>> The DVD or tape drives have direct backplane connections to the disk
>>> arrays so they can read the disks without impacting the network or
>>> competing with the server's own IO requests.
>>
>> So how does it obtain a consistent image of the filesystem?
>
> The same way you do with a local backup agent. The SAN is not a magical
> invention that removes all worldly constraints. In other words, you
> schedule the backup at the time where its least likely there will be
> changes to the file system, and pray to the deity of your choice.
That's /not/ how a local backup agent works. Indeed, the entire
/purpose/ of a local backup agent is to construct a consistent image (by
whatever means that requires) to write to tape.
> In other words, in a data center for a multinational corporation where
> planes are up in the air, or trading is going on in a stock exchange
> somewhere 24 hrs a day, 7 days a week, you NEVER have a consistent copy.
> You simply hope to have a GOOD ENOUGH copy.
Nonsense. If you're doing something like that, you're probably using a
relational database. And any half-decent product of that type provides a
way to construct a consistent image without ever stopping the database.
(You do, however, need the cooperation of the server that's running the
thing. You can't just take a block-level image of the filesystem and
expect to get anything sensible out of it.)
>> There is one network switch. All data must pass through this single
>> device. If it only processes X bytes per second, than all the servers
>> between them cannot access more than X bytes per second. The end.
>
> There are multiple SAN switches (you definitely DON'T want a single
> point of failure). Each with enough backplane bandwidth, processor power
> and ports to handle I/O requests at wire speed.
So what you're saying is that you wire up two independent networks, and
connect every device to both of them, so that if the switch running
network A fails, you can use network B instead?
>> Unless every single server and every single disk has its own dedicated
>> channel to the switch (which is presumably infeasible), there will be
>> additional bottlenecks for components sharing channels too.
>
> You have presumed wrong. This is exactly what happens.
Damn. This stuff is more expensive than I thought... Still, great way to
make your department look important, I guess.
>>> Tapes degrade over time faster than DVDs do.
>>
>> In which universe? I would have said the exact opposite was true. I've
>> seen CDs and DVDs burned 6 months ago where the ink has faded to the
>> point where the disk is utterly unusable. I've yet to see a tape fail to
>> read.
>
> Stop buying cheap DVDs then ;)
To be fair, this seems to have become less of a problem has CD-R
technology has improved over the years. Much like anything else, I imagine.
> I've seen tapes fail in drives.
> I've seen tapes where someone had to actually change the plastic leader
> thingie because it had become so mangled from repeated use that it
> wouldn't engage in the drive any more. Etc...
Hell, one time I tried to destroy a tape on purpose and I actually
couldn't do it. We soaked the tape in lab chemicals and it wouldn't
degrade, I tried to cut it up but it was too indestructible, it was
really, really hard to get rid of this stuff. (Although probably a lot
easier to just make the drive not read it any more.)
>>> I/O is sequential on a tape. If you need a specific file, you need to
>>> read through the whole tape from the beginning.
>>
>> That's true though. Fortunately, file restoration is a rare event, so it
>> doesn't matter too much.
>
> Again that depends on the application.
OK, for /our/ application, it's a rare event. And that's really what I'm
interested in. Does this technology make any sense at all for us? From
what I can see, no, it doesn't.
> I've since moved away from server support, so I can't comment on what
> the storage admins do nowadays. All I know if they have a really cool
> robot that fetches tapes and DVDs from the racks instead of relying on
> "tape monkeys" as we used to call them.
I have a small tape robot. If I wanted to, I could load a year's worth
of tapes into it and never have to change tape until next year.
(Unfortunately, I can't actually do that, because then none of the tapes
would be "off site".) It even has a cute little barcode reader.
That was HQ's idea. They don't often have good ideas, but this was one
of them. ;-)
Now of course, they're talking about taking that away and using our slow
Internet connection to do the same job... *sigh*
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Le 2011-08-15 09:29, Invisible a écrit :
>>>
>>> So how does it obtain a consistent image of the filesystem?
>>
>> The same way you do with a local backup agent. The SAN is not a magical
>> invention that removes all worldly constraints. In other words, you
>> schedule the backup at the time where its least likely there will be
>> changes to the file system, and pray to the deity of your choice.
>
> That's /not/ how a local backup agent works. Indeed, the entire
> /purpose/ of a local backup agent is to construct a consistent image (by
> whatever means that requires) to write to tape.
No. the purpose of a local backup agent is:
1) to have the means to back the files up at shceduled intervals, and
2) talk to the backup unit (be it another locally-mounted file system,
removable storage media, or file transfer over a network).
Both of these functions are nowadays available directly from the OS, but
most entreprises will have a separate tool to do this as there are other
advantages when it comes to centralized management, or the ability to
lock access to certain files or directories, or able to speak in a
dialect understood by the application (e.g. Oracle "live backup" agent),
but in many cases, it is not necessary and the backup job can run while
the users access their files. At that point, it becomes a management
decision based on the relative costs penalty of scheduling down time vs.
losing absolute file consistency.
>
>> In other words, in a data center for a multinational corporation where
>> planes are up in the air, or trading is going on in a stock exchange
>> somewhere 24 hrs a day, 7 days a week, you NEVER have a consistent copy.
>> You simply hope to have a GOOD ENOUGH copy.
>
> Nonsense. If you're doing something like that, you're probably using a
> relational database. And any half-decent product of that type provides a
> way to construct a consistent image without ever stopping the database.
> (You do, however, need the cooperation of the server that's running the
> thing. You can't just take a block-level image of the filesystem and
> expect to get anything sensible out of it.)
>
Agreed for databases, but even just a source tree could become
inconsistent if a programmer is making changes at the same time as the
backup job is running. As I said, if it absolutely, positiviely,
legally requires a consistent copy, means will be taken to achieve that.
But if it's just a minor inconvenience for a few persons to have an
older draft version of their weekly powerpoint presentation to upper
management having been saved prior to the server failure, then so be it.
>>> There is one network switch. All data must pass through this single
>>> device. If it only processes X bytes per second, than all the servers
>>> between them cannot access more than X bytes per second. The end.
>>
>> There are multiple SAN switches (you definitely DON'T want a single
>> point of failure). Each with enough backplane bandwidth, processor power
>> and ports to handle I/O requests at wire speed.
>
> So what you're saying is that you wire up two independent networks, and
> connect every device to both of them, so that if the switch running
> network A fails, you can use network B instead?
The storage network hardware is all interconnected, but the storage
network is independant from the user-traffic network.
----------
/ \
| IP Network |
\ /
-----------
/ / \ \
-------- --------
| server | | server |
| 1 | | 2 | Server N, etc...
-------- --------
| \ / | /
| x | /
| / \ | /
-------- --------
| SAN sw |---| SAN sw |--- Switch N, etc...
| 1 |---| 2 |---
-------- --------
| \ / |
| x |
| / \ |
-------- --------
| Drive | | Drive | --> Repeat if uptime service
| bay | | bay | level agreements require
| | | | site-level diversity, or if
| 1 | | 2 | required by capacity.
-------- --------
\ /
---------
| Tape |
| Robot |
---------
Typicially, the SAN switches will have up to 16 or so "front end" ports
and 2 or more higher-speed "Back-end" ports to the drive bays. Some
high-load servers might have direct connections to the drive bays.
>
>>> Unless every single server and every single disk has its own dedicated
>>> channel to the switch (which is presumably infeasible), there will be
>>> additional bottlenecks for components sharing channels too.
>>
>> You have presumed wrong. This is exactly what happens.
>
> Damn. This stuff is more expensive than I thought... Still, great way to
> make your department look important, I guess.
>
It's the only way to achieve "five 9s" uptime (i.e. 99.999%), which is
what most banks and multinationals require. Again, for a small to
medium shop, with a dozen to 50 servers or so servers, or no
requirements for such high SLAs, then it's overkill.
It's worthwhile to note that five-9s uptime also require considerable
thoughts to the choice of applications, hardware, network design,
climate control, power management, staffing arrangements, telco
diversity, disaster prevention and business continuity, in the event
that a disaster can not be avoided. This can add one or two orders of
magnitude to a company's IT budget, so it's not just to make your dept.
look important. There has to be a sufficient increase in revenues (or
decrease in losses!) to justify it.
--
/*Francois Labreque*/#local a=x+y;#local b=x+a;#local c=a+b;#macro P(F//
/* flabreque */L)polygon{5,F,F+z,L+z,L,F pigment{rgb 9}}#end union
/* @ */{P(0,a)P(a,b)P(b,c)P(2*a,2*b)P(2*b,b+c)P(b+c,<2,3>)
/* gmail.com */}camera{orthographic location<6,1.25,-6>look_at a }
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>>>> So how does it obtain a consistent image of the filesystem?
>>>
>>> The same way you do with a local backup agent. The SAN is not a magical
>>> invention that removes all worldly constraints. In other words, you
>>> schedule the backup at the time where its least likely there will be
>>> changes to the file system, and pray to the deity of your choice.
>>
>> That's /not/ how a local backup agent works. Indeed, the entire
>> /purpose/ of a local backup agent is to construct a consistent image (by
>> whatever means that requires) to write to tape.
>
> No. the purpose of a local backup agent is:
> 1) to have the means to back the files up at shceduled intervals, and
> 2) talk to the backup unit (be it another locally-mounted file system,
> removable storage media, or file transfer over a network).
>
> Both of these functions are nowadays available directly from the OS, but
> most entreprises will have a separate tool to do this as there are other
> advantages when it comes to centralized management, or the ability to
> lock access to certain files or directories, or able to speak in a
> dialect understood by the application (e.g. Oracle "live backup" agent),
> but in many cases, it is not necessary and the backup job can run while
> the users access their files. At that point, it becomes a management
> decision based on the relative costs penalty of scheduling down time vs.
> losing absolute file consistency.
OK, first of all, let me split two things out:
First, we have consistency of the physical file system itself.
Second, we have consistency of the logical files contained on disk.
If the FS becomes inconsistent, usually that renders it completely
unreadable without advanced recovery tools. An unreadable backup is useless.
If a logical file becomes inconsistent, that might just mean that Word
crashes when you try to open your document. So the backup copy of that
document is useless, but other documents on the disk are probably usable.
Either way, the point of having a remote backup agent is to prevent
these kinds of inconsistencies. Firstly, by accessing the file system
through the OS, you guarantee that what you see is a consistent image of
the file system. Secondly, I know on Windows you can do things like
Shadow Copy, which guarantees a consistent image of all logical files,
even if people are editing them while you back them up.
It's completely possible to back up remote disks without installing a
remote agent. But nobody does this, because the file consistency issues
are too bad.
(And yes, then there are other advantages. A local backup agent can use
the NTFS journal to work out which files have changed recently for
incremental backups. It can compress data before sending it over the
network. And so forth.)
> As I said, if it absolutely, positiviely, legally
> requires a consistent copy, means will be taken to achieve that. But if
> it's just a minor inconvenience for a few persons to have an older draft
> version of their weekly powerpoint presentation to upper management
> having been saved prior to the server failure, then so be it.
An older draft isn't so bad. When the file becomes completely
unreadable, that generally upsets people.
>> So what you're saying is that you wire up two independent networks, and
>> connect every device to both of them, so that if the switch running
>> network A fails, you can use network B instead?
>
> The storage network hardware is all interconnected, but the storage
> network is independant from the user-traffic network.
Interesting that you've drawn the tape robot at the opposite end of the
SAN. Usually that's /controlled by/ one of the servers.
> Typicially, the SAN switches will have up to 16 or so "front end" ports
> and 2 or more higher-speed "Back-end" ports to the drive bays. Some
> high-load servers might have direct connections to the drive bays.
I still find it hard to believe that a shared fabric can come anywhere
close to the performance of a dedicated fabric.
>>>> Unless every single server and every single disk has its own dedicated
>>>> channel to the switch (which is presumably infeasible), there will be
>>>> additional bottlenecks for components sharing channels too.
>>>
>>> You have presumed wrong. This is exactly what happens.
>>
>> Damn. This stuff is more expensive than I thought... Still, great way to
>> make your department look important, I guess.
>
> It's the only way to achieve "five 9s" uptime (i.e. 99.999%), which is
> what most banks and multinationals require. Again, for a small to medium
> shop, with a dozen to 50 servers or so servers, or no requirements for
> such high SLAs, then it's overkill.
So people really are willing to sacrifice huge amounts of performance
just for increased uptime?
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 8/15/2011 6:04, Francois Labreque wrote:
> you schedule
> the backup at the time where its least likely there will be changes to the
> file system, and pray to the deity of your choice.
You don't need to do that. You just make a virtual snapshot of the drive
you're backing up and back that up.
> In other words, in a data center for a multinational corporation where
> planes are up in the air, or trading is going on in a stock exchange
> somewhere 24 hrs a day, 7 days a week, you NEVER have a consistent copy.
You can. That's exactly what Windows VSS is for. Linux has something almost
as sophisticated on some of its systems.
> For systems that require a "snapshot" of the state at specific times, you
> schedule downtime at the application level and take the backup during that
> time, for that application.
You usually want to do this for upgrades and such also. Having a fixed
schedule of downtime is useful for all kinds of stuff other than just taking
backups.
--
Darren New, San Diego CA, USA (PST)
How come I never get only one kudo?
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Le 2011-08-16 10:29, Invisible a écrit :
>> As I said, if it absolutely, positiviely, legally
>> requires a consistent copy, means will be taken to achieve that. But if
>> it's just a minor inconvenience for a few persons to have an older draft
>> version of their weekly powerpoint presentation to upper management
>> having been saved prior to the server failure, then so be it.
>
> An older draft isn't so bad. When the file becomes completely
> unreadable, that generally upsets people.
>
That depends on the people and the amount of changes that occured
between saves.
>
> Interesting that you've drawn the tape robot at the opposite end of the
> SAN. Usually that's /controlled by/ one of the servers.
>
I didn't draw all the servers. Secondly, the tape robot would be
controlled by a dedicated servers that did nothing else than control the
back up schedules and keep the inventory of what is backed up where.
>>
>> It's the only way to achieve "five 9s" uptime (i.e. 99.999%), which is
>> what most banks and multinationals require. Again, for a small to medium
>> shop, with a dozen to 50 servers or so servers, or no requirements for
>> such high SLAs, then it's overkill.
>
> So people really are willing to sacrifice huge amounts of performance
> just for increased uptime?
You have that wrong. First, under normal conditions SANs perform at the
approximately the same level or better than dedicated disks*.
Secondly, the increase in the number of servers required to achieve
these levels of uptime will spread the users, reducing the strain on
each server and increasing overall performance of the system. Finally,
as I said, Five-9 uptime is a change in mentality. These corporations
will usually also have more performance measuring tools** to prove that
they are indeed reaching these service levels, and more troubleshooting
tools available to fix a problem fast. So in their quest to achieve
these levels of uptime, people will often find hidden problems in
applications that would have been very difficult to find if they were
running on dedicated hardware.
*http://www.sqlteam.com/article/which-is-faster-san-or-directly-attached-storage
**That's what is currently putting food on my table.
--
/*Francois Labreque*/#local a=x+y;#local b=x+a;#local c=a+b;#macro P(F//
/* flabreque */L)polygon{5,F,F+z,L+z,L,F pigment{rgb 9}}#end union
/* @ */{P(0,a)P(a,b)P(b,c)P(2*a,2*b)P(2*b,b+c)P(b+c,<2,3>)
/* gmail.com */}camera{orthographic location<6,1.25,-6>look_at a }
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 03/08/2011 03:44 PM, Invisible wrote:
> First, it appears that our head office is replacing their current
> arrangement with a big new SAN.
> Fortunately, this only affects HQ, so in a sense I don't have to care
> about this. Even so, it still baffles me.
WRONG!
Apparently they changed their minds. Each site will have their own SAN.
Looks like the plan is for each site to have their own SAN, and any
changes to the data on that SAN gets replicated to HQ and a second site
in the background.
So /that/ should nail our Internet connection nicely then...
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
|
|
