 |
|
|
|
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> * Tens of thousands of times more expensive than a direct connection.
>
> For a small or remote office with a couple file-servers, I agree that it
> doesn't make sense but for a data center, it's a different story. If you
> have 200 servers connected to the SAN, the "unit cost" of the SAN
> hardware becomes a lot more palatable, and you can leverage unused disk
> space that nay servers will have.
AFAIK, our data centre has about 10 physical servers. (And a slightly
larger number of virtual servers.)
They're talking about increasing the number of virtual servers (possibly
making /every/ end-user server virtual), which makes being able to swap
disks around a more attractive proposition. If physical server X fails,
move all the disks to physical server Y and start all the VMs back up.
I'm still not convinced we need to spend 150,000 USD just to make it
easier to move disks though.
>> * Hundreds of thousands of times reduced performance compared to a
>> direct connection.
>
> Not necessarily. It depends a lot on the configuration of the locally
> attached drives vs. SAN logical disks, and the types of IO requests that
> are made (e.g. long sequential reads or writes vs. lots of short random
> file accesses). SANs have all kinds of caching mechanisms that can make
> the IO actually faster than a SATA connection.
SATA 3.0 connection: 6.0 GB/sec
Fast Ethernet connection: 0.1 GB/sec
That's, what, 6000x slower?
Sure, you can add caching, which might speed up write access. But it
can't speed up read access that misses the cache.
>> * Radically increased complexity compared to a direct connection.
>
> That depends on the point of view. From the server administrator's point
> of view, it's much simpler. She no longer has to worry about spreading
> her disks across multiple controllers, mirroring, etc.... on which
> partition should the indexes go? What level of RAID to use? Is there
> enough power in the server's power supplies to drive an additional pair
> of disks? It's all taken care of at the other end of a pair of fibre
> connectors.
I fail to see why RAID is no longer an issue. (Unless you mean that it's
now acceptable to lose data in the event of a hardware fault.)
I've yet to see any system where power consumption was even remotely
important. Servers seem to come with power supplies that can handle
*way* more current than you will ever need.
I've also never seen a sever with more than one disk controller, come to
mention it...
> From the point of view of the backup administrator, it's much simpler
> since he no longer has to worry about making sure that the backup agent
> is running on all the servers and that their schedules don't actually
> step over each other's toes, etc... it can all be done off-line in the
> background.
I don't see how that works.
> You just had a very weird problem? Just clone the DB to an new set of
> disks, and assign those on a lab server so that the developpers can work
> on the issue all weekend.
You can trivially do that without a SAN. Disk imaging tools have existed
for decades. Either way, you still have to take the DB server offline
while you do this.
Of course, if your DB server is a VM, the problem just became even more
trivial. Depending on your virtualisation technology, maybe you don't
even have to stop the server to copy the disk image. And you /still/
don't need a SAN.
> The biggest added level of complexity is that all these people have to
> talk to one another. Scary. I know.
Nah. Our entire IT function (including me) is about half a dozen humans.
>> * If the storage network fails, ALL servers fail, so you're adding a new
>> single point of failure.
>
> True, but that rarely happens.
OK, fair enough.
>> * All the servers now have to share the limited bandwidth available. One
>> busy server can bring all the others to a crawl.
>
> What limited bandwidth? Even a SAN that's been improperly designed, and
> that has seen 5 years of unchecked growth will have more than enough
> bandwitdh to accomodate all the servers. Have you seen the
> specifications of the SAN your head office wants to implement and were
> able to determine there will be bottlenecks?
My point is, if you have 10 servers, each with their own dedicated SATA
connection (or whatever), you have 10 lots of bandwidth. And if one
server decides to hog it all, it can't slow the others down.
If you have a SAN, you have /one/ access channel, which all the servers
have to share between them. If one server decides to hit it really hard
(or just crashes and starts sending gibberish), all the others get
clobbered.
>> Oh, wait, I just found a real advantage: If you want to move a disk from
>> one server to another, you just press a button, rather than having to
>> physically move the device.
>>
>> Oh, well, that /totally/ makes it worth it. And, uh, when will you
>> *ever* need to perform this operation? I mean, seriously, I've been
>> working at the UK site for almost 10 years now, and I've never once
>> wanted to do this. And even if I did, if on one day in 10 years I have
>> to shut both servers down to physically move a drive, I think I can
>> probably live with that.
>
> My current customer has two data centers with 350 servers in each. "We
> need more disk space on server XYZ" happens on a daily basis, in fact,
> multiple times a day, in some cases (e.g. a fast growing log file due to
> a change in user traffic, or the activation of a debbugging feature
> during problem troubleshooting)
OK, well, perhaps if you have hundreds of servers, the massive
performance hit /might/ be justified by the advantages of being able to
keep more systems up more of the time. But for our 10 servers or so,
which are almost all up almost all of the time, the advantage seems
vanishingly small.
>> Second, what is the purpose of a backup? The purpose of having a "backup
>> copy" of your data is so that if your working copy dies somehow, you can
>> use the backup copy to get your data back.
>>
>> This only works if the backup copy is more reliable than the working
>> copy. If your working copy is on spinning disk, and you're stupid enough
>> to put your backup copy on spinning disk as well... then it becomes
>> equally likely that the /backup copy/ will die and you'll be left with
>> just the working copy.
>
> As long as your working copy doesn't disappear before your can recreate
> the backup copy, where's the problem? Are you saying that RAID is a
> waste of time because there's an equal likelyhood that any of the disks
> in the array will die?
A backup copy is supposed to be an image of the system frozen at a
particular point in time. Once that data is gone, you can't get it back.
Because that moment in time has gone.
RAID isn't the same time; it's supposed to keep the system running in
the event of a fault, not preserve historical data.
>> Third, having access only to the most recent backup is no good. There
>> are scenarios where that would be perfectly OK, but in our line of
>> business, we sometimes have to recover data from /months/ ago. Data that
>> has been accidentally deleted. Data which got corrupted. Data which we
>> thought we didn't need any more but actually we do. And so forth.
>
> Have they actually stated that you would lose the ability of going back
> to older copies of your backups?
No.
I'm not saying we will lose this ability. I'm just noting that we need
to be careful that we don't lose this ability. Simply mirroring the
disks to the SAN at HQ or whatever is not sufficient. We are legally
required to be able to restore old data.
Hopefully the guys setting this up know that. As I say, they haven't
actually decided how this is going to work yet.
>> And then there's the fact that you either need to keep all this disk
>> spinning (i.e., ever increasing power and cabling demands), or only keep
>> the recent backups spinning (i.e., managing powering off and powering on
>> drives, which supposedly shortens their lifetime).
>>
>> In all, this seems like an extremely bad idea.
>
> Have they actually stated that they would not be taking tape (or more
> than likely DVD) backups of that data for archival?
Bizarro. Why would you back up to a DVD that holds a piffling 4.2 GB of
data when a single tape holds 400 GB? That makes no sense at all.
> In all likelyhood, that remote copy will probably be backed up to
> offsite storage - if not, it should be, just for the sake of business
> continuity, but in many cases, for legal reasons as well.
Nope. The idea, apparently, is that there will be two remote copies, at
two different sites. If one dies, we've still got the other site.
They're planning to have no tape at all, just spinning disk.
>> Still, what I do know? Apparently not a lot.
>
> No offense, but I thought this was an established fact. ;)
Hey, the greatest knowledge is in knowing that you know nothing.
Remember that...
> Don't get me wrong, you are extremely knowledgeable in many areas, but
> you seem to be unable to admit that other people may have other reasons
> to do something that you deem unnecessary because it doesn't apply to
> your situation.
I'm saying that from what I can understand about this company's
situation, the proposed course of action doesn't make sense. That's all.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> Not necessarily. It depends a lot on the configuration of the locally
>> attached drives vs. SAN logical disks, and the types of IO requests that
>> are made (e.g. long sequential reads or writes vs. lots of short random
>> file accesses). SANs have all kinds of caching mechanisms that can make
>> the IO actually faster than a SATA connection.
>
> SATA 3.0 connection: 6.0 GB/sec
>
> Fast Ethernet connection: 0.1 GB/sec
>
> That's, what, 6000x slower?
No, that's 60x times slower.
Besides, SAN connections are NOT fast ethernet. Depending on the length
and type of fibre, the speed will vary from 100 MBps to 1.6 GBps, which
is still admittedly slower than a SATA 3 controller, but unless you are
copying one large sequential file, you will also need to worry about
drive seek time and file system layout to get acutal throughput
comparisons, and Google says it's a wash.
>
> Sure, you can add caching, which might speed up write access. But it
> can't speed up read access that misses the cache.
>
>>> * Radically increased complexity compared to a direct connection.
>>
>> That depends on the point of view. From the server administrator's point
>> of view, it's much simpler. She no longer has to worry about spreading
>> her disks across multiple controllers, mirroring, etc.... on which
>> partition should the indexes go? What level of RAID to use? Is there
>> enough power in the server's power supplies to drive an additional pair
>> of disks? It's all taken care of at the other end of a pair of fibre
>> connectors.
>
> I fail to see why RAID is no longer an issue. (Unless you mean that it's
> now acceptable to lose data in the event of a hardware fault.)
RAID is an issue for the SAN administrator, not the server
administrator. The server admin gets assigned one (or more) virtual
drive(s) and can safely assume that the data integrity will be
maintained. The SAN administrator will make sure that the logical drive
has been allocated out of properly configured pools of RAIDs.
>
> I've yet to see any system where power consumption was even remotely
> important. Servers seem to come with power supplies that can handle
> *way* more current than you will ever need.
>
Usually, yes, but not always.
> I've also never seen a sever with more than one disk controller, come to
> mention it...
>
Then your servers are far from optimal. In my work, It's not uncommon
to see servers with 3 or 4 controllers.
1 for the OS, with its own set of RAID drives.
1 for the apps with its own set of RAID drives. (you don't want log
file or library file I/O to interfere with DB performance)
1 or more for the database, with its own sets of drives. The indexes
and redo logs all on different drives than the tables, to make sure that
you don't cause bottlenecks by having the drive heads jumping from the
index to the table back to the index...
With the exception of the OS drive (You don't want to boot off the SAN),
all of those* can be sent to the SAN for where they will be spread
across many disks and controllers for an optimal performance.
*Don't run your domain controllers or DNSes from a SAN either.
>> From the point of view of the backup administrator, it's much simpler
>> since he no longer has to worry about making sure that the backup agent
>> is running on all the servers and that their schedules don't actually
>> step over each other's toes, etc... it can all be done off-line in the
>> background.
>
> I don't see how that works.
The DVD or tape drives have direct backplane connections to the disk
arrays so they can read the disks without impacting the network or
competing with the server's own IO requests.
Just like a hardware RAID controller can rebuild a failed drive without
the any noticeable performance degradation.
>
>> You just had a very weird problem? Just clone the DB to an new set of
>> disks, and assign those on a lab server so that the developpers can work
>> on the issue all weekend.
>
> You can trivially do that without a SAN. Disk imaging tools have existed
> for decades. Either way, you still have to take the DB server offline
> while you do this.
No. That's the point. The SAN administrator simply mirrors the disks
in the background, without the server even knowing about it.
>
> Of course, if your DB server is a VM, the problem just became even more
> trivial. Depending on your virtualisation technology, maybe you don't
> even have to stop the server to copy the disk image. And you /still/
> don't need a SAN.
>
>> The biggest added level of complexity is that all these people have to
>> talk to one another. Scary. I know.
>
> Nah. Our entire IT function (including me) is about half a dozen humans.
>
>>> * If the storage network fails, ALL servers fail, so you're adding a new
>>> single point of failure.
>>
>> True, but that rarely happens.
>
> OK, fair enough.
>
>>> * All the servers now have to share the limited bandwidth available. One
>>> busy server can bring all the others to a crawl.
>>
>> What limited bandwidth? Even a SAN that's been improperly designed, and
>> that has seen 5 years of unchecked growth will have more than enough
>> bandwitdh to accomodate all the servers. Have you seen the
>> specifications of the SAN your head office wants to implement and were
>> able to determine there will be bottlenecks?
>
> My point is, if you have 10 servers, each with their own dedicated SATA
> connection (or whatever), you have 10 lots of bandwidth. And if one
> server decides to hog it all, it can't slow the others down.
>
> If you have a SAN, you have /one/ access channel, which all the servers
> have to share between them. If one server decides to hit it really hard
> (or just crashes and starts sending gibberish), all the others get
> clobbered.
Again, wrong. Each server has dedicated access to the SAN fabric and
the SAN switches will dispatch I/O requests to the proper drive(s)
without impacting other servers' I/O requests.
>
>>> Oh, wait, I just found a real advantage: If you want to move a disk from
>>> one server to another, you just press a button, rather than having to
>>> physically move the device.
>>>
>>> Oh, well, that /totally/ makes it worth it. And, uh, when will you
>>> *ever* need to perform this operation? I mean, seriously, I've been
>>> working at the UK site for almost 10 years now, and I've never once
>>> wanted to do this. And even if I did, if on one day in 10 years I have
>>> to shut both servers down to physically move a drive, I think I can
>>> probably live with that.
>>
>> My current customer has two data centers with 350 servers in each. "We
>> need more disk space on server XYZ" happens on a daily basis, in fact,
>> multiple times a day, in some cases (e.g. a fast growing log file due to
>> a change in user traffic, or the activation of a debbugging feature
>> during problem troubleshooting)
>
> OK, well, perhaps if you have hundreds of servers, the massive
> performance hit /might/ be justified by the advantages of being able to
> keep more systems up more of the time. But for our 10 servers or so,
> which are almost all up almost all of the time, the advantage seems
> vanishingly small.
>
>>> Second, what is the purpose of a backup? The purpose of having a "backup
>>> copy" of your data is so that if your working copy dies somehow, you can
>>> use the backup copy to get your data back.
>>>
>>> This only works if the backup copy is more reliable than the working
>>> copy. If your working copy is on spinning disk, and you're stupid enough
>>> to put your backup copy on spinning disk as well... then it becomes
>>> equally likely that the /backup copy/ will die and you'll be left with
>>> just the working copy.
>>
>> As long as your working copy doesn't disappear before your can recreate
>> the backup copy, where's the problem? Are you saying that RAID is a
>> waste of time because there's an equal likelyhood that any of the disks
>> in the array will die?
>
> A backup copy is supposed to be an image of the system frozen at a
> particular point in time. Once that data is gone, you can't get it back.
> Because that moment in time has gone.
>
> RAID isn't the same time; it's supposed to keep the system running in
> the event of a fault, not preserve historical data.
>
>>> Third, having access only to the most recent backup is no good. There
>>> are scenarios where that would be perfectly OK, but in our line of
>>> business, we sometimes have to recover data from /months/ ago. Data that
>>> has been accidentally deleted. Data which got corrupted. Data which we
>>> thought we didn't need any more but actually we do. And so forth.
>>
>> Have they actually stated that you would lose the ability of going back
>> to older copies of your backups?
>
> No.
>
> I'm not saying we will lose this ability. I'm just noting that we need
> to be careful that we don't lose this ability. Simply mirroring the
> disks to the SAN at HQ or whatever is not sufficient. We are legally
> required to be able to restore old data.
>
> Hopefully the guys setting this up know that. As I say, they haven't
> actually decided how this is going to work yet.
>
>>> And then there's the fact that you either need to keep all this disk
>>> spinning (i.e., ever increasing power and cabling demands), or only keep
>>> the recent backups spinning (i.e., managing powering off and powering on
>>> drives, which supposedly shortens their lifetime).
>>>
>>> In all, this seems like an extremely bad idea.
>>
>> Have they actually stated that they would not be taking tape (or more
>> than likely DVD) backups of that data for archival?
>
> Bizarro. Why would you back up to a DVD that holds a piffling 4.2 GB of
> data when a single tape holds 400 GB? That makes no sense at all.
>
Tapes degrade over time faster than DVDs do.
I/O is sequential on a tape. If you need a specific file, you need to
read through the whole tape from the beginning.
If you only have a 10GB incremental per day, and need to sent it to
offsite storage every day (sometimes required by law), using 400GB tape
is a serious waste of money.
Each application has its preferred solution. Your mileage may vary.
>> In all likelyhood, that remote copy will probably be backed up to
>> offsite storage - if not, it should be, just for the sake of business
>> continuity, but in many cases, for legal reasons as well.
>
> Nope. The idea, apparently, is that there will be two remote copies, at
> two different sites. If one dies, we've still got the other site.
> They're planning to have no tape at all, just spinning disk.
>
That _is_ silly.
--
/*Francois Labreque*/#local a=x+y;#local b=x+a;#local c=a+b;#macro P(F//
/* flabreque */L)polygon{5,F,F+z,L+z,L,F pigment{rgb 9}}#end union
/* @ */{P(0,a)P(a,b)P(b,c)P(2*a,2*b)P(2*b,b+c)P(b+c,<2,3>)
/* gmail.com */}camera{orthographic location<6,1.25,-6>look_at a }
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> SATA 3.0 connection: 6.0 GB/sec
>>
>> Fast Ethernet connection: 0.1 GB/sec
>>
>> That's, what, 6000x slower?
>
> No, that's 60x times slower.
Epic math fail. :'{
> Besides, SAN connections are NOT fast ethernet. Depending on the length
> and type of fibre, the speed will vary from 100 MBps to 1.6 GBps, which
> is still admittedly slower than a SATA 3 controller, but unless you are
> copying one large sequential file, you will also need to worry about
> drive seek time and file system layout to get acutal throughput
> comparisons, and Google says it's a wash.
I rather doubt that drive seek time is longer than network latency. I
also rather doubt that you can't saturate a single network link with
half a dozen drives.
>> I fail to see why RAID is no longer an issue. (Unless you mean that it's
>> now acceptable to lose data in the event of a hardware fault.)
>
> RAID is an issue for the SAN administrator, not the server
> administrator.
Oh, I see.
This applies only if the SAN administrator and the server administrator
aren't the same person. In our setup, with a grand total of 4 IT staff,
I doubt this is relevant.
Like I said, increased complexity.
>> I've also never seen a sever with more than one disk controller, come to
>> mention it...
>
> Then your servers are far from optimal.
Well, I guess you can /always/ have more fault tolerance. But in 10
years of working here, I've only seen one controller fail. Come to think
of it, I've only seen about 6 drives fail (including desktop PCs and
laptops).
Currently, I have two servers. One has 2 disks, the other has 3 active
disks plus a hot spare. I doubt adding more controllers is worth the effort.
> With the exception of the OS drive (You don't want to boot off the SAN),
> *Don't run your domain controllers or DNSes from a SAN either.
Any specific reason?
>> I don't see how that works.
>
> The DVD or tape drives have direct backplane connections to the disk
> arrays so they can read the disks without impacting the network or
> competing with the server's own IO requests.
So how does it obtain a consistent image of the filesystem?
> Just like a hardware RAID controller can rebuild a failed drive without
> the any noticeable performance degradation.
[RAID does it by constantly resynchronising.]
>>> You just had a very weird problem? Just clone the DB to an new set of
>>> disks, and assign those on a lab server so that the developpers can work
>>> on the issue all weekend.
>>
>> You can trivially do that without a SAN. Disk imaging tools have existed
>> for decades. Either way, you still have to take the DB server offline
>> while you do this.
>
> No. That's the point. The SAN administrator simply mirrors the disks in
> the background, without the server even knowing about it.
Like I say, I don't see how you can obtain a consistent image of the
filesystem without cooperation from the server.
>> My point is, if you have 10 servers, each with their own dedicated SATA
>> connection (or whatever), you have 10 lots of bandwidth. And if one
>> server decides to hog it all, it can't slow the others down.
>>
>> If you have a SAN, you have /one/ access channel, which all the servers
>> have to share between them. If one server decides to hit it really hard
>> (or just crashes and starts sending gibberish), all the others get
>> clobbered.
>
> Again, wrong. Each server has dedicated access to the SAN fabric and the
> SAN switches will dispatch I/O requests to the proper drive(s) without
> impacting other servers' I/O requests.
There is one network switch. All data must pass through this single
device. If it only processes X bytes per second, than all the servers
between them cannot access more than X bytes per second. The end.
Unless every single server and several single disk has its own dedicated
channel to the switch (which is presumably infeasible), there will be
additional bottlenecks for components sharing channels too.
>>>> In all, this seems like an extremely bad idea.
>>>
>>> Have they actually stated that they would not be taking tape (or more
>>> than likely DVD) backups of that data for archival?
>>
>> Bizarro. Why would you back up to a DVD that holds a piffling 4.2 GB of
>> data when a single tape holds 400 GB? That makes no sense at all.
>
> Tapes degrade over time faster than DVDs do.
In which universe? I would have said the exact opposite was true. I've
seen CDs and DVDs burned 6 months ago where the ink has faded to the
point where the disk is utterly unusable. I've yet to see a tape fail to
read.
> I/O is sequential on a tape. If you need a specific file, you need to
> read through the whole tape from the beginning.
That's true though. Fortunately, file restoration is a rare event, so it
doesn't matter too much.
> If you only have a 10GB incremental per day, and need to sent it to
> offsite storage every day (sometimes required by law), using 400GB tape
> is a serious waste of money.
The way we do it is to store all the tapes off-site, except today's
tape. Rotate the tapes until they become full, then add in a new tape.
All 400GB of capacity gets used, and yet at any time the majority of
tapes are off-site.
> Each application has its preferred solution. Your mileage may vary.
Fair enough. I'm just saying, for our setup, this wouldn't be a good idea.
>>> In all likelyhood, that remote copy will probably be backed up to
>>> offsite storage - if not, it should be, just for the sake of business
>>> continuity, but in many cases, for legal reasons as well.
>>
>> Nope. The idea, apparently, is that there will be two remote copies, at
>> two different sites. If one dies, we've still got the other site.
>> They're planning to have no tape at all, just spinning disk.
>
> That _is_ silly.
...which is what I said a week ago. ;-)
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>>> I fail to see why RAID is no longer an issue. (Unless you mean that it's
>>> now acceptable to lose data in the event of a hardware fault.)
>>
>> RAID is an issue for the SAN administrator, not the server
>> administrator.
>
> Oh, I see.
>
> This applies only if the SAN administrator and the server administrator
> aren't the same person. In our setup, with a grand total of 4 IT staff,
> I doubt this is relevant.
This is why I said it would be silly to have a SAN for only a handful of
devices. In a data center with 200+ servers, you'll usually end up with
entirely different support teams for Mainframe, mid-range (AS/400,
Tandem, etc...), Unix (sometimes different groups for different flavours
of), Windows, and Netware (if there are still some of it), having a
group of 4 or 5 people dedicated to the SAN administration allows them
to specialize in their field, and hopefully become more competent at it,
and leaving the OS folks deal with patching, tuning processes, etc...
>>> I've also never seen a sever with more than one disk controller, come to
>>> mention it...
>>
>> Then your servers are far from optimal.
>
> Well, I guess you can /always/ have more fault tolerance. But in 10
> years of working here, I've only seen one controller fail. Come to think
> of it, I've only seen about 6 drives fail (including desktop PCs and
> laptops).
Not fault tolerance. PERFORMANCE. You know, that thing you were saying
would degrade with a SAN.
>> With the exception of the OS drive (You don't want to boot off the SAN),
>
>> *Don't run your domain controllers or DNSes from a SAN either.
>
> Any specific reason?
>
Because in the event that the SAN went away, you may end up in a
situation where the DNS or AD is actually required to allow support
personnel to work on the issue (think VPN access for remote support
personnel). As much as you want to, in an environment with hundreds of
servers, you're bound to end up with either having to rely on DNS
services or deal with updating hundreds of local HOSTS files, etc.
>>> I don't see how that works.
>>
>> The DVD or tape drives have direct backplane connections to the disk
>> arrays so they can read the disks without impacting the network or
>> competing with the server's own IO requests.
>
> So how does it obtain a consistent image of the filesystem?
>
The same way you do with a local backup agent. The SAN is not a magical
invention that removes all worldly constraints. In other words, you
schedule the backup at the time where its least likely there will be
changes to the file system, and pray to the deity of your choice.
In other words, in a data center for a multinational corporation where
planes are up in the air, or trading is going on in a stock exchange
somewhere 24 hrs a day, 7 days a week, you NEVER have a consistent copy.
You simply hope to have a GOOD ENOUGH copy.
For systems that require a "snapshot" of the state at specific times,
you schedule downtime at the application level and take the backup
during that time, for that application. For example, at one of my
former customers, the HR/Finance ERP system would switch to read-only
mode from 0h00 to 06h00, every day to allow consistent backups to be
taken. This allowed an accountant burning the midnight oil to be able
to still read the general ledger, while allowing for a clean backup to
be taken.
>>>
>>> If you have a SAN, you have /one/ access channel, which all the servers
>>> have to share between them. If one server decides to hit it really hard
>>> (or just crashes and starts sending gibberish), all the others get
>>> clobbered.
>>
>> Again, wrong. Each server has dedicated access to the SAN fabric and the
>> SAN switches will dispatch I/O requests to the proper drive(s) without
>> impacting other servers' I/O requests.
>
> There is one network switch. All data must pass through this single
> device. If it only processes X bytes per second, than all the servers
> between them cannot access more than X bytes per second. The end.
There are multiple SAN switches (you definitely DON'T want a single
point of failure). Each with enough backplane bandwidth, processor
power and ports to handle I/O requests at wire speed.
A typical set-up would have each server connected to two different SAN
switches (sometimes refered to as "front-end" switches), and these would
in turn be connected to each of the disk bays, on top of having
interconnections between the switches, to prevent a fibre cut from
shutting a part of the storage network down. Said typical set-up would
also probably have performance monitoring software available allowing
the SAN administrators to see where potential bottelnecks will be and
take approriate measure during a scheduled maintenance window.
>
> Unless every single server and several single disk has its own dedicated
> channel to the switch (which is presumably infeasible), there will be
> additional bottlenecks for components sharing channels too.
>
You have presumed wrong. This is exactly what happens.
>>>> Have they actually stated that they would not be taking tape (or more
>>>> than likely DVD) backups of that data for archival?
>>>
>>> Bizarro. Why would you back up to a DVD that holds a piffling 4.2 GB of
>>> data when a single tape holds 400 GB? That makes no sense at all.
>>
>> Tapes degrade over time faster than DVDs do.
>
> In which universe? I would have said the exact opposite was true. I've
> seen CDs and DVDs burned 6 months ago where the ink has faded to the
> point where the disk is utterly unusable. I've yet to see a tape fail to
> read.
Stop buying cheap DVDs then ;). I've seen tapes fail in drives.
<strike>I've seen attack ships fire off the shoulder of Orion</strike>,
I've seen tapes where someone had to actually change the plastic
leader thingie because it had become so mangled from repeated use that
it wouldn't engage in the drive any more. Etc...
>
>> I/O is sequential on a tape. If you need a specific file, you need to
>> read through the whole tape from the beginning.
>
> That's true though. Fortunately, file restoration is a rare event, so it
> doesn't matter too much.
Again that depends on the application. Most large corporations will
have data exchange that still takes place over physical media being
FEDEXed (interbank transfers, payroll data for the bank that issues the
cheques, airline schedule exchanges, government regulatory filings,
etc.) as in some cases, it is still cheaper than setting up dedicated
inter-entreprise connections. Some backups are only for business
recovery, but others are intended for use in the very near future.
>
>> If you only have a 10GB incremental per day, and need to sent it to
>> offsite storage every day (sometimes required by law), using 400GB tape
>> is a serious waste of money.
>
> The way we do it is to store all the tapes off-site, except today's
> tape. Rotate the tapes until they become full, then add in a new tape.
> All 400GB of capacity gets used, and yet at any time the majority of
> tapes are off-site.
>
See above. You can't always reuse the same tape. There may ver Very
Good(tm) reasons for that.
Also, I know that multiple backups can be combined on a tape. This is
exactly what we used to do for the file servers, when I did server
support 15 years ago. All incrementals go on rotating tapes, and the
weekly full backup is done on a different tape, which was then
duplicated and one copy was sent to the offsite vault. However, even in
this scenario, if one of the rotating tapes got damaged or lost, the
missing incrementals could still cause major legal woes for the company.
I've since moved away from server support, so I can't comment on what
the storage admins do nowadays. All I know if they have a really cool
robot that fetches tapes and DVDs from the racks instead of relying on
"tape monkeys" as we used to call them.
--
/*Francois Labreque*/#local a=x+y;#local b=x+a;#local c=a+b;#macro P(F//
/* flabreque */L)polygon{5,F,F+z,L+z,L,F pigment{rgb 9}}#end union
/* @ */{P(0,a)P(a,b)P(b,c)P(2*a,2*b)P(2*b,b+c)P(b+c,<2,3>)
/* gmail.com */}camera{orthographic location<6,1.25,-6>look_at a }
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> Oh, I see.
>>
>> This applies only if the SAN administrator and the server administrator
>> aren't the same person. In our setup, with a grand total of 4 IT staff,
>> I doubt this is relevant.
>
> This is why I said it would be silly to have a SAN for only a handful of
> devices.
Fair enough.
>> Well, I guess you can /always/ have more fault tolerance. But in 10
>> years of working here, I've only seen one controller fail. Come to think
>> of it, I've only seen about 6 drives fail (including desktop PCs and
>> laptops).
>
> Not fault tolerance. PERFORMANCE. You know, that thing you were saying
> would degrade with a SAN.
How does adding more controllers increase performance? The controller is
usually the fastest component in the system. The bottleneck is usually
either the bus to the CPU, or the connections to the HDs.
>>> The DVD or tape drives have direct backplane connections to the disk
>>> arrays so they can read the disks without impacting the network or
>>> competing with the server's own IO requests.
>>
>> So how does it obtain a consistent image of the filesystem?
>
> The same way you do with a local backup agent. The SAN is not a magical
> invention that removes all worldly constraints. In other words, you
> schedule the backup at the time where its least likely there will be
> changes to the file system, and pray to the deity of your choice.
That's /not/ how a local backup agent works. Indeed, the entire
/purpose/ of a local backup agent is to construct a consistent image (by
whatever means that requires) to write to tape.
> In other words, in a data center for a multinational corporation where
> planes are up in the air, or trading is going on in a stock exchange
> somewhere 24 hrs a day, 7 days a week, you NEVER have a consistent copy.
> You simply hope to have a GOOD ENOUGH copy.
Nonsense. If you're doing something like that, you're probably using a
relational database. And any half-decent product of that type provides a
way to construct a consistent image without ever stopping the database.
(You do, however, need the cooperation of the server that's running the
thing. You can't just take a block-level image of the filesystem and
expect to get anything sensible out of it.)
>> There is one network switch. All data must pass through this single
>> device. If it only processes X bytes per second, than all the servers
>> between them cannot access more than X bytes per second. The end.
>
> There are multiple SAN switches (you definitely DON'T want a single
> point of failure). Each with enough backplane bandwidth, processor power
> and ports to handle I/O requests at wire speed.
So what you're saying is that you wire up two independent networks, and
connect every device to both of them, so that if the switch running
network A fails, you can use network B instead?
>> Unless every single server and every single disk has its own dedicated
>> channel to the switch (which is presumably infeasible), there will be
>> additional bottlenecks for components sharing channels too.
>
> You have presumed wrong. This is exactly what happens.
Damn. This stuff is more expensive than I thought... Still, great way to
make your department look important, I guess.
>>> Tapes degrade over time faster than DVDs do.
>>
>> In which universe? I would have said the exact opposite was true. I've
>> seen CDs and DVDs burned 6 months ago where the ink has faded to the
>> point where the disk is utterly unusable. I've yet to see a tape fail to
>> read.
>
> Stop buying cheap DVDs then ;)
To be fair, this seems to have become less of a problem has CD-R
technology has improved over the years. Much like anything else, I imagine.
> I've seen tapes fail in drives.
> I've seen tapes where someone had to actually change the plastic leader
> thingie because it had become so mangled from repeated use that it
> wouldn't engage in the drive any more. Etc...
Hell, one time I tried to destroy a tape on purpose and I actually
couldn't do it. We soaked the tape in lab chemicals and it wouldn't
degrade, I tried to cut it up but it was too indestructible, it was
really, really hard to get rid of this stuff. (Although probably a lot
easier to just make the drive not read it any more.)
>>> I/O is sequential on a tape. If you need a specific file, you need to
>>> read through the whole tape from the beginning.
>>
>> That's true though. Fortunately, file restoration is a rare event, so it
>> doesn't matter too much.
>
> Again that depends on the application.
OK, for /our/ application, it's a rare event. And that's really what I'm
interested in. Does this technology make any sense at all for us? From
what I can see, no, it doesn't.
> I've since moved away from server support, so I can't comment on what
> the storage admins do nowadays. All I know if they have a really cool
> robot that fetches tapes and DVDs from the racks instead of relying on
> "tape monkeys" as we used to call them.
I have a small tape robot. If I wanted to, I could load a year's worth
of tapes into it and never have to change tape until next year.
(Unfortunately, I can't actually do that, because then none of the tapes
would be "off site".) It even has a cute little barcode reader.
That was HQ's idea. They don't often have good ideas, but this was one
of them. ;-)
Now of course, they're talking about taking that away and using our slow
Internet connection to do the same job... *sigh*
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>>>
>>> So how does it obtain a consistent image of the filesystem?
>>
>> The same way you do with a local backup agent. The SAN is not a magical
>> invention that removes all worldly constraints. In other words, you
>> schedule the backup at the time where its least likely there will be
>> changes to the file system, and pray to the deity of your choice.
>
> That's /not/ how a local backup agent works. Indeed, the entire
> /purpose/ of a local backup agent is to construct a consistent image (by
> whatever means that requires) to write to tape.
No. the purpose of a local backup agent is:
1) to have the means to back the files up at shceduled intervals, and
2) talk to the backup unit (be it another locally-mounted file system,
removable storage media, or file transfer over a network).
Both of these functions are nowadays available directly from the OS, but
most entreprises will have a separate tool to do this as there are other
advantages when it comes to centralized management, or the ability to
lock access to certain files or directories, or able to speak in a
dialect understood by the application (e.g. Oracle "live backup" agent),
but in many cases, it is not necessary and the backup job can run while
the users access their files. At that point, it becomes a management
decision based on the relative costs penalty of scheduling down time vs.
losing absolute file consistency.
>
>> In other words, in a data center for a multinational corporation where
>> planes are up in the air, or trading is going on in a stock exchange
>> somewhere 24 hrs a day, 7 days a week, you NEVER have a consistent copy.
>> You simply hope to have a GOOD ENOUGH copy.
>
> Nonsense. If you're doing something like that, you're probably using a
> relational database. And any half-decent product of that type provides a
> way to construct a consistent image without ever stopping the database.
> (You do, however, need the cooperation of the server that's running the
> thing. You can't just take a block-level image of the filesystem and
> expect to get anything sensible out of it.)
>
Agreed for databases, but even just a source tree could become
inconsistent if a programmer is making changes at the same time as the
backup job is running. As I said, if it absolutely, positiviely,
legally requires a consistent copy, means will be taken to achieve that.
But if it's just a minor inconvenience for a few persons to have an
older draft version of their weekly powerpoint presentation to upper
management having been saved prior to the server failure, then so be it.
>>> There is one network switch. All data must pass through this single
>>> device. If it only processes X bytes per second, than all the servers
>>> between them cannot access more than X bytes per second. The end.
>>
>> There are multiple SAN switches (you definitely DON'T want a single
>> point of failure). Each with enough backplane bandwidth, processor power
>> and ports to handle I/O requests at wire speed.
>
> So what you're saying is that you wire up two independent networks, and
> connect every device to both of them, so that if the switch running
> network A fails, you can use network B instead?
The storage network hardware is all interconnected, but the storage
network is independant from the user-traffic network.
----------
/ \
| IP Network |
\ /
-----------
/ / \ \
-------- --------
| server | | server |
| 1 | | 2 | Server N, etc...
-------- --------
| \ / | /
| x | /
| / \ | /
-------- --------
| SAN sw |---| SAN sw |--- Switch N, etc...
| 1 |---| 2 |---
-------- --------
| \ / |
| x |
| / \ |
-------- --------
| Drive | | Drive | --> Repeat if uptime service
| bay | | bay | level agreements require
| | | | site-level diversity, or if
| 1 | | 2 | required by capacity.
-------- --------
\ /
---------
| Tape |
| Robot |
---------
Typicially, the SAN switches will have up to 16 or so "front end" ports
and 2 or more higher-speed "Back-end" ports to the drive bays. Some
high-load servers might have direct connections to the drive bays.
>
>>> Unless every single server and every single disk has its own dedicated
>>> channel to the switch (which is presumably infeasible), there will be
>>> additional bottlenecks for components sharing channels too.
>>
>> You have presumed wrong. This is exactly what happens.
>
> Damn. This stuff is more expensive than I thought... Still, great way to
> make your department look important, I guess.
>
It's the only way to achieve "five 9s" uptime (i.e. 99.999%), which is
what most banks and multinationals require. Again, for a small to
medium shop, with a dozen to 50 servers or so servers, or no
requirements for such high SLAs, then it's overkill.
It's worthwhile to note that five-9s uptime also require considerable
thoughts to the choice of applications, hardware, network design,
climate control, power management, staffing arrangements, telco
diversity, disaster prevention and business continuity, in the event
that a disaster can not be avoided. This can add one or two orders of
magnitude to a company's IT budget, so it's not just to make your dept.
look important. There has to be a sufficient increase in revenues (or
decrease in losses!) to justify it.
--
/*Francois Labreque*/#local a=x+y;#local b=x+a;#local c=a+b;#macro P(F//
/* flabreque */L)polygon{5,F,F+z,L+z,L,F pigment{rgb 9}}#end union
/* @ */{P(0,a)P(a,b)P(b,c)P(2*a,2*b)P(2*b,b+c)P(b+c,<2,3>)
/* gmail.com */}camera{orthographic location<6,1.25,-6>look_at a }
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>>>> So how does it obtain a consistent image of the filesystem?
>>>
>>> The same way you do with a local backup agent. The SAN is not a magical
>>> invention that removes all worldly constraints. In other words, you
>>> schedule the backup at the time where its least likely there will be
>>> changes to the file system, and pray to the deity of your choice.
>>
>> That's /not/ how a local backup agent works. Indeed, the entire
>> /purpose/ of a local backup agent is to construct a consistent image (by
>> whatever means that requires) to write to tape.
>
> No. the purpose of a local backup agent is:
> 1) to have the means to back the files up at shceduled intervals, and
> 2) talk to the backup unit (be it another locally-mounted file system,
> removable storage media, or file transfer over a network).
>
> Both of these functions are nowadays available directly from the OS, but
> most entreprises will have a separate tool to do this as there are other
> advantages when it comes to centralized management, or the ability to
> lock access to certain files or directories, or able to speak in a
> dialect understood by the application (e.g. Oracle "live backup" agent),
> but in many cases, it is not necessary and the backup job can run while
> the users access their files. At that point, it becomes a management
> decision based on the relative costs penalty of scheduling down time vs.
> losing absolute file consistency.
OK, first of all, let me split two things out:
First, we have consistency of the physical file system itself.
Second, we have consistency of the logical files contained on disk.
If the FS becomes inconsistent, usually that renders it completely
unreadable without advanced recovery tools. An unreadable backup is useless.
If a logical file becomes inconsistent, that might just mean that Word
crashes when you try to open your document. So the backup copy of that
document is useless, but other documents on the disk are probably usable.
Either way, the point of having a remote backup agent is to prevent
these kinds of inconsistencies. Firstly, by accessing the file system
through the OS, you guarantee that what you see is a consistent image of
the file system. Secondly, I know on Windows you can do things like
Shadow Copy, which guarantees a consistent image of all logical files,
even if people are editing them while you back them up.
It's completely possible to back up remote disks without installing a
remote agent. But nobody does this, because the file consistency issues
are too bad.
(And yes, then there are other advantages. A local backup agent can use
the NTFS journal to work out which files have changed recently for
incremental backups. It can compress data before sending it over the
network. And so forth.)
> As I said, if it absolutely, positiviely, legally
> requires a consistent copy, means will be taken to achieve that. But if
> it's just a minor inconvenience for a few persons to have an older draft
> version of their weekly powerpoint presentation to upper management
> having been saved prior to the server failure, then so be it.
An older draft isn't so bad. When the file becomes completely
unreadable, that generally upsets people.
>> So what you're saying is that you wire up two independent networks, and
>> connect every device to both of them, so that if the switch running
>> network A fails, you can use network B instead?
>
> The storage network hardware is all interconnected, but the storage
> network is independant from the user-traffic network.
Interesting that you've drawn the tape robot at the opposite end of the
SAN. Usually that's /controlled by/ one of the servers.
> Typicially, the SAN switches will have up to 16 or so "front end" ports
> and 2 or more higher-speed "Back-end" ports to the drive bays. Some
> high-load servers might have direct connections to the drive bays.
I still find it hard to believe that a shared fabric can come anywhere
close to the performance of a dedicated fabric.
>>>> Unless every single server and every single disk has its own dedicated
>>>> channel to the switch (which is presumably infeasible), there will be
>>>> additional bottlenecks for components sharing channels too.
>>>
>>> You have presumed wrong. This is exactly what happens.
>>
>> Damn. This stuff is more expensive than I thought... Still, great way to
>> make your department look important, I guess.
>
> It's the only way to achieve "five 9s" uptime (i.e. 99.999%), which is
> what most banks and multinationals require. Again, for a small to medium
> shop, with a dozen to 50 servers or so servers, or no requirements for
> such high SLAs, then it's overkill.
So people really are willing to sacrifice huge amounts of performance
just for increased uptime?
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 8/15/2011 6:04, Francois Labreque wrote:
> you schedule
> the backup at the time where its least likely there will be changes to the
> file system, and pray to the deity of your choice.
You don't need to do that. You just make a virtual snapshot of the drive
you're backing up and back that up.
> In other words, in a data center for a multinational corporation where
> planes are up in the air, or trading is going on in a stock exchange
> somewhere 24 hrs a day, 7 days a week, you NEVER have a consistent copy.
You can. That's exactly what Windows VSS is for. Linux has something almost
as sophisticated on some of its systems.
> For systems that require a "snapshot" of the state at specific times, you
> schedule downtime at the application level and take the backup during that
> time, for that application.
You usually want to do this for upgrades and such also. Having a fixed
schedule of downtime is useful for all kinds of stuff other than just taking
backups.
--
Darren New, San Diego CA, USA (PST)
How come I never get only one kudo?
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> As I said, if it absolutely, positiviely, legally
>> requires a consistent copy, means will be taken to achieve that. But if
>> it's just a minor inconvenience for a few persons to have an older draft
>> version of their weekly powerpoint presentation to upper management
>> having been saved prior to the server failure, then so be it.
>
> An older draft isn't so bad. When the file becomes completely
> unreadable, that generally upsets people.
>
That depends on the people and the amount of changes that occured
between saves.
>
> Interesting that you've drawn the tape robot at the opposite end of the
> SAN. Usually that's /controlled by/ one of the servers.
>
I didn't draw all the servers. Secondly, the tape robot would be
controlled by a dedicated servers that did nothing else than control the
back up schedules and keep the inventory of what is backed up where.
>>
>> It's the only way to achieve "five 9s" uptime (i.e. 99.999%), which is
>> what most banks and multinationals require. Again, for a small to medium
>> shop, with a dozen to 50 servers or so servers, or no requirements for
>> such high SLAs, then it's overkill.
>
> So people really are willing to sacrifice huge amounts of performance
> just for increased uptime?
You have that wrong. First, under normal conditions SANs perform at the
approximately the same level or better than dedicated disks*.
Secondly, the increase in the number of servers required to achieve
these levels of uptime will spread the users, reducing the strain on
each server and increasing overall performance of the system. Finally,
as I said, Five-9 uptime is a change in mentality. These corporations
will usually also have more performance measuring tools** to prove that
they are indeed reaching these service levels, and more troubleshooting
tools available to fix a problem fast. So in their quest to achieve
these levels of uptime, people will often find hidden problems in
applications that would have been very difficult to find if they were
running on dedicated hardware.
*http://www.sqlteam.com/article/which-is-faster-san-or-directly-attached-storage
**That's what is currently putting food on my table.
--
/*Francois Labreque*/#local a=x+y;#local b=x+a;#local c=a+b;#macro P(F//
/* flabreque */L)polygon{5,F,F+z,L+z,L,F pigment{rgb 9}}#end union
/* @ */{P(0,a)P(a,b)P(b,c)P(2*a,2*b)P(2*b,b+c)P(b+c,<2,3>)
/* gmail.com */}camera{orthographic location<6,1.25,-6>look_at a }
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 03/08/2011 03:44 PM, Invisible wrote:
> First, it appears that our head office is replacing their current
> arrangement with a big new SAN.
> Fortunately, this only affects HQ, so in a sense I don't have to care
> about this. Even so, it still baffles me.
WRONG!
Apparently they changed their minds. Each site will have their own SAN.
Looks like the plan is for each site to have their own SAN, and any
changes to the data on that SAN gets replicated to HQ and a second site
in the background.
So /that/ should nail our Internet connection nicely then...
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
|
|
