Guess what? I need to recover some data.

 From March.

Our backup rotation doesn't go back nearly that far. In other words, if 
there is to be *any hope* of getting that data back, I'm going to have to...

...undelete files.

 From the server's harddrive. o_O

Now, taking a disk image of a 64 MB flash drive and playing with it is 
one thing. But one does not simply walk into Mordor - er, I mean, one 
does not simply copy a 103 GB RAID array. It's far too huge for there to 
be anywhere to copy it *to*. And finding anything will be fun; any idea 
how many thousands of files are on there *now*? Never mind the deleted ones.

Oh, and it's NTFS, not FAT. Oh yeah, and the data I want to find is not 
in any "common" file format; it's a proprietry format used by our lab 
software.

And of course, on top of all that, what are the chances of actually 
finding deleted files on a busy server? How many minutes do you think a 
file survives before being completely overwritten?

To summarise: It's Wild Goose time! :-D

It *must* be friday...

Post a reply to this message

Invisible <voi### [at] devnull> wrote:
> Our backup rotation doesn't go back nearly that far.

  I think this is the WTF in this post.

  What good is a backup if you are going to destroy backups which are
"too old"? Do they assume that nobody will ever want to restore a file
which was deleted more than a certain amount of time ago?

  Sure, after a certain time it may not be necessary to store daily
backups, but IMO after a certain time eg. weekly backups could be
preserved, and after a longer time eg. monthly backups. You never know
what you may want to restore.

  Besides, if storage space is an issue, haven't they heard of incremental
backups? It is possible to store eg. daily backups (which are fully
restorable to any of the days in question) of an entire month in a way
that requires significantly less space than 30*(size of data being backed up).

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:
> Invisible <voi### [at] devnull> wrote:
>> Our backup rotation doesn't go back nearly that far.
> 
>   I think this is the WTF in this post.
> 
>   What good is a backup if you are going to destroy backups which are
> "too old"? Do they assume that nobody will ever want to restore a file
> which was deleted more than a certain amount of time ago?

Basically... yes, that's the assumption. They assume that if somebody 
deleted a file accidentally, they'll quickly realise and request to have 
the file restored.

Unfortuantely, what happened here is that somebody burned some files to 
CD and then deleted them, and 8 months later we figure out that actually 
they burned the wrong files - yet deleted the right ones. (In fairness, 
we're talking about projects which have names such as 45748-54867-15486. 
Very easy to muddle them up!)

This exact thing has happened once before. I guess two problems in 6 
years is two too many...

The new global backup procedure that's being drafted will mean that 
backup tapes are stored *forever*. (With the corresponding astronomical 
increase in expenditure on new tapes and physical storage space. But 
hey, it's not my money.)

Unfortunately, the new procedure document is also vague as hell and 
pretty hard to comprehend. Clearly the guy who wrote it understands what 
he means... but I don't.

Post a reply to this message

Invisible <voi### [at] devnull> wrote:
> The new global backup procedure that's being drafted will mean that 
> backup tapes are stored *forever*. (With the corresponding astronomical 
> increase in expenditure on new tapes and physical storage space. But 
> hey, it's not my money.)

  As I said, that doesn't need to be the case. For example daily backups
worth an entire month can be stored in an incremental backup system. This
way only one backup per month has to be stored permanently, but it will
still allow restoring any single day of that month, and the size of the
backup will be only somewhat larger than the size of the entire system
being backed up. (Basically it's the entire system plus modifications
during that month.)

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:
> Invisible <voi### [at] devnull> wrote:
>> The new global backup procedure that's being drafted will mean that 
>> backup tapes are stored *forever*. (With the corresponding astronomical 
>> increase in expenditure on new tapes and physical storage space. But 
>> hey, it's not my money.)
> 
>   As I said, that doesn't need to be the case. For example daily backups
> worth an entire month can be stored in an incremental backup system. This
> way only one backup per month has to be stored permanently, but it will
> still allow restoring any single day of that month, and the size of the
> backup will be only somewhat larger than the size of the entire system
> being backed up. (Basically it's the entire system plus modifications
> during that month.)

The plan is to store only the monthly full backup tapes - so that's only 
actually 12 tapes per year. Of course, in 20 years' time we'll have 240 
of the suckers - not to mention the fact that in 20 years' time, people 
probably won't be using LTO1 any more so there probably won't be any 
equipment that can still read the tapes thus making their retention 
utterly pointless. But in the main it's a sound system...

Post a reply to this message

Invisible wrote:
> Warp wrote:
>> Invisible <voi### [at] devnull> wrote:
>>> The new global backup procedure that's being drafted will mean that 
>>> backup tapes are stored *forever*. (With the corresponding 
>>> astronomical increase in expenditure on new tapes and physical 
>>> storage space. But hey, it's not my money.)
>>
>>   As I said, that doesn't need to be the case. For example daily backups
>> worth an entire month can be stored in an incremental backup system. This
>> way only one backup per month has to be stored permanently, but it will
>> still allow restoring any single day of that month, and the size of the
>> backup will be only somewhat larger than the size of the entire system
>> being backed up. (Basically it's the entire system plus modifications
>> during that month.)
> 
> The plan is to store only the monthly full backup tapes - so that's only 
> actually 12 tapes per year. Of course, in 20 years' time we'll have 240 
> of the suckers - not to mention the fact that in 20 years' time, people 
> probably won't be using LTO1 any more so there probably won't be any 
> equipment that can still read the tapes thus making their retention 
> utterly pointless. But in the main it's a sound system...


Back when I was helping to build cars I deleted a file and only noticed 
that it was gone something like 6 months later.

Luckily, their rotation kept some very old tapes.

I don't remember specifically, but their rotation was something like:

daily for a month
after the month, just keep the weekly tapes
After 3 months keep only the monthly - permanently

Or something roughly like that.
It won't catch everything, but it catches most things.



Tom

Post a reply to this message

Invisible wrote:

> To summarise: It's Wild Goose time! :-D
> 
> It *must* be friday...

Hey, neato:

http://ntfsundelete.com/

It works as well! ;-)


But in the case of today's missing files... guess what?... they're not 
there any more. *Such* a surprise! ;-)

Post a reply to this message

Invisible wrote:
> does not simply copy a 103 GB RAID array. It's far too huge for there to 
> be anywhere to copy it *to*.

Huh?  I can't even *buy* a hard drive that small any more, except on a 
laptop disk. A 160G disk is like $50 at the local shop.  I have 10x that 
much space stuffed in my desktop machine.

> To summarise: It's Wild Goose time! :-D

Sounds like it!

-- 
Darren New / San Diego, CA, USA (PST)

Post a reply to this message

Darren New wrote:
> Invisible wrote:
>> does not simply copy a 103 GB RAID array. It's far too huge for there 
>> to be anywhere to copy it *to*.
> 
> Huh?  I can't even *buy* a hard drive that small any more, except on a 
> laptop disk. A 160G disk is like $50 at the local shop.  I have 10x that 
> much space stuffed in my desktop machine.

Are you serious?

Last time I checked, 60 GB is the smallest. (They stopped selling 40 GB 
a while ago though.) For whatever reason, HDs never sell for less than 
about £30. No matter what size or interface type. I guess that's what 
the amount of metal in the case costs or something.

(At the exchange rate *before* the world entered a global recession, $50 
would be just £25 - cheaper than any HD I've ever seen on sale, ever.)

Keep in mind that the servers all use SCSI, which for some reason is 10x 
more expensive than PATA or SATA. None of the servers has more than 200 
GB of storage online. (Some of them do have a single hot spare.) Most of 
the drives are 36 GB each. Even on the brand new Dell rack-mount things 
with the hot-swap brive bays.

>> To summarise: It's Wild Goose time! :-D
> 
> Sounds like it!

Heh. Yeah, well, a challenge can be fun, right?

(Ooo, did I mention? The drive holding the data is in "dynamic disk" 
mode. I forgot about that! TestDisk is very confused by this... I did 
however find a tool which will happily undelete data from a mounted NTFS 
partition. But surprise surprise, there's nothing to undelete.)

Hey, it's not *my* fault somebody screwed up. It's not like *I* get 
yelled at. ;-)

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

Orchid XP v8 wrote:
> Are you serious?

Yeah. I just picked up a terabyte SATA drive for $120 a couple days ago, 
for a backup. The smallest thing on the shelf that wasn't a laptop drive 
was 160G, and those were all on clearance.

> Keep in mind that the servers all use SCSI, which for some reason is 10x 
> more expensive than PATA or SATA. 

Because there's much more logic on the hard drive than for the other 
interfaces.

> But surprise surprise, there's nothing to undelete.)

Yeah. NTFS tends to reuse directory entries pretty aggressively, and 
it's not like you can look at some bitmap somewhere to find erased data.

-- 
Darren New / San Diego, CA, USA (PST)

Post a reply to this message

"Orchid XP v8" <voi### [at] devnull> wrote in message 
news:490b4e5d@news.povray.org...
>
> Last time I checked, 60 GB is the smallest. (They stopped selling 40 GB a 
> while ago though.) For whatever reason, HDs never sell for less than about 
> £30. No matter what size or interface type. I guess that's what the amount 
> of metal in the case costs or something.

You can buy 80GB here, if you look hard enough. I looked for quite a while 
and bought one that wasn't much cheaper than a 160. (80 GB drive for the OS. 
More would have been wasted)

> (At the exchange rate *before* the world entered a global recession, $50 
> would be just £25 - cheaper than any HD I've ever seen on sale, ever.)
>
Amazon.co.uk lists a 500GB hard drive (western digital) at £35.75
Pricegrabber.co.uk has an 80GB hard drive for £18.00, prices and sizes 
increasing up to some 1TB drives for over £100.

Post a reply to this message

"Darren New" <dne### [at] sanrrcom> wrote in message 
news:490b643a$1@news.povray.org...
> Orchid XP v8 wrote:
>> Are you serious?
>
> Yeah. I just picked up a terabyte SATA drive for $120 a couple days ago, 
> for a backup. The smallest thing on the shelf that wasn't a laptop drive 
> was 160G, and those were all on clearance.

Sweet. I'm considering picking up a 500GB or 1TB drive while I'm over there 
for an external. I should be able to get a casing here without much trouble, 
but the drivers are definitly cheaper in the US, even considering the 
exchange rate.

Post a reply to this message

On 31-Oct-08 13:13, Invisible wrote:

> The new global backup procedure that's being drafted will mean that 
> backup tapes are stored *forever*. (With the corresponding astronomical 
> increase in expenditure on new tapes and physical storage space. But 
> hey, it's not my money.)

As long as the amount of data your company produces increases less than 
the equivalent of moore's law for harddisk space (which I am pretty sure 
it does) the expenditure will exponentially go down in time.

I first used floppy disks then switched to CD copies of my data, then 
switched to DVD, now I simply buy a new harddisk. In the video about the 
economic crash they said somewhere that people simply don't understand 
exponential growth (IIRC), well they don't. Our 24 hour ECG (holter) 
system does not make backups. People accept that because it takes a 
whopping 40MB of storage per patient. That actually costs €0.004, surely 
we can not afford that, don't we? We have entered the era where storage 
of a full CT scan is cheaper than the lab results because the latter is 
on paper. If you still have copy machines at work, something has gone 
wrong (we have and it did).

Post a reply to this message

Gail wrote:
> Sweet. I'm considering picking up a 500GB or 1TB drive while I'm over 

You'll want to hit some sort of "warehouse" type store, aka a "big box" 
store. Some place that would also sell (for example) bare motherboards 
or empty cases. I tried googling appropriate terms, but I couldn't find 
anything like that.  Again, I suggest asking the concierge at the hotel.

-- 
Darren New / San Diego, CA, USA (PST)

Post a reply to this message

"Darren New" <dne### [at] sanrrcom> wrote in message 
news:490b9767$1@news.povray.org...
> Gail wrote:
>> Sweet. I'm considering picking up a 500GB or 1TB drive while I'm over
>
> You'll want to hit some sort of "warehouse" type store, aka a "big box" 
> store. Some place that would also sell (for example) bare motherboards or 
> empty cases. I tried googling appropriate terms, but I couldn't find 
> anything like that.  Again, I suggest asking the concierge at the hotel.

There's a Fry's just outseide of Seattle. Planning on going there.

Post a reply to this message

Invisible wrote:
> Guess what? I need to recover some data.
> 
> From March.
> 
> Our backup rotation doesn't go back nearly that far. In other words, if
> there is to be *any hope* of getting that data back, I'm going to have
> to...
> 
> ...undelete files.

Your backup rotation doesn't need to go back to date the file was
created. It just needs to go back to yesterday of the date the file was
*deleted*.

If the file was deleted in March, I'd put my bet on "there's no
possibility for undelete to work".

-- 
Eero "Aero" Ahonen
   http://www.zbxt.net
      aer### [at] removethiszbxtnetinvalid

Post a reply to this message

Eero Ahonen wrote:

> Your backup rotation doesn't need to go back to date the file was
> created. It just needs to go back to yesterday of the date the file was
> *deleted*.

Correct.

> If the file was deleted in March, I'd put my bet on "there's no
> possibility for undelete to work".

Again... correct. :-D

(Actually, that's not completely true. A few of the files in the project 
were actually undeleted successfuly. But not the ones we had no other 
copies of. Obviously.)

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

Gail wrote:
> There's a Fry's just outseide of Seattle. Planning on going there.

Cool. That's a good place. I looked on their web site's "store locator" 
and I didn't see anything listed near Seattle, but maybe I'm just blind.

-- 
Darren New / San Diego, CA, USA (PST)

Post a reply to this message

"Darren New" <dne### [at] sanrrcom> wrote in message 
news:490c81db$1@news.povray.org...
> Gail wrote:
>> There's a Fry's just outseide of Seattle. Planning on going there.
>
> Cool. That's a good place. I looked on their web site's "store locator" 
> and I didn't see anything listed near Seattle, but maybe I'm just blind.
>

The store's actually in Renton. If I recall it's about an hour from Seattle 
by bus, that's if you get the schedules right. Otherwise it's an hour by bus 
and an hour waiting for the bus.
http://shop2.frys.com/isp/renton.html

Post a reply to this message

On Sat, 01 Nov 2008 21:22:24 +0200, Gail wrote:

> The store's actually in Renton. If I recall it's about an hour from
> Seattle by bus, that's if you get the schedules right.

Yeah, I'd say that's about right by bus.  By car, obviously, faster (I've 
a friend who lives in Renton who works for the city of Seattle).

Jim

Post a reply to this message

Invisible wrote:

> To summarise: It's Wild Goose time! :-D
> 
> It *must* be friday...

HAHAHAHAHAHAHAHA!!!! :-D

Alright, CHECK THIS OUT!

Apparently they burned 2 CDs of the project. *One* of them does indeed 
contain the wrong project (some of the digits are in a different order), 
but the *other* CD actually contains the correct project. So we haven't 
actually lost the data at all!

I *told* you it was a wild goose chase! :-D

Post a reply to this message

On 03-Nov-08 11:59, Invisible wrote:
> Invisible wrote:
> 
>> To summarise: It's Wild Goose time! :-D
>>
>> It *must* be friday...
> 
> HAHAHAHAHAHAHAHA!!!! :-D
> 
> Alright, CHECK THIS OUT!
> 
> Apparently they burned 2 CDs of the project. *One* of them does indeed 
> contain the wrong project (some of the digits are in a different order), 
> but the *other* CD actually contains the correct project. So we haven't 
> actually lost the data at all!
> 
> I *told* you it was a wild goose chase! :-D
I don't think you did and even if you did it is more a statement that 
belongs in advertising or astrology. I.e. one you can always you to 
prove you were right afterwards.

That doesn't change the fact that you have a serious problem. There was 
indeed a way to loose vital data even if you got lucky this time. You 
have to change the procedures immediately anyway.

Post a reply to this message

andrel wrote:

> That doesn't change the fact that you have a serious problem. There was 
> indeed a way to loose vital data even if you got lucky this time. You 
> have to change the procedures immediately anyway.

Yeah: Delete the data, and then don't ask for it to be recovered until 3 
months later. This isn't news; it's part of the design specification of 
the current backup rotation. (That is: the rotation is explicitly 
designed to recover data from up to 3 months ago. That is the design goal.)

Anyway, as it happens the procedure document is being rewritten as we 
speak. Unfortunately, it's being rewritten by Americans, so it's vague 
as hell. (The Americans like to make all their procedure documents as 
vague as possible so that they can change what they do without having to 
alter the document. Obviously, needing to alter the document is the 
entire *purpose* of having a document in writing...)

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

On Mon, 03 Nov 2008 18:15:27 +0000, Orchid XP v8 wrote:

> it's being rewritten by Americans

Maybe yours like to be vague, but certainly not all of us are.  If you 
have any doubt, I'll put you in touch with my boss (and my previous boss) 
and my VP who I have provided documentation in too much detail. ;-)

Jim

Post a reply to this message

On 03-Nov-08 22:24, Jim Henderson wrote:
> On Mon, 03 Nov 2008 18:15:27 +0000, Orchid XP v8 wrote:
> 
>> it's being rewritten by Americans
> 
> Maybe yours like to be vague, but certainly not all of us are.  If you 
> have any doubt, I'll put you in touch with my boss (and my previous boss) 
> and my VP who I have provided documentation in too much detail. ;-)
> 
It is indeed probably mostly a sign that they don't understand the 
issues. While in your case OTOH it might be ill advised showing off.

Post a reply to this message

On Mon, 03 Nov 2008 22:35:10 +0100, andrel wrote:

> It is indeed probably mostly a sign that they don't understand the
> issues. While in your case OTOH it might be ill advised showing off.

Well, yes, quite.  The last time I did this was about a month ago, maybe 
two months ago, and I actually rewrote it to make it shorter and it ended 
up longer.

It got read - in part -much later than was actually needed.

Jim

Post a reply to this message

On 04-Nov-08 6:35, Jim Henderson wrote:
> On Mon, 03 Nov 2008 22:35:10 +0100, andrel wrote:
> 
>> It is indeed probably mostly a sign that they don't understand the
>> issues. While in your case OTOH it might be ill advised showing off.
> 
> Well, yes, quite.  The last time I did this was about a month ago, maybe 
> two months ago, and I actually rewrote it to make it shorter and it ended 
> up longer.

That happens to me too. Quite often. You all should know by now, as this 
newsgroup is a major victim of it.

Post a reply to this message

On Tue, 04 Nov 2008 20:46:59 +0100, andrel wrote:

> On 04-Nov-08 6:35, Jim Henderson wrote:
>> On Mon, 03 Nov 2008 22:35:10 +0100, andrel wrote:
>> 
>>> It is indeed probably mostly a sign that they don't understand the
>>> issues. While in your case OTOH it might be ill advised showing off.
>> 
>> Well, yes, quite.  The last time I did this was about a month ago,
>> maybe two months ago, and I actually rewrote it to make it shorter and
>> it ended up longer.
> 
> That happens to me too. Quite often. You all should know by now, as this
> newsgroup is a major victim of it.

It is for me as well, though not so much recently - had too much to do at 
work....

Jim

Post a reply to this message