Guess what? I need to recover some data.

 From March.

Our backup rotation doesn't go back nearly that far. In other words, if 
there is to be *any hope* of getting that data back, I'm going to have to...

...undelete files.

 From the server's harddrive. o_O

Now, taking a disk image of a 64 MB flash drive and playing with it is 
one thing. But one does not simply walk into Mordor - er, I mean, one 
does not simply copy a 103 GB RAID array. It's far too huge for there to 
be anywhere to copy it *to*. And finding anything will be fun; any idea 
how many thousands of files are on there *now*? Never mind the deleted ones.

Oh, and it's NTFS, not FAT. Oh yeah, and the data I want to find is not 
in any "common" file format; it's a proprietry format used by our lab 
software.

And of course, on top of all that, what are the chances of actually 
finding deleted files on a busy server? How many minutes do you think a 
file survives before being completely overwritten?

To summarise: It's Wild Goose time! :-D

It *must* be friday...

Post a reply to this message

Invisible <voi### [at] devnull> wrote:
> Our backup rotation doesn't go back nearly that far.

  I think this is the WTF in this post.

  What good is a backup if you are going to destroy backups which are
"too old"? Do they assume that nobody will ever want to restore a file
which was deleted more than a certain amount of time ago?

  Sure, after a certain time it may not be necessary to store daily
backups, but IMO after a certain time eg. weekly backups could be
preserved, and after a longer time eg. monthly backups. You never know
what you may want to restore.

  Besides, if storage space is an issue, haven't they heard of incremental
backups? It is possible to store eg. daily backups (which are fully
restorable to any of the days in question) of an entire month in a way
that requires significantly less space than 30*(size of data being backed up).

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:
> Invisible <voi### [at] devnull> wrote:
>> Our backup rotation doesn't go back nearly that far.
> 
>   I think this is the WTF in this post.
> 
>   What good is a backup if you are going to destroy backups which are
> "too old"? Do they assume that nobody will ever want to restore a file
> which was deleted more than a certain amount of time ago?

Basically... yes, that's the assumption. They assume that if somebody 
deleted a file accidentally, they'll quickly realise and request to have 
the file restored.

Unfortuantely, what happened here is that somebody burned some files to 
CD and then deleted them, and 8 months later we figure out that actually 
they burned the wrong files - yet deleted the right ones. (In fairness, 
we're talking about projects which have names such as 45748-54867-15486. 
Very easy to muddle them up!)

This exact thing has happened once before. I guess two problems in 6 
years is two too many...

The new global backup procedure that's being drafted will mean that 
backup tapes are stored *forever*. (With the corresponding astronomical 
increase in expenditure on new tapes and physical storage space. But 
hey, it's not my money.)

Unfortunately, the new procedure document is also vague as hell and 
pretty hard to comprehend. Clearly the guy who wrote it understands what 
he means... but I don't.

Post a reply to this message

Invisible <voi### [at] devnull> wrote:
> The new global backup procedure that's being drafted will mean that 
> backup tapes are stored *forever*. (With the corresponding astronomical 
> increase in expenditure on new tapes and physical storage space. But 
> hey, it's not my money.)

  As I said, that doesn't need to be the case. For example daily backups
worth an entire month can be stored in an incremental backup system. This
way only one backup per month has to be stored permanently, but it will
still allow restoring any single day of that month, and the size of the
backup will be only somewhat larger than the size of the entire system
being backed up. (Basically it's the entire system plus modifications
during that month.)

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:
> Invisible <voi### [at] devnull> wrote:
>> The new global backup procedure that's being drafted will mean that 
>> backup tapes are stored *forever*. (With the corresponding astronomical 
>> increase in expenditure on new tapes and physical storage space. But 
>> hey, it's not my money.)
> 
>   As I said, that doesn't need to be the case. For example daily backups
> worth an entire month can be stored in an incremental backup system. This
> way only one backup per month has to be stored permanently, but it will
> still allow restoring any single day of that month, and the size of the
> backup will be only somewhat larger than the size of the entire system
> being backed up. (Basically it's the entire system plus modifications
> during that month.)

The plan is to store only the monthly full backup tapes - so that's only 
actually 12 tapes per year. Of course, in 20 years' time we'll have 240 
of the suckers - not to mention the fact that in 20 years' time, people 
probably won't be using LTO1 any more so there probably won't be any 
equipment that can still read the tapes thus making their retention 
utterly pointless. But in the main it's a sound system...

Post a reply to this message

Invisible wrote:
> Warp wrote:
>> Invisible <voi### [at] devnull> wrote:
>>> The new global backup procedure that's being drafted will mean that 
>>> backup tapes are stored *forever*. (With the corresponding 
>>> astronomical increase in expenditure on new tapes and physical 
>>> storage space. But hey, it's not my money.)
>>
>>   As I said, that doesn't need to be the case. For example daily backups
>> worth an entire month can be stored in an incremental backup system. This
>> way only one backup per month has to be stored permanently, but it will
>> still allow restoring any single day of that month, and the size of the
>> backup will be only somewhat larger than the size of the entire system
>> being backed up. (Basically it's the entire system plus modifications
>> during that month.)
> 
> The plan is to store only the monthly full backup tapes - so that's only 
> actually 12 tapes per year. Of course, in 20 years' time we'll have 240 
> of the suckers - not to mention the fact that in 20 years' time, people 
> probably won't be using LTO1 any more so there probably won't be any 
> equipment that can still read the tapes thus making their retention 
> utterly pointless. But in the main it's a sound system...


Back when I was helping to build cars I deleted a file and only noticed 
that it was gone something like 6 months later.

Luckily, their rotation kept some very old tapes.

I don't remember specifically, but their rotation was something like:

daily for a month
after the month, just keep the weekly tapes
After 3 months keep only the monthly - permanently

Or something roughly like that.
It won't catch everything, but it catches most things.



Tom

Post a reply to this message

Invisible wrote:

> To summarise: It's Wild Goose time! :-D
> 
> It *must* be friday...

Hey, neato:

http://ntfsundelete.com/

It works as well! ;-)


But in the case of today's missing files... guess what?... they're not 
there any more. *Such* a surprise! ;-)

Post a reply to this message

Invisible wrote:
> does not simply copy a 103 GB RAID array. It's far too huge for there to 
> be anywhere to copy it *to*.

Huh?  I can't even *buy* a hard drive that small any more, except on a 
laptop disk. A 160G disk is like $50 at the local shop.  I have 10x that 
much space stuffed in my desktop machine.

> To summarise: It's Wild Goose time! :-D

Sounds like it!

-- 
Darren New / San Diego, CA, USA (PST)

Post a reply to this message

Darren New wrote:
> Invisible wrote:
>> does not simply copy a 103 GB RAID array. It's far too huge for there 
>> to be anywhere to copy it *to*.
> 
> Huh?  I can't even *buy* a hard drive that small any more, except on a 
> laptop disk. A 160G disk is like $50 at the local shop.  I have 10x that 
> much space stuffed in my desktop machine.

Are you serious?

Last time I checked, 60 GB is the smallest. (They stopped selling 40 GB 
a while ago though.) For whatever reason, HDs never sell for less than 

the amount of metal in the case costs or something.

(At the exchange rate *before* the world entered a global recession, $50 


Keep in mind that the servers all use SCSI, which for some reason is 10x 
more expensive than PATA or SATA. None of the servers has more than 200 
GB of storage online. (Some of them do have a single hot spare.) Most of 
the drives are 36 GB each. Even on the brand new Dell rack-mount things 
with the hot-swap brive bays.

>> To summarise: It's Wild Goose time! :-D
> 
> Sounds like it!

Heh. Yeah, well, a challenge can be fun, right?

(Ooo, did I mention? The drive holding the data is in "dynamic disk" 
mode. I forgot about that! TestDisk is very confused by this... I did 
however find a tool which will happily undelete data from a mounted NTFS 
partition. But surprise surprise, there's nothing to undelete.)

Hey, it's not *my* fault somebody screwed up. It's not like *I* get 
yelled at. ;-)

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

Orchid XP v8 wrote:
> Are you serious?

Yeah. I just picked up a terabyte SATA drive for $120 a couple days ago, 
for a backup. The smallest thing on the shelf that wasn't a laptop drive 
was 160G, and those were all on clearance.

> Keep in mind that the servers all use SCSI, which for some reason is 10x 
> more expensive than PATA or SATA. 

Because there's much more logic on the hard drive than for the other 
interfaces.

> But surprise surprise, there's nothing to undelete.)

Yeah. NTFS tends to reuse directory entries pretty aggressively, and 
it's not like you can look at some bitmap somewhere to find erased data.

-- 
Darren New / San Diego, CA, USA (PST)

Post a reply to this message