A flag that says "don't back up this file, ever."

Set on cache directories/files, especially those where it's faster to 
rebuild the file than to restore it, or where it's likely to be out of 
date if you restore something.

Bonus points for adding the user(s) and program(s) that use the files, 
so you can safely delete them to recover space whenever that program 
isn't running.

Because, damn, people spray those things all *over* the file system.

-- 
Darren New / San Diego, CA, USA (PST)

Post a reply to this message

Darren New wrote:
> A flag that says "don't back up this file, ever."
> 

From chattr manpage:

  A file with the ‘d’ attribute set is not candidate for backup when the
  dump(8) program is run.

(and of course any other program could read the attribute and do something
with it)

Post a reply to this message

Nicolas Alvarez wrote:
>   A file with the ‘d’ attribute set is not candidate for backup when the
>   dump(8) program is run.

Yeah. Sadly, as far as I can tell, nobody actually uses it. Windows 
programs don't even turn off the archive bit on cache files.

-- 
Darren New / San Diego, CA, USA (PST)

Post a reply to this message

On Sat, 23 Aug 2008 08:46:33 -0700, Darren New wrote:

> Nicolas Alvarez wrote:
>>   A file with the ‘d’ attribute set is not candidate for backup when
>>   the dump(8) program is run.
> 
> Yeah. Sadly, as far as I can tell, nobody actually uses it. Windows
> programs don't even turn off the archive bit on cache files.

So maybe it's not that the filesystem needs the capability, but that 
programmers need to use it?

Jim

Post a reply to this message

Jim Henderson wrote:
> So maybe it's not that the filesystem needs the capability, but that 
> programmers need to use it?

Generally, when something like this isn't used, it's due to either it 
being bolted on the side after the fact, not being ubiquitous, or the 
programs that come with the OS not doing it and thus discouraging others 
from bothering to do it.

-- 
Darren New / San Diego, CA, USA (PST)

Post a reply to this message

On Mon, 25 Aug 2008 08:40:10 -0700, Darren New wrote:

> Jim Henderson wrote:
>> So maybe it's not that the filesystem needs the capability, but that
>> programmers need to use it?
> 
> Generally, when something like this isn't used, it's due to either it
> being bolted on the side after the fact, not being ubiquitous, or the
> programs that come with the OS not doing it and thus discouraging others
> from bothering to do it.

Well, from a FAT standpoint, I doubt anyone can say the archive bit is 
"bolted on the side".

Maybe what needs to happen is for programmers to use what's there rather 
than invent something else for them to ignore, though - or worse, create 
something additional that means now there's more than one way to 
accomplish the task, making it more confusing for people writing backup 
software.

Jim

Post a reply to this message

Jim Henderson wrote:
> Well, from a FAT standpoint, I doubt anyone can say the archive bit is 
> "bolted on the side".

True. On the other hand, the original FAT file system had the semantics 
of the CP/M file system. But yes, by the time it got to the point where 
you actually did backups and had temp files you left hanging around, the 
archive bit was pretty well established. That MS's own programs didn't 
mark things as "don't archive this" (and that it really isn't a "don't 
backup" bit but more a "don't incrementally back up" bit) didn't help.

> Maybe what needs to happen is for programmers to use what's there rather 
> than invent something else for them to ignore, though 

Pretty much anything that isn't lowest-common-denominator is going to 
get ignored by the majority, I expect.  That's why it would be nice if 
the bit actually got by-default inherited by files inside directories 
with the bit set. Then you could set that bit on (say) /tmp and be done 
with it.

-- 
Darren New / San Diego, CA, USA (PST)

Post a reply to this message

On Mon, 25 Aug 2008 11:25:47 -0700, Darren New wrote:

> Jim Henderson wrote:
>> Well, from a FAT standpoint, I doubt anyone can say the archive bit is
>> "bolted on the side".
> 
> True. On the other hand, the original FAT file system had the semantics
> of the CP/M file system. But yes, by the time it got to the point where
> you actually did backups and had temp files you left hanging around, the
> archive bit was pretty well established. 

True.

> That MS's own programs didn't
> mark things as "don't archive this" (and that it really isn't a "don't
> backup" bit but more a "don't incrementally back up" bit) didn't help.

I do recall that MS' DOS-based backup utility (I forget what it was 
called) did actually observe the archive bit.  The apps didn't use it 
often enough, though; it was typically used by the backup utilities to 
flag that a file had been archived by a backup system, not as a means of 
skipping files you didn't want backed up (but a little creative thinking 
would've come to the conclusion you could use it for both).

>> Maybe what needs to happen is for programmers to use what's there
>> rather than invent something else for them to ignore, though
> 
> Pretty much anything that isn't lowest-common-denominator is going to
> get ignored by the majority, I expect.  That's why it would be nice if
> the bit actually got by-default inherited by files inside directories
> with the bit set. Then you could set that bit on (say) /tmp and be done
> with it.

True.  And I agree it would be nice if it recursed; considering the speed 
of computers of the day, though, I wonder how practical it would've been 
to implement - true you could do a dynamic evaluation, but MS so likes 
doing the "traverse and stamp every object", as evidenced by the horribly 
broken behaviour in AD version 1 when assigning rights through a 
filesystem.   Thank goodness they fixed that behaviour.

Jim

Post a reply to this message

Jim Henderson wrote:
> I do recall that MS' DOS-based backup utility (I forget what it was 
> called) did actually observe the archive bit. 

All the backup utilities did (and do) pay attention to it. But the flag 
automatically gets turned on whenever you open a file for writing, so 
its use is primarily for differential backups. You do a full backup and 
while doing so turn off the bit, then later you can back up just the 
files with the bit set.

> skipping files you didn't want backed up (but a little creative thinking 
> would've come to the conclusion you could use it for both).

It would only skip files you didn't want *incrementally* backed up.  A 
full backup would still copy the file, and there's no semantics (in UNIX 
or Windows file systems at least) for "this is a temp file" sort of 
thing, so don't ever back it up, or at least throw it away if you need 
the room.

> True.  And I agree it would be nice if it recursed; considering the speed 
> of computers of the day, though, I wonder how practical it would've been 
> to implement - true you could do a dynamic evaluation, but MS so likes 
> doing the "traverse and stamp every object", as evidenced by the horribly 
> broken behaviour in AD version 1 when assigning rights through a 
> filesystem.   Thank goodness they fixed that behaviour.

That must have been a while ago. All the MS file systems I've ever 
looked at had heritable privileges. (Given they had privileges to start 
with, of course.) I'm not real sure where AD comes into it.

-- 
Darren New / San Diego, CA, USA (PST)

Post a reply to this message

On Mon, 25 Aug 2008 11:44:47 -0700, Darren New wrote:

> All the backup utilities did (and do) pay attention to it. But the flag
> automatically gets turned on whenever you open a file for writing, so
> its use is primarily for differential backups. You do a full backup and
> while doing so turn off the bit, then later you can back up just the
> files with the bit set.

Oh, right, I forgot about that bit with it getting reset when a file is 
opened for writing.

> It would only skip files you didn't want *incrementally* backed up.  A
> full backup would still copy the file, and there's no semantics (in UNIX
> or Windows file systems at least) for "this is a temp file" sort of
> thing, so don't ever back it up, or at least throw it away if you need
> the room.

Well, arguably, there is - you put it in /tmp - most programs do that.  
Then you just exclude the directory from your backups.

> That must have been a while ago. All the MS file systems I've ever
> looked at had heritable privileges. (Given they had privileges to start
> with, of course.) I'm not real sure where AD comes into it.

Demonstration I saw showed it pegging the CPU at 100% for nearly 40 
minutes because the rights were stamped on each object.  Come to think, 
it wasn't the filesystem - it had to do with assigning rights in AD 
itself, not to the filesystem.  It's been a few years.

Jim

Post a reply to this message