On Wed, 18 May 2011 22:30:32 +0100, Orchid XP v8 wrote:

> I gather that Linux only just got the ability to notify you if a file
> changes. (At least modern Linux desktops manage to *use* this
> information correctly, which is more than Windows Explorer manages...)

The notification API has been around for several years now.

As for ACLs, it depends on the tools.  I've seen proper ACLs overlaid on 
*nix filesystems for several years now (as well) and it works properly.  
Sometimes you have to map the bitmask rights to something in the new ACL 
filesystem, but having MAC (ie, mandatory access controls) can certainly 
help enforce permissions that would otherwise not be done.

Jim

Post a reply to this message

Am Wed, 18 May 2011 15:28:44 -0700 schrieb Darren New:

> On 5/18/2011 14:50, Florian Pesth wrote:
>> Sure that there is no call for saying "Lock this file definitely" and
>> the other lock mechanisms are just kept for POSIX compability?
> 
> Yes. The basic problem is that none of the lock mechanisms actually work
> reliably in all situations.
> 

How about this:

http://www.hackinglinuxexposed.com/articles/20030623.html

Granted, the described problem, that if a program opens before and writes 
after the lock (therefore overwriting the file) persists, but I don't 
think this is really a locking problem or is it?

>> I think this is also older...
> 
> One problem is that there's at least 3 ways of doing this in Linux, all
> three relatively recent.
> 
> One would give you a signal when a file in the one single directory
> you're watching (maybe just the cwd) changed. Not very handy except for
> the refreshing of a window in a file manager.
> 
> One would do some sort of something or other with messages sent to you
> somehow I forget when anything under a particular file tree changed,
> IIRC. It was poor because, IIRC, you had to block to wait for the
> messages or something.
> 
> The latest one, done right, lets you open a file handle, writing to the
> handle says what you're waiting to see change, reading the (selectable)
> handle lets you know when something there changed and what it was.

Ok, thanks. I just was thinking of tailf which advertises with hard disk 
spin down when the file does not change - so it definitely does not poll 
and is around since at least 2003. I just searched for it and it seems 
that there are two mechanisms: dnotify and the more recently inotify. Now 
they might not do what you like to do with them (sorry I didn't 
understand what you wanted to do exactly) but for being notified when a 
file changes (which is what Andrew was talking about) even dnotify seems 
to be fine (and is around since kernel 2.4.). Has your problem something 
to do with blocking / non-blocking access of this information?

Post a reply to this message

On 5/18/2011 15:53, Florian Pesth wrote:
> Granted, the described problem, that if a program opens before and writes
> after the lock (therefore overwriting the file) persists, but I don't
> think this is really a locking problem or is it?

That's not the only problem. The other problem is, as he said, taking down 
an NFS server from user mode, for example.

Here's the article I was thinking of:

http://0pointer.de/blog/projects/locking

http://0pointer.de/blog/projects/locking2

Contrast with (for example) mainframe locking, where advisory locks usually 
don't even use files, and mandatory locks are generally properly enforced 
because large amounts of peoples' money depends on them working. (Which is 
why all the stuff on Linux where peoples' money *does* depend on locking 
working, like an RDBMS, you access all the relevant files via a socket to a 
single process that does the locking without going through the file system 
to do so.)

And in the 1.x kernels (and maybe later), locking could also lead to 
starvation in Linux. It's all getting cleaned up over time, of course, but 
us old farts remember this stuff and are leery about anything new. Linux has 
it's own "wait for version 3 of anything MS releases".

Here's an exercise for you:
Write a backup program under Linux that correctly backs up files, even ones 
you might be editing, without ever losing both the new and old versions of a 
file.  I don't think that's possible.  I.e., I want to run a backup program 
using normal file system access (rather than reading raw disk data) that 
will reliably copy a local directory full of files, even if you're changing 
them. There's several ways of doing this on Windows. I don't know any way of 
doing it on Linux.

> Ok, thanks. I just was thinking of tailf which advertises with hard disk
> spin down when the file does not change

The hard disk spins down when the disk doesn't generate interrupts for some 
length of time. That doesn't mean there's no file activity. It just means 
all the file activity is using the buffers.

The three interfaces I remember are one that generates a SIGIO when your 
registered directory changes (which apparently nobody even talks about any 
more, but it was quite a ways back), dnotify (which required a separate file 
handle for each directory and communicates using signals, which means you 
can't really use threads to deal with it and you can't really be doing 
something else at the same time in the same process because a signal handler 
can't do I/O so you wind up polling to see if a signal happened), and 
inotify which finally got it right.

 > - so it definitely does not poll

That doesn't follow. The fact the disk spins down just means the kernel 
doesn't have to look at the disk to know the file hasn't changed.

> Has your problem something
> to do with blocking / non-blocking access of this information?

I don't really have a problem. We were just comparing areas in which Windows 
seems to do a superior job to Linux (or at least Linux as of not too long ago).

The dnotify is just an exceptionally kludgey interface, requiring one handle 
for each directory, "the problem of hard links to files has been ignored" 
(my favorite gripe about all UNIX kernel functionality), and requiring you 
to open each nested directory explicitly along with reading the entire 
directory every time a file changes to see what changed. That's why they 
replaced it. :-)

inotify is about as close as you're likely to get, letting you add the 
sockets to a select() list so you can (for example) wait for data from the 
keyboard while polling for directory changes.

None of them provide what Windows provides, which includes notifying you 
about changes since last you asked, even if you weren't running when the 
files changed.

-- 
Darren New, San Diego CA, USA (PST)
   "Coding without comments is like
    driving without turn signals."

Post a reply to this message

>> There might be a tiny few things which really are hard-coded to
>> Administrator (the user) and/or Administrators (the group), but I haven't
>> run into that.
>
> The only one I know of for sure is manipulating file system stuff.
> Partitioning a disk, reading the UNC, marking a file system dirty, etc.
> I don't know what privilege it is - maybe you actually can assign it.

Control Panel > Administrative Tools > Local Security Settings

Left pane: Local Policies > User Rights Assignment

Right pane says:

   Perform volume maintenance tasks: Administrators

Presumably this covers things like mounting/unmounting filesystems, 
creating partitions, formatting them, and so forth. (The frustrating 
thing about Windows is that you have to *guess*; this isn't written down 
anywhere.) If you change the setting here, you can allow other users to 
do this.

Also in there is "take ownership of files". In other words, bypass ACLs.

Note that everything in this pane refers to "Administrators" (the user 
group, not "Administrator" the user).

If you flip the left page to Local Policies > Security Options, there's 
more fun stuff to play with, for example:

   Devices: Allowed to format and eject removable media: Administrators

>>> That is the basic problem with complex permissions. :-)
>> I assume that comment is directed at AD.
>
> Well, all complex permissions systems.

As far as I can see, the existence of 3 different kinds of groups is an 
artefact of the way AD is implemented internally, and how it maintains 
backwards compatibility. Which is frustrating, because I don't care how 
it works, and I don't need backwards compatibility. I just want 
something comprehensible.

>> ...except that it's still implemented the old way.
>
> No it isn't. Well, maybe with XP, but I don't think that's the case any
> more.

And who's using anything newer than XP? I mean, really.

>> Which means that changing the parent's permissions has the intended
>> effect,
>> but if the parent has 80,000,000 children, it still takes forever to
>> set the
>> permissions, because you literally have to touch every child and
>> update its
>> ACL on disk.
>
> Maybe if you're doing it thru the explorer?

How else would you change a file's permissions?

>> Oh yeah. TRWTF is 8.3 names.
>
> What about them?

They still exist. And grown up programs sometimes end up using them.

>> Like I say, it's a shame you can't use this feature to launch stuff
>> programmatically. (At least, not without linking to C.)
>
> Well, what are you going to launch? You can't launch *anything* without
> linking to some sort of C code.

I mean, if you want to launch a program from the CLI, you just type its 
name. But there isn't a command like "open Blue.ogg" which will fire up 
the default application and load the named file. If you want to do this, 
you have to write C code to query the registry and so forth.

>> how it's implemented. (I have a sinking feeling it's a type database for
>> each file manager, rather than something that arbitrary applications
>> can use.)
>
> I wouldn't be surprised.

Actually, you know what? There's probably one type database for KDE, 
which only KDE applications can access using the KDE application 
libraries, and then a second, completely independent type database for 
GNOME, which only GNOME applications can access using the GNOME 
libraries. And then half a dozen CLI applications which use their own 
built-in mechanism.

And I've seen at least one Debian package which claims to contain a 
program for analysing file contents to determine their actual type...

In short, it's an incoherent nightmare, just like almost everything else 
in Unix.

>>> Even better, of course, would be having it in the actual metadata of the
>>> file, like the Mac's OS tried to do, except they did it before MIME was
>>> around, so it didn't work out as well as it might.
>>
>> Is that the whole resource fork / data fork thing? I never heard how that
>> actually works. (Other than "it doesn't".)
>
> No, it was the Creator/Type thing, each a four-character code. So a text
> file made by MS Word might have a creator/type or MSWD/TEXT or something.
>
> Mac files had two forks. One was the data fork, which is just like a
> UNIX file. The other is the resource fork, which had structured blobs,
> each defined by a type and a size, maybe a name, etc. So what you see in
> a Windows executable as a "resource" file was actually the resource fork
> of a Mac application or data file.

I see... I think! o_O

Post a reply to this message

On 5/19/2011 1:35, Invisible wrote:
> Perform volume maintenance tasks: Administrators

OK, thanks!

> Also in there is "take ownership of files". In other words, bypass ACLs.

Take ownership isn't exactly bypass ACLs.

>> Maybe if you're doing it thru the explorer?
> How else would you change a file's permissions?

Via the API that explorer uses.

>>> Oh yeah. TRWTF is 8.3 names.
>> What about them?
> They still exist. And grown up programs sometimes end up using them.

Oh, well, yes. Backwards compatibility and all.

> I mean, if you want to launch a program from the CLI, you just type its
> name. But there isn't a command like "open Blue.ogg" which will fire up the
> default application and load the named file.

start index.html

The command is "start".

> Actually, you know what? There's probably one type database for KDE,

Originally it was Apache that needed it, since Apache had to deliver mime 
types based on file extensions or content.

> In short, it's an incoherent nightmare, just like almost everything else in
> Unix.

A strength and a weakness.

>> Mac files had two forks. One was the data fork, which is just like a
>> UNIX file. The other is the resource fork, which had structured blobs,
>> each defined by a type and a size, maybe a name, etc. So what you see in
>> a Windows executable as a "resource" file was actually the resource fork
>> of a Mac application or data file.
>
> I see... I think! o_O

It was a nice idea. You could install fonts into the global space, or into 
the executable for the word processor, or into an individual document that 
used that font. You could localize a program after it was completely 
finished just by adding resources to the executable. Stuff like that.

Oh, and chunks of code were also resources, so by calling code chunks, you'd 
wind up loading dynamically the appropriate code resources, so you got 
overlays for free, basically.

-- 
Darren New, San Diego CA, USA (PST)
   "Coding without comments is like
    driving without turn signals."

Post a reply to this message

On 5/19/2011 8:14, Darren New wrote:
>>> Maybe if you're doing it thru the explorer?
>> How else would you change a file's permissions?
>
> Via the API that explorer uses.

Or the CLI for that matter.

-- 
Darren New, San Diego CA, USA (PST)
   "Coding without comments is like
    driving without turn signals."

Post a reply to this message

On 19/05/2011 16:14, Darren New wrote:
> On 5/19/2011 1:35, Invisible wrote:
>> Perform volume maintenance tasks: Administrators
>
> OK, thanks!
>
>> Also in there is "take ownership of files". In other words, bypass ACLs.
>
> Take ownership isn't exactly bypass ACLs.

No. But if you can take ownership of a file, you can give yourself 
permission to access it, even if you didn't have that permission to 
start with. In other words, it allows you to bypass ACLs.

(Which is just as well, because our project management application 
occasionally deletes the ACL list for files, which makes them rather 
difficult to work with...)

>>> Maybe if you're doing it thru the explorer?
>> How else would you change a file's permissions?
>
> Via the API that explorer uses.

Well, yeah, if I were a C programmer that might be an option. :-P

>>>> Oh yeah. TRWTF is 8.3 names.
>>> What about them?
>> They still exist. And grown up programs sometimes end up using them.
>
> Oh, well, yes. Backwards compatibility and all.

Backwards compatibility = EVIL!

>> I mean, if you want to launch a program from the CLI, you just type its
>> name. But there isn't a command like "open Blue.ogg" which will fire
>> up the default application and load the named file.
>
> start index.html
>
> The command is "start".

And that works for any file type which Windows has a file association for?

>> Actually, you know what? There's probably one type database for KDE,
>
> Originally it was Apache that needed it, since Apache had to deliver
> mime types based on file extensions or content.

Well, no, Apache doesn't need to know what to open it with at all. It 
just needs to know what MIME type to claim it is. (Which might 
potentially vary on a file-by-file basis in a way unrelated to the 
extension on the name...)

>> In short, it's an incoherent nightmare, just like almost everything
>> else in Unix.
>
> A strength and a weakness.

The Unix philosophy: Every tool should do one job, and do it well.

Unfortunately this leads to 25 different tools doing the same job 
equally poorly in incompatible ways. Hence why your typical Linux distro 
has (for example) a dozen different packages which implement MD5.

>> I see... I think! o_O
>
> It was a nice idea. You could install fonts into the global space, or
> into the executable for the word processor, or into an individual
> document that used that font. You could localize a program after it was
> completely finished just by adding resources to the executable. Stuff
> like that.

Still not really seeing why you need this implemented at the filesystem 
level, but yeah...

> Oh, and chunks of code were also resources, so by calling code chunks,
> you'd wind up loading dynamically the appropriate code resources, so you
> got overlays for free, basically.

Do overlays matter any more? (Clearly they did once...)

Post a reply to this message

On 5/19/2011 8:41, Invisible wrote:
> No. But if you can take ownership of a file, you can give yourself
> permission to access it, even if you didn't have that permission to start
> with. In other words, it allows you to bypass ACLs.

Sure, but then you own the file. I.e., you get caught. :-)  And you can't 
put it back the way it was, either.

>> The command is "start".
> And that works for any file type which Windows has a file association for?

As far as I know, yes.

> Well, no, Apache doesn't need to know what to open it with at all. It just
> needs to know what MIME type to claim it is.

Well, yes, that too. Which is why the desktops probably have different files.

> (Which might potentially vary
> on a file-by-file basis in a way unrelated to the extension on the name...)

Not in the original apache. :-)

> Still not really seeing why you need this implemented at the filesystem
> level, but yeah...

You don't *need* anything at all at the filesystem level. You don't need 
directories, or locking, or ACLs. FORTH works fine without any of that.

That said, there's a benefit to having everyone use the same abstractions by 
building them into the file system. When they're baked into the OS, everyone 
does it the same way (so they're compatible) and nobody needs to have code 
to deal with it in every application. The idea that "we don't need that in 
the OS because we can implement it again and again in every application that 
needs it" (especially when it's very close to every application that needs 
it) is a broken philosophy.

As another example, the mainframe I worked on had keyed files. That meant 
you could put a number on each line, and the editor could change individual 
lines of the files without rewriting the entire file or even reading the 
whole thing into memory. If you read the file "sequentially", you got it 
back in line-number order. So you had a bunch of editors that physically 
stored basically sorted hash tables (b-trees) of text lines, and a bunch of 
compilers each of which could read what the editor wrote in the order the 
editor expected them to read it. Now, if I want to edit a file bigger than 
memory, I need to use sed instead of vi.

> Do overlays matter any more? (Clearly they did once...)

Not unless you're using a machine that can't handle demand paging. 
Relatively few such machines nowadays don't have enough RAM to run 
everything at once, but there are still some around. Stuff like credit card 
terminals, for example.

-- 
Darren New, San Diego CA, USA (PST)
   "Coding without comments is like
    driving without turn signals."

Post a reply to this message

On 19/05/2011 05:03 PM, Darren New wrote:
> On 5/19/2011 8:41, Invisible wrote:
>> No. But if you can take ownership of a file, you can give yourself
>> permission to access it, even if you didn't have that permission to start
>> with. In other words, it allows you to bypass ACLs.
>
> Sure, but then you own the file. I.e., you get caught. :-) And you can't
> put it back the way it was, either.

Well, yeah, if you're worried about catching people, turn on file access 
auditing. (Something else which, AFAIK, Linux doesn't have.) I'm just 
saying people with this priviledge can access stuff they shouldn't be 
able to look at.

>> Still not really seeing why you need this implemented at the filesystem
>> level, but yeah...
>
> You don't *need* anything at all at the filesystem level. You don't need
> directories, or locking, or ACLs. FORTH works fine without any of that.
>
> That said, there's a benefit to having everyone use the same
> abstractions by building them into the file system. When they're baked
> into the OS, everyone does it the same way (so they're compatible) and
> nobody needs to have code to deal with it in every application. The idea
> that "we don't need that in the OS because we can implement it again and
> again in every application that needs it" (especially when it's very
> close to every application that needs it) is a broken philosophy.

Fair enough. For example, under Linux there are apparently at least 3 
different [incompatible] ways to access the sound hardware. Isn't that 
wonderful?

>> Do overlays matter any more? (Clearly they did once...)
>
> Not unless you're using a machine that can't handle demand paging.
> Relatively few such machines nowadays don't have enough RAM to run
> everything at once, but there are still some around. Stuff like credit
> card terminals, for example.

Right. So for desktop computers it's a non-issue?

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

On 5/19/2011 11:18, Orchid XP v8 wrote:
> people with this priviledge can access stuff they shouldn't be able to look at.

Sure. But that's different from bypassing the ACLs. :-)

If you really want to prevent that, encrypt the file. (Something else Linux 
doesn't have quite the same.)

> Fair enough. For example, under Linux there are apparently at least 3
> different [incompatible] ways to access the sound hardware. Isn't that
> wonderful?

Actually, I think there's one way to access the sound hardware: /dev/audio. 
There's a variety of servers acting as intermediaries, because for the most 
part, while UNIX is a multi-user system, they never quite got the whole 
"login" concept into the system.

> Right. So for desktop computers it's a non-issue?

Pretty much, yes. As I said, anything with demand paging is probably not 
going to bother supporting overlays.

-- 
Darren New, San Diego CA, USA (PST)
   "Coding without comments is like
    driving without turn signals."

Post a reply to this message