Jim Henderson wrote:
> On Sat, 22 Nov 2008 09:46:38 -0800, Darren New wrote:
> 
>> Jim Henderson wrote:
>>> why would you not be able to update it consistently?
>> You write the .h file. While trying to write the .so file, you A) don't
>> have permissions to do so,
> 
> The updater requires root rights to run, so it has permissions to do that.

Not always. See network-mounted file systems, file systems mounted 
read-only, write-protect tabs, etc.

>> B) find that it's open by someone else as a shared text segment, 
> 
> I don't believe that would matter,

You're mistaken.

> again, it's the inode that's open, and 
> when the file is overwritten a new inode is created and the old one is 
> destroyed.

Only if you unlink the old file then creat() a new one.  If you actually 
open the current file for writing (and it's executing or sticky in swap, 
and it has a shared text segment), you get an error, even as root.

>> C) run out of disk space,
> 
> That would create other problems as well

Yes? So?  We know it's bad and you should avoid it, yes.

Note that nobody expects "I ran out of disk space" (or "I ran out of 
file handles or swap space") to result in their root directory getting 
truncated, for example. So it's not so bad that it should leave other 
parts of your system in an inconsistent state.

I also left out "Linux out-of-memory killer randomly decided to nuke my 
process because someone else grabbed a bunch of memory."

>> D) have your process killed,
>> E) have the power fail,
>> ....
> 
> Yeah, those could happen and could introduce problems - so you just 
> reinstall the packages and that makes things consistent.

And until you do, things are broken. Which is kind of my point.

It's also not an orthogonal solution. Everyone has to reinvent the wheel 
for themselves when trying to update multiple files consistently.

And if the package manager updates the database *before* it updates the 
files, you might never know it.

-- 
Darren New / San Diego, CA, USA (PST)

Post a reply to this message

On Sat, 22 Nov 2008 21:42:12 -0800, Darren New wrote:

> Jim Henderson wrote:
>> On Sat, 22 Nov 2008 09:46:38 -0800, Darren New wrote:
>> 
>>> Jim Henderson wrote:
>>>> why would you not be able to update it consistently?
>>> You write the .h file. While trying to write the .so file, you A)
>>> don't have permissions to do so,
>> 
>> The updater requires root rights to run, so it has permissions to do
>> that.
> 
> Not always. See network-mounted file systems, file systems mounted
> read-only, write-protect tabs, etc.

You'd have to configure things pretty weird to get part of the relevant 
filesystems mounted in a way that was inconsistent with the requirements 
- such as being able to update development header files but not shared 
libraries.  You'd almost have to be trying to create a situation where 
the updater doesn't work - and sure, you could do that, but what would be 
the point other than to break things?

>>> B) find that it's open by someone else as a shared text segment,
>> 
>> I don't believe that would matter,
> 
> You're mistaken.

Well, it's something I've never run into, and I've only been running 
Linux for about 12 years...

>> again, it's the inode that's open, and when the file is overwritten a
>> new inode is created and the old one is destroyed.
> 
> Only if you unlink the old file then creat() a new one.  If you actually
> open the current file for writing (and it's executing or sticky in swap,
> and it has a shared text segment), you get an error, even as root.

Well, the installer does just that - it unlinks and creates new files, 
near as I can tell.  But even in a case like that, the updater will error 
and say that a file it needs to access can't be updated, and it will give 
you the option of aborting or attempting to continue - same kind of error 
message when it tries to auto-refresh after making a network change 
(which I think is stupid, because it takes the network connection down 
and ALWAYS errors once).

>>> C) run out of disk space,
>> 
>> That would create other problems as well
> 
> Yes? So?  We know it's bad and you should avoid it, yes.

My point is that you would notice this probably before running out of 
disk space.  Thinking about it, though, I have run into this before, and 
the integrity didn't get messed up.

> Note that nobody expects "I ran out of disk space" (or "I ran out of
> file handles or swap space") to result in their root directory getting
> truncated, for example. So it's not so bad that it should leave other
> parts of your system in an inconsistent state.
> 
> I also left out "Linux out-of-memory killer randomly decided to nuke my
> process because someone else grabbed a bunch of memory."
> 
>>> D) have your process killed,
>>> E) have the power fail,
>>> ....
>> 
>> Yeah, those could happen and could introduce problems - so you just
>> reinstall the packages and that makes things consistent.
> 
> And until you do, things are broken. Which is kind of my point.
> 
> It's also not an orthogonal solution. Everyone has to reinvent the wheel
> for themselves when trying to update multiple files consistently.
> 
> And if the package manager updates the database *before* it updates the
> files, you might never know it.

If you never know it, though, then things are working as expected - if 
they're not, you can run a comparison of the database to the filesystem 
using the RPM options to do so.

Jim

Post a reply to this message

Jim Henderson wrote:
> I don't believe that would matter, again, it's the inode that's open, and
> when the file is overwritten a new inode is created and the old one is
> destroyed.

Uh, it's not. Depends on your definition of "overwritten".

Open an existing file (inode), write into it. Did the inode change?

Post a reply to this message

Jim Henderson wrote:
> You'd have to configure things pretty weird to get part of the relevant 
> filesystems mounted in a way that was inconsistent with the requirements 

Sure.  However, you're looking at it from the narrow context of one 
particular program under Linux.  Being able to deal with such problems 
in a generic way is a benefit.

>>>> B) find that it's open by someone else as a shared text segment,
>>> I don't believe that would matter,
>> You're mistaken.
> 
> Well, it's something I've never run into, and I've only been running 
> Linux for about 12 years...

http://www.derkeiler.com/Mailing-Lists/FreeBSD-Security/2002-07/10992.html

It's not really common, unless you're trying to update executables while 
they're running. It's not hard to make it happen. We've been over that 
here, a few weeks ago.

> Well, the installer does just that - it unlinks and creates new files, 
> near as I can tell.  

Most likely. Probably because (d'oh) you can't write to a file that's 
being executed, yes? ;-)

>>>> C) run out of disk space,
>>> That would create other problems as well
>> Yes? So?  We know it's bad and you should avoid it, yes.
> 
> My point is that you would notice this probably before running out of 
> disk space.

Unless the update is what runs you out of space, or (just maybe) there's 
more than one person using the computer, like? Or some background job 
that's maybe doing something like receiving email?

>> And if the package manager updates the database *before* it updates the
>> files, you might never know it.
> 
> If you never know it, though, then things are working as expected

And if "as expected" means "still has the security holes in it that the 
update was supposed to patch", then yes, that's as expected. Not what 
you want, but as expected. :-)


But yes, if you're only talking about package management per se (which 
is indeed where we started) then there are obviously solutions in place. 
It's just a shame they didn't generalize it so you can make it work for 
other people too, or over the network, or in spite of failures, etc.

-- 
Darren New / San Diego, CA, USA (PST)

Post a reply to this message

On Sun, 23 Nov 2008 12:19:30 -0800, Darren New wrote:

> Jim Henderson wrote:
>> You'd have to configure things pretty weird to get part of the relevant
>> filesystems mounted in a way that was inconsistent with the
>> requirements
> 
> Sure.  However, you're looking at it from the narrow context of one
> particular program under Linux.  Being able to deal with such problems
> in a generic way is a benefit.

Well, sure, but dealing with every possible and conceivable exception 
gets into areas where even in professional development it's decided not 
to go.  It's good to deal with common exceptions, but more obscure and 
unlikely possiblities aren't any more likely to be dealt with in OSS 
development than in commercial software development.

About the only place I've ever seen 100% exception handling expected is 
in ACM programming contests.

>>>>> B) find that it's open by someone else as a shared text segment,
>>>> I don't believe that would matter,
>>> You're mistaken.
>> 
>> Well, it's something I've never run into, and I've only been running
>> Linux for about 12 years...
> 
> http://www.derkeiler.com/Mailing-Lists/FreeBSD-
Security/2002-07/10992.html

FreeBSD != Linux, but you knew that already. ;-)

> It's not really common, unless you're trying to update executables while
> they're running. It's not hard to make it happen. We've been over that
> here, a few weeks ago.

And in such a case IME the updater errors and tells you there was a 
problem.

>> Well, the installer does just that - it unlinks and creates new files,
>> near as I can tell.
> 
> Most likely. Probably because (d'oh) you can't write to a file that's
> being executed, yes? ;-)

Well, yeah, that would fall under the category of "handling the 
exception", no? ;-)

>>>>> C) run out of disk space,
>>>> That would create other problems as well
>>> Yes? So?  We know it's bad and you should avoid it, yes.
>> 
>> My point is that you would notice this probably before running out of
>> disk space.
> 
> Unless the update is what runs you out of space, or (just maybe) there's
> more than one person using the computer, like? Or some background job
> that's maybe doing something like receiving email?

Sure, and as I said, I've had that happen to me - didn't need to 
reinstall the system, had to clean a few things up, but this sort of 
thing can happen with any OS, not just Linux.  Ever try writing to the 
Windows registry with C: full?  (I knew you were waiting for me to bring 
Windows into the discussion - there you go. ;-) )

>>> And if the package manager updates the database *before* it updates
>>> the files, you might never know it.
>> 
>> If you never know it, though, then things are working as expected
> 
> And if "as expected" means "still has the security holes in it that the
> update was supposed to patch", then yes, that's as expected. Not what
> you want, but as expected. :-)

So you need to run a validation - but again as I've said a few times, my 
experience has been that an *error* condition like you describe stops the 
upgrade process, it doesn't silently say "well, that's OK" and continue 
without notifying the user.

> But yes, if you're only talking about package management per se (which
> is indeed where we started) then there are obviously solutions in place.
> It's just a shame they didn't generalize it so you can make it work for
> other people too, or over the network, or in spite of failures, etc.

"or over the network" - yeah, that works, but you have to follow special 
procedures.  Anyone who sets a system up with remote /usr or /var 
partitions knows that - or should know that.

"in spite of failures" - huh?  Update a system by writing files that 
cannot be written because of a full disk (say) even though the disk is 
full?  I'm not quite sure how you'd achieve the feat of writing to a full 
disk without causing corruption.  But again, that's why the updater 
displays an error message rather than silently failing. :-)

Jim

Post a reply to this message

On Sun, 23 Nov 2008 18:07:45 -0200, Nicolas Alvarez wrote:

> Jim Henderson wrote:
>> I don't believe that would matter, again, it's the inode that's open,
>> and when the file is overwritten a new inode is created and the old one
>> is destroyed.
> 
> Uh, it's not. Depends on your definition of "overwritten".
> 
> Open an existing file (inode), write into it. Did the inode change?

That's not the situation we're talking about - we're talking about 
replacing something like a shared library - I've *never* seen shared 
libraries updated by opening them and writing a change into them, then 
saving them out again.

Jim

Post a reply to this message

Jim Henderson wrote:
> Well, sure, but dealing with every possible and conceivable exception 
> gets into areas where even in professional development it's decided not 
> to go. 

No, I  mean you can make it generic. Instead of looking at each of the 
30+ possible causes of error, you can say "Any error rolls back the 
changes."

You don't write a database transaction and say "What if the power fails? 
What if the process is killed? What if I run out of space? What if ...?"
Instead, you just build a transaction, and if anything fails, you roll 
back the transaction.

That's what I'm talking about. Then you don't have to build things like 
"check the package database against the file system to make sure it's 
consistent."

> FreeBSD != Linux, but you knew that already. ;-)

Except this error has been out there since UNIX v7 at least. But you 
knew that already.

> Well, yeah, that would fall under the category of "handling the 
> exception", no? ;-)

Not really. It's correcting an error you detected. If the error 
correction step also fails, you're pretty screwed.

For example, to make things work, you need to up date the executable and 
three dynamic libraries. You update the executable, and one of the 
libraries, and then your network connection to the machine hosting the 
files fails. You can't handle that exception by rolling back your 
changes manually.

> Sure, and as I said, I've had that happen to me - didn't need to 
> reinstall the system, had to clean a few things up, but this sort of 
> thing can happen with any OS, not just Linux.  Ever try writing to the 
> Windows registry with C: full?  (I knew you were waiting for me to bring 
> Windows into the discussion - there you go. ;-) )

Well, yes. That's why you have file system transactions in Windows. 
That's kinda exactly my point. If you start a kernel transaction, copy 
some files, update the registry, then bomb out, your changes get rolled 
back automatically. Just like any other database system, and regardless 
of why you bombed out or over which network the files are mounted.

> So you need to run a validation - but again as I've said a few times, my 
> experience has been that an *error* condition like you describe stops the 
> upgrade process, it doesn't silently say "well, that's OK" and continue 
> without notifying the user.

Unless the error is "you pulled the plug" or "the RAID fell over" or 
something like that.

> "in spite of failures" - huh?  Update a system by writing files that 
> cannot be written because of a full disk (say) even though the disk is 
> full?

No. Not having half-written files caused by the disk being full. Or 
having only (say) two out of file files updated because the disk got 
full while you were writing the third file.

-- 
Darren New / San Diego, CA, USA (PST)

Post a reply to this message

Darren New wrote:
> files fails. You can't handle that exception by rolling back your 
> changes manually.

By which I mean, you can't handle that exception by writing code to 
inspect errno and automatically undoing the changes you made. Of course 
you can handle it "manually" as in a human intervening.

-- 
Darren New / San Diego, CA, USA (PST)

Post a reply to this message

Jim Henderson wrote:
> I've *never* seen shared 
> libraries updated by opening them and writing a change into them, then 
> saving them out again.

So, you haven't used gcc much, then? :-)

-- 
Darren New / San Diego, CA, USA (PST)

Post a reply to this message

On Sun, 23 Nov 2008 23:15:01 -0800, Darren New wrote:

> Jim Henderson wrote:
>> I've *never* seen shared
>> libraries updated by opening them and writing a change into them, then
>> saving them out again.
> 
> So, you haven't used gcc much, then? :-)

I use it quite a bit.  A recompile of a library generates a new file, it 
doesn't modify an existing library like you would a text document = open, 
modify, save by writing to same inode.

Jim

Post a reply to this message

On Sun, 23 Nov 2008 22:57:32 -0800, Darren New wrote:

> Jim Henderson wrote:
>> Well, sure, but dealing with every possible and conceivable exception
>> gets into areas where even in professional development it's decided not
>> to go.
> 
> No, I  mean you can make it generic. Instead of looking at each of the
> 30+ possible causes of error, you can say "Any error rolls back the
> changes."

So like "an error occurs, let's ask the user what should happen"?  Which 
is what it does. :-)

>> Well, yeah, that would fall under the category of "handling the
>> exception", no? ;-)
> 
> Not really. It's correcting an error you detected. If the error
> correction step also fails, you're pretty screwed.

That would likely happen with a disk full error - if you can't commit one 
transaction, it's going to be hard to verify and write out the old state 
as well, no?

> For example, to make things work, you need to up date the executable and
> three dynamic libraries. You update the executable, and one of the
> libraries, and then your network connection to the machine hosting the
> files fails. You can't handle that exception by rolling back your
> changes manually.

You can by reinstalling the package, though.

>> Sure, and as I said, I've had that happen to me - didn't need to
>> reinstall the system, had to clean a few things up, but this sort of
>> thing can happen with any OS, not just Linux.  Ever try writing to the
>> Windows registry with C: full?  (I knew you were waiting for me to
>> bring Windows into the discussion - there you go. ;-) )
> 
> Well, yes. That's why you have file system transactions in Windows.
> That's kinda exactly my point. If you start a kernel transaction, copy
> some files, update the registry, then bomb out, your changes get rolled
> back automatically. Just like any other database system, and regardless
> of why you bombed out or over which network the files are mounted.

And on Linux you have ext3, reiser, jfs, and other journaling filesystems.

>> So you need to run a validation - but again as I've said a few times,
>> my experience has been that an *error* condition like you describe
>> stops the upgrade process, it doesn't silently say "well, that's OK"
>> and continue without notifying the user.
> 
> Unless the error is "you pulled the plug" or "the RAID fell over" or
> something like that.

If RAID falls over, I'd say you've got a more serious problem on your 
hands....I've had that happen a few times, and it generally leads to a 
system reinstall.

>> "in spite of failures" - huh?  Update a system by writing files that
>> cannot be written because of a full disk (say) even though the disk is
>> full?
> 
> No. Not having half-written files caused by the disk being full. Or
> having only (say) two out of file files updated because the disk got
> full while you were writing the third file.

Sure, and that's what package validation is for - but again it becomes 
moot because when the SUSE updater tries to update a file and can't 
because the disk is full, you get a big really obvious dialogue box that 
says "hey, we're out of disk space, what do you want to do?" - so you can 
clear up some disk space and say "try again".

Jim

Post a reply to this message

Jim Henderson wrote:
>> No, I  mean you can make it generic. Instead of looking at each of the
>> 30+ possible causes of error, you can say "Any error rolls back the
>> changes."
> 
> So like "an error occurs, let's ask the user what should happen"?  Which 
> is what it does. :-)

Uh, no. That's kind of my point. You can set things up such that it 
works *JUST LIKE* a database transaction. Something fails in the middle, 
and the system automatically puts things back to the way they were.

> That would likely happen with a disk full error - if you can't commit one 
> transaction, it's going to be hard to verify and write out the old state 
> as well, no?

No. Just like a real database system, things don't get updated 
permanently until you're sure it'll work.

>> For example, to make things work, you need to up date the executable and
>> three dynamic libraries. You update the executable, and one of the
>> libraries, and then your network connection to the machine hosting the
>> files fails. You can't handle that exception by rolling back your
>> changes manually.
> 
> You can by reinstalling the package, though.

Only after you get your network connection back. Until then, everyone 
using the machine locally is screwed.

>> Well, yes. That's why you have file system transactions in Windows.
>> That's kinda exactly my point. If you start a kernel transaction, copy
>> some files, update the registry, then bomb out, your changes get rolled
>> back automatically. Just like any other database system, and regardless
>> of why you bombed out or over which network the files are mounted.
> 
> And on Linux you have ext3, reiser, jfs, and other journaling filesystems.


Cool. What's the system call in Linux that lets me change three files 
consistemtly?  I.e., I have files /tmp/One, /tmp/Two, and /tmp/Three, 
and I want to rename them respectively to /tmp/1, /tmp/2, and /tmp/3, 
and I never want any possibility of an "ls" operation on the /tmp/ 
directory to show my /tmp/One and /tmp/3 at the same time, or /tmp/1 and 
/tmp/Three.  Is there some way to accomplish that?

>> Unless the error is "you pulled the plug" or "the RAID fell over" or
>> something like that.
> 
> If RAID falls over, I'd say you've got a more serious problem on your 
> hands....I've had that happen a few times, and it generally leads to a 
> system reinstall.

And you've never lost power either, I take it?

> Sure, and that's what package validation is for

Except that's a manual post-hoc fix that only works for package 
installs. I'm talking about something that works without intervention on 
a more generic kind of problem.

-- 
Darren New / San Diego, CA, USA (PST)

Post a reply to this message

Jim Henderson wrote:
> I use it quite a bit.  A recompile of a library generates a new file, it 
> doesn't modify an existing library 

Did you try it?  It certainly reuses the same file for me.

-- 
Darren New / San Diego, CA, USA (PST)

Post a reply to this message

On Tue, 25 Nov 2008 12:33:56 -0800, Darren New wrote:

> Jim Henderson wrote:
>> I use it quite a bit.  A recompile of a library generates a new file,
>> it doesn't modify an existing library
> 
> Did you try it?  It certainly reuses the same file for me.

Logically it doesn't make sense - since offsets change in a library and 
so on when compiling code.....

And in my test, no, it didn't use the same inode for different 
compilations of the same C file.

Jim

Post a reply to this message

On Tue, 25 Nov 2008 12:33:26 -0800, Darren New wrote:

> Jim Henderson wrote:
>>> No, I  mean you can make it generic. Instead of looking at each of the
>>> 30+ possible causes of error, you can say "Any error rolls back the
>>> changes."
>> 
>> So like "an error occurs, let's ask the user what should happen"? 
>> Which is what it does. :-)
> 
> Uh, no. That's kind of my point. You can set things up such that it
> works *JUST LIKE* a database transaction. Something fails in the middle,
> and the system automatically puts things back to the way they were.

I prefer that the system let me know about an error condition so I can 
correct the underlying issue, when it comes to things like filesystem 
errors.

>> That would likely happen with a disk full error - if you can't commit
>> one transaction, it's going to be hard to verify and write out the old
>> state as well, no?
> 
> No. Just like a real database system, things don't get updated
> permanently until you're sure it'll work.

A silent failure wouldn't be useful - no indication of success or failure 
would be even more confusing, IMO.

>>> For example, to make things work, you need to up date the executable
>>> and three dynamic libraries. You update the executable, and one of the
>>> libraries, and then your network connection to the machine hosting the
>>> files fails. You can't handle that exception by rolling back your
>>> changes manually.
>> 
>> You can by reinstalling the package, though.
> 
> Only after you get your network connection back. Until then, everyone
> using the machine locally is screwed.

A good case to be made for doing updates on the local machine rather than 
the remote machine.  Like I said, management procedures in place for non-
standard deployments.

>>> Well, yes. That's why you have file system transactions in Windows.
>>> That's kinda exactly my point. If you start a kernel transaction, copy
>>> some files, update the registry, then bomb out, your changes get
>>> rolled back automatically. Just like any other database system, and
>>> regardless of why you bombed out or over which network the files are
>>> mounted.
>> 
>> And on Linux you have ext3, reiser, jfs, and other journaling
>> filesystems.
> 
> 
> Cool. What's the system call in Linux that lets me change three files
> consistemtly?  I.e., I have files /tmp/One, /tmp/Two, and /tmp/Three,
> and I want to rename them respectively to /tmp/1, /tmp/2, and /tmp/3,
> and I never want any possibility of an "ls" operation on the /tmp/
> directory to show my /tmp/One and /tmp/3 at the same time, or /tmp/1 and
> /tmp/Three.  Is there some way to accomplish that?

Again, if it's handled as part of the system update, the updater takes 
care of that for you, not the system.

>>> Unless the error is "you pulled the plug" or "the RAID fell over" or
>>> something like that.
>> 
>> If RAID falls over, I'd say you've got a more serious problem on your
>> hands....I've had that happen a few times, and it generally leads to a
>> system reinstall.
> 
> And you've never lost power either, I take it?

Got a system here with a bad power supply, as a matter of fact - and yes, 
it screwed a VMDK file that was open at the time.

You want to implement a filesystem as a database, knock yourself 
out. ;-)  It's overkill for most applications.

>> Sure, and that's what package validation is for
> 
> Except that's a manual post-hoc fix that only works for package
> installs. I'm talking about something that works without intervention on
> a more generic kind of problem.

....

Jim

Post a reply to this message

Jim Henderson wrote:
> I prefer that the system let me know about an error condition so I can 
> correct the underlying issue, when it comes to things like filesystem 
> errors.

Of course you find out about the error. I'd just prefer the system to not 
corrupt the file system when something fails.

> A silent failure wouldn't be useful - no indication of success or failure 
> would be even more confusing, IMO.

What makes you think it would be silent??

I take it you've never used a database that has transactions. What happens 
is you get back a message saying "your transaction failed because the disk 
was full" (or whatever the reason), and you don't have to do anything to put 
your changes back to how they were before you started. So if you say (for 
example) "delete all the .doc files from this directory", and one of them 
isn't deletable, none of them get deleted, and you're told which one can't 
be deleted.

> A good case to be made for doing updates on the local machine rather than 
> the remote machine.  Like I said, management procedures in place for non-
> standard deployments.

Right. That's what I'm saying.  Management procedures to work around the 
lack of functionality.

>> Cool. What's the system call in Linux that lets me change three files
>> consistemtly?  I.e., I have files /tmp/One, /tmp/Two, and /tmp/Three,
>> and I want to rename them respectively to /tmp/1, /tmp/2, and /tmp/3,
>> and I never want any possibility of an "ls" operation on the /tmp/
>> directory to show my /tmp/One and /tmp/3 at the same time, or /tmp/1 and
>> /tmp/Three.  Is there some way to accomplish that?
> 
> Again, if it's handled as part of the system update, the updater takes 
> care of that for you, not the system.

That didn't answer the question.  How does the updater make sure that you 
never see a half-renamed collection of files, even assuming we're only 
talking about the updater here?

> You want to implement a filesystem as a database, knock yourself 
> out. ;-)  It's overkill for most applications.

Not really. ext3 and reiser and all that are already implemented with 
transactions. You just can't get to them.

Post a reply to this message

Jim Henderson wrote:
> Logically it doesn't make sense - since offsets change in a library and 
> so on when compiling code.....

I haven't the slightest idea why that would have anything to do with the 
file-system level of whether you delete the file before you recreate it or not?

There's good reasons *not* to do that too, like not messing up permissions 
and symlinks.

> And in my test, no, it didn't use the same inode for different 
> compilations of the same C file.

So you ran
   gcc xyz.c
   ls -i a.out
   gcc xyz.c
   ls -i a.out
and you got two different inodes for a.out?

Post a reply to this message

On Tue, 25 Nov 2008 17:30:32 -0800, Darren New wrote:

> Jim Henderson wrote:
>> Logically it doesn't make sense - since offsets change in a library and
>> so on when compiling code.....
> 
> I haven't the slightest idea why that would have anything to do with the
> file-system level of whether you delete the file before you recreate it
> or not?
> 
> There's good reasons *not* to do that too, like not messing up
> permissions and symlinks.
> 
>> And in my test, no, it didn't use the same inode for different
>> compilations of the same C file.
> 
> So you ran
>    gcc xyz.c
>    ls -i a.out
>    gcc xyz.c
>    ls -i a.out
> and you got two different inodes for a.out?

Modified the code inbetween.  If the compiled code doesn't change, then 
the point's moot - you'd be building the same library and there'd be no 
difference, so who cares if it gets overwritten or not?

Jim

Post a reply to this message

Jim Henderson wrote:
> Modified the code inbetween.  If the compiled code doesn't change, then 
> the point's moot - you'd be building the same library and there'd be no 
> difference, so who cares if it gets overwritten or not?

I'm not sure that answered the question, tho.  I get the same inode for both 
a.outs.  Anyway, you care if the code gets overwritten because if it's 
already running (and it has a shared text segment) you can't overwrite it, 
and if it's not already running, you can't start it when it's being written.

Post a reply to this message

On Tue, 25 Nov 2008 22:16:19 -0800, Darren New wrote:

> Jim Henderson wrote:
>> Modified the code inbetween.  If the compiled code doesn't change, then
>> the point's moot - you'd be building the same library and there'd be no
>> difference, so who cares if it gets overwritten or not?
> 
> I'm not sure that answered the question, tho.  I get the same inode for
> both a.outs.  Anyway, you care if the code gets overwritten because if
> it's already running (and it has a shared text segment) you can't
> overwrite it, and if it's not already running, you can't start it when
> it's being written.

Your point is that updating a file while it's open and has a shared text 
segment would cause a problem.  My point is that if you're *updating* the 
file, the inode changes *unless* the file is being overwritten by an 
unmodified version of that file (and that's probably not likely to happen 
often), so if the file doesn't get updated with an identical copy of 
itself, big deal.  The file is as it should be regardless of whether it 
gets overwritten with an identical copy if its previous version or not.

The SUSE updater, though, applies patches, so only files that have 
changed would get updated.......so again, not seeing what the problem is 
here.

Jim

Post a reply to this message

On Tue, 25 Nov 2008 17:28:49 -0800, Darren New wrote:

> Jim Henderson wrote:
>> I prefer that the system let me know about an error condition so I can
>> correct the underlying issue, when it comes to things like filesystem
>> errors.
> 
> Of course you find out about the error. I'd just prefer the system to
> not corrupt the file system when something fails.

Corrupting a single file isn't exactly the same thing as corrupting the 
filesystem.....cf reference to our previously crashed RAID arrays.

>> A silent failure wouldn't be useful - no indication of success or
>> failure would be even more confusing, IMO.
> 
> What makes you think it would be silent??

That seemed to be what you were promoting.  My mistake if I misread.

> I take it you've never used a database that has transactions. What
> happens is you get back a message saying "your transaction failed
> because the disk was full" (or whatever the reason), and you don't have
> to do anything to put your changes back to how they were before you
> started. So if you say (for example) "delete all the .doc files from
> this directory", and one of them isn't deletable, none of them get
> deleted, and you're told which one can't be deleted.

I've dealt with transactional systems quite a bit, now I see what you 
were saying - treat the "package" update as a single transaction.  
Somehow I missed that before.  I guess it's been a busy week trying to 
wrap things up before the holiday. :-)

>> A good case to be made for doing updates on the local machine rather
>> than the remote machine.  Like I said, management procedures in place
>> for non- standard deployments.
> 
> Right. That's what I'm saying.  Management procedures to work around the
> lack of functionality.

But there again we're talking about configurations that aren't terribly 
common these days - they used to be, sure - I can remember Sun systems 
with diskless workstations because the versions with disks were so much 
more expensive - so the system was completely remotely booted from a 
4/290.

>>> Cool. What's the system call in Linux that lets me change three files
>>> consistemtly?  I.e., I have files /tmp/One, /tmp/Two, and /tmp/Three,
>>> and I want to rename them respectively to /tmp/1, /tmp/2, and /tmp/3,
>>> and I never want any possibility of an "ls" operation on the /tmp/
>>> directory to show my /tmp/One and /tmp/3 at the same time, or /tmp/1
>>> and /tmp/Three.  Is there some way to accomplish that?
>> 
>> Again, if it's handled as part of the system update, the updater takes
>> care of that for you, not the system.
> 
> That didn't answer the question.  How does the updater make sure that
> you never see a half-renamed collection of files, even assuming we're
> only talking about the updater here?

Because it deals with the individual package updates one RPM or delta 
file at a time.

>> You want to implement a filesystem as a database, knock yourself out.
>> ;-)  It's overkill for most applications.
> 
> Not really. ext3 and reiser and all that are already implemented with
> transactions. You just can't get to them.

Where there's a will, there's a way - it is all open source code, so it 
wouldn't be impossible to implement for someone who was really determined 
to do so.  :-)

Jim

Post a reply to this message

Jim Henderson wrote:
> Your point is that updating a file while it's open and has a shared text 
> segment would cause a problem. 

No, my point is that there are numerous ways that things can fail, and if 
your OS doesn't supply the right tools, you wind up writing ad-hoc manual 
work-arounds to recover when the program tells you it failed.

> My point is that if you're *updating* the 
> file, the inode changes *unless* the file is being overwritten by an 
> unmodified version of that file 

Uh, not according to my runs of gcc. Maybe the SuSE updater works that way, 
but that just reinforces my point: they have to have an ad-hoc 
application-specific work-around for the problem.

> The SUSE updater, though, applies patches, so only files that have 
> changed would get updated.......so again, not seeing what the problem is 
> here.

That you're discussing something different from what I'm trying to talk about?

You're failing to understand that I'm pointing out something entirely 
different than what you're arguing.  I'm pointing out that UNIX has 
traditionally addressed limitations in UNIX with ad-hoc manual and/or 
application-specific work-arounds, while most other systems incorporate into 
the operating system those operations that most applications need. When I 
say "it doesn't need to be this bad," you keep answering with "in this one 
case, it's possible to work around the problems most of the time if you're 
careful."  That isn't the point, tho.

--
   Darren New, San Diego CA, USA (PST)
   The NFL should go international. I'd pay to
   see the Detroit Lions vs the Roman Catholics.

Post a reply to this message

Jim Henderson wrote:
> Corrupting a single file isn't exactly the same thing as corrupting the 
> filesystem.....cf reference to our previously crashed RAID arrays.

No. But since the file system already supports not corrupting the file 
system, why wouldn't it support not corrupting the file?

> I've dealt with transactional systems quite a bit, now I see what you 
> were saying - treat the "package" update as a single transaction.  

Yes, exactly. Which is hard to do on a file system that doesn't support 
transactions, because UNIX traditionally layered transactions and locking on 
top of the file system in an application-specific way, just like it layers 
"login" on top of the permission system in an application-specific way.

Hence, you have people creating unwritable lock files to lock things, and 
the file system can't support things like temp files that only survive until 
you log out, and things like that.

> But there again we're talking about configurations that aren't terribly 
> common these days 

Sure.  But again, you're looking at the one example and failing to generalize.

>>>> Cool. What's the system call in Linux that lets me change three files
>>>> consistemtly?  I.e., I have files /tmp/One, /tmp/Two, and /tmp/Three,
>>>> and I want to rename them respectively to /tmp/1, /tmp/2, and /tmp/3,
>>>> and I never want any possibility of an "ls" operation on the /tmp/
>>>> directory to show my /tmp/One and /tmp/3 at the same time, or /tmp/1
>>>> and /tmp/Three.  Is there some way to accomplish that?
>>> Again, if it's handled as part of the system update, the updater takes
>>> care of that for you, not the system.
>> That didn't answer the question.  How does the updater make sure that
>> you never see a half-renamed collection of files, even assuming we're
>> only talking about the updater here?
> 
> Because it deals with the individual package updates one RPM or delta 
> file at a time.

You still didn't answer the question. Look at the actual question:
     What's the system call in Linux that lets me change three files
     consistently?
You said the Linux journaling file systems let you have file system 
transactions. I believe it might be there, but I've never found it; it might 
be something new. What's the system call?

> Where there's a will, there's a way - it is all open source code, so it 
> wouldn't be impossible to implement for someone who was really determined 
> to do so.  :-)

Sure. But you implied it's already there.

I just pine for the days when it was possible to write a program that ran 
reliably even in edge conditions, rather than being impossible to ensure 
that things work correctly because the underlying systems are so half-ass. :-)

--
   Darren New, San Diego CA, USA (PST)
   The NFL should go international. I'd pay to
   see the Detroit Lions vs the Roman Catholics.

Post a reply to this message

On Wed, 26 Nov 2008 10:22:44 -0800, Darren New wrote:

> Jim Henderson wrote:
>> Your point is that updating a file while it's open and has a shared
>> text segment would cause a problem.
> 
> No, my point is that there are numerous ways that things can fail, and
> if your OS doesn't supply the right tools, you wind up writing ad-hoc
> manual work-arounds to recover when the program tells you it failed.
> 
>> My point is that if you're *updating* the file, the inode changes
>> *unless* the file is being overwritten by an unmodified version of that
>> file
> 
> Uh, not according to my runs of gcc. Maybe the SuSE updater works that
> way, but that just reinforces my point: they have to have an ad-hoc
> application-specific work-around for the problem.

You didn't modify the code between compilation runs.  I did - and got 
different inodes.

>> The SUSE updater, though, applies patches, so only files that have
>> changed would get updated.......so again, not seeing what the problem
>> is here.
> 
> That you're discussing something different from what I'm trying to talk
> about?

I'm keeping to the initial point of discussion, which was a perceived 
problem with the SUSE updater application.

Jim

Post a reply to this message

On Wed, 26 Nov 2008 10:29:51 -0800, Darren New wrote:

> Jim Henderson wrote:
>> Corrupting a single file isn't exactly the same thing as corrupting the
>> filesystem.....cf reference to our previously crashed RAID arrays.
> 
> No. But since the file system already supports not corrupting the file
> system, why wouldn't it support not corrupting the file?

<shrug>  Maybe it does.  I'm not a kernel developer.

>> I've dealt with transactional systems quite a bit, now I see what you
>> were saying - treat the "package" update as a single transaction.
> 
> Yes, exactly. Which is hard to do on a file system that doesn't support
> transactions, because UNIX traditionally layered transactions and
> locking on top of the file system in an application-specific way, just
> like it layers "login" on top of the permission system in an
> application-specific way.

Just at a guess, abstraction layers tend to impact performance.  Look at 
the performance of a fuse-based NFS server vs. a kernel-based NFS server.

>>>>> Cool. What's the system call in Linux that lets me change three
>>>>> files consistemtly?  I.e., I have files /tmp/One, /tmp/Two, and
>>>>> /tmp/Three, and I want to rename them respectively to /tmp/1,
>>>>> /tmp/2, and /tmp/3, and I never want any possibility of an "ls"
>>>>> operation on the /tmp/ directory to show my /tmp/One and /tmp/3 at
>>>>> the same time, or /tmp/1 and /tmp/Three.  Is there some way to
>>>>> accomplish that?
>>>> Again, if it's handled as part of the system update, the updater
>>>> takes care of that for you, not the system.
>>> That didn't answer the question.  How does the updater make sure that
>>> you never see a half-renamed collection of files, even assuming we're
>>> only talking about the updater here?
>> 
>> Because it deals with the individual package updates one RPM or delta
>> file at a time.
> 
> You still didn't answer the question. Look at the actual question:
>      What's the system call in Linux that lets me change three files
>      consistently?
> You said the Linux journaling file systems let you have file system
> transactions. I believe it might be there, but I've never found it; it
> might be something new. What's the system call?

I'm not a kernel developer, so I don't know what the system call is.  
That would be a question for the kernel developers. :-)

>> Where there's a will, there's a way - it is all open source code, so it
>> wouldn't be impossible to implement for someone who was really
>> determined to do so.  :-)
> 
> Sure. But you implied it's already there.

It seems to be IME.  Maybe that's because it's an application-specific 
solution implemented - but I point back to the start of the discussion 
where you cited problems with the SUSE updater application.  I was 
answering that question (at least initially), not getting into a 
discussion of kernel-level system calls for transactional dealings with 
multiple files, because I'm not a kernel guru.

> I just pine for the days when it was possible to write a program that
> ran reliably even in edge conditions, rather than being impossible to
> ensure that things work correctly because the underlying systems are so
> half-ass. :-)

Well, most development is done to the 80/20 rule rather than to "edge 
conditions" for economic reasons (at least that's what I'm told).  OSS is 
a wonderful thing, though, because those who want to develop to those 
"edge conditions" can do so.  :-)

Jim

Post a reply to this message

Jim Henderson wrote:
> You didn't modify the code between compilation runs.  I did - and got 
> different inodes.

Of course I did.

> I'm keeping to the initial point of discussion, which was a perceived 
> problem with the SUSE updater application.

OK.

-- 
   Darren New, San Diego CA, USA (PST)
   The NFL should go international. I'd pay to
   see the Detroit Lions vs the Roman Catholics.

Post a reply to this message

Jim Henderson wrote:
> I'm keeping to the initial point of discussion, which was a perceived 
> problem with the SUSE updater application.

Actually, it was along the lines of "I wish the Suse updater let me have 
multiple programs accessing the database it keeps." :-) Since BDB apparently 
allows for multiple readers and writers, I'm not sure there's any 
infrastructure reason the updater wouldn't.

-- 
   Darren New, San Diego CA, USA (PST)
   The NFL should go international. I'd pay to
   see the Detroit Lions vs the Roman Catholics.

Post a reply to this message

On Wed, 26 Nov 2008 17:46:59 -0800, Darren New wrote:

> Jim Henderson wrote:
>> I'm keeping to the initial point of discussion, which was a perceived
>> problem with the SUSE updater application.
> 
> Actually, it was along the lines of "I wish the Suse updater let me have
> multiple programs accessing the database it keeps." :-) Since BDB
> apparently allows for multiple readers and writers, I'm not sure there's
> any infrastructure reason the updater wouldn't.

On that point we are agreed. :-)

Jim

Post a reply to this message

On Wed, 26 Nov 2008 14:45:50 -0800, Darren New wrote:

> Jim Henderson wrote:
>> You didn't modify the code between compilation runs.  I did - and got
>> different inodes.
> 
> Of course I did.

That's not what I read in your post.  I duplicated with the code staying 
the same, but not with it changing.

> 
>> I'm keeping to the initial point of discussion, which was a perceived
>> problem with the SUSE updater application.
> 
> OK.

Jim

Post a reply to this message

Jim Henderson wrote:
> That's not what I read in your post.  I duplicated with the code staying 
> the same, but not with it changing.

I take advantage of X windows, ya see. :-)

I find it really hard to believe GCC will compile a program, then read the 
output file to see if the result is the same, and if not, only then will 
unlink the old code and write the new code.

I'll give it a try next time I'm in Linux, tho.  I suspect it's more some 
options on the compiler or file system we have set differently or something.

-- 
   Darren New, San Diego CA, USA (PST)
   The NFL should go international. I'd pay to
   see the Detroit Lions vs the Roman Catholics.

Post a reply to this message

Darren New wrote:
> 
> I find it really hard to believe GCC will compile a program, then read
> the output file to see if the result is the same, and if not, only then
> will unlink the old code and write the new code.
> 

OTOH it could check mtime of the source and the binary and choose the
way to work. But somehow I even that hard to believe.

-Aero

Post a reply to this message

Eero Ahonen wrote:
> OTOH it could check mtime of the source and the binary and choose the
> way to work. But somehow I even that hard to believe.

Except it would have to pretty much compile everything anyway, since 
environment variables, command line options, and include files can all 
affect the results too. And anyway, I changed the file between runs.

-- 
   Darren New, San Diego CA, USA (PST)
   The NFL should go international. I'd pay to
   see the Detroit Lions vs the Roman Catholics.

Post a reply to this message

On Mon, 01 Dec 2008 10:55:03 -0800, Darren New wrote:

> Jim Henderson wrote:
>> That's not what I read in your post.  I duplicated with the code
>> staying the same, but not with it changing.
> 
> I take advantage of X windows, ya see. :-)
> 
> I find it really hard to believe GCC will compile a program, then read
> the output file to see if the result is the same, and if not, only then
> will unlink the old code and write the new code.

It probably looks to see if the source file changed since the last 
compilation (not sure how it tracks that).  All I know is that when I 
didn't change the code, the inode stayed the same, and when I did, it 
didn't.

> I'll give it a try next time I'm in Linux, tho.  I suspect it's more
> some options on the compiler or file system we have set differently or
> something.

Could be.

Jim

Post a reply to this message

Jim Henderson wrote:
> It probably looks to see if the source file changed since the last 
> compilation (not sure how it tracks that).

I find it hard to believe that GCC would be that broken.

-- 
   Darren New, San Diego CA, USA (PST)
   The NFL should go international. I'd pay to
   see the Detroit Lions vs the Roman Catholics.

Post a reply to this message

On Mon, 01 Dec 2008 18:57:12 -0800, Darren New wrote:

> Jim Henderson wrote:
>> It probably looks to see if the source file changed since the last
>> compilation (not sure how it tracks that).
> 
> I find it hard to believe that GCC would be that broken.

<shrug>  I'm just goin' by my observation.....if you've got a better 
explanation, I'm all ears.

Or maybe that's "eyes". ;-)

Jim

Post a reply to this message

Jim Henderson wrote:
> <shrug>  I'm just goin' by my observation.....if you've got a better 
> explanation, I'm all ears.

Dunno. Like I said, it may be the compiler version or something. Now, if 
you're talking Java, I could understand this, because the Java equivalent of 
#include looks at .class files to see if something needs to be recompiled. 
I.e., "make" is sort of built in to Java compilers.

darren@dog:~/test> gcc -v
Using built-in specs.
Target: x86_64-suse-linux
Configured with: ../configure --prefix=/usr --with-local-prefix=/usr/local 
--infodir=/usr/share/info --mandir=/usr/share/man --libdir=/usr/lib64 
--libexecdir=/usr/lib64 
--enable-languages=c,c++,objc,fortran,obj-c++,java,ada 
--enable-checking=release --with-gxx-include-dir=/usr/include/c++/4.3 
--enable-ssp --disable-libssp --with-bugurl=http://bugs.opensuse.org/ 
--with-pkgversion='SUSE Linux' --disable-libgcj --with-slibdir=/lib64 
--with-system-zlib --enable-__cxa_atexit --enable-libstdcxx-allocator=new 
--disable-libstdcxx-pch --program-suffix=-4.3 
--enable-version-specific-runtime-libs --enable-linux-futex 
--without-system-libunwind --with-cpu=generic --build=x86_64-suse-linux
Thread model: posix
gcc version 4.3.1 20080507 (prerelease) [gcc-4_3-branch revision 135036] 
(SUSE Linux)
darren@dog:~/test> ls -l
total 28
-rw-r--r-- 1 darren users 67 2008-12-02 10:25 a.c
-rw-r--r-- 1 darren users 67 2008-12-02 10:26 b.c
-rw-r--r-- 1 darren users 22 2008-12-02 11:07 c.c
-rw-r--r-- 1 darren users 22 2008-12-02 10:25 d.c
darren@dog:~/test> cat a.c
#include "c.c"
int main() { write(1, "Hello " XYZ, 12); return 0;}
darren@dog:~/test> cat b.c
#include "c.c"
int main() { write(1, "Zing! " XYZ, 12); return 0;}
darren@dog:~/test> cat c.c
#define XYZ "World\n"
darren@dog:~/test> cat d.c
#define XYZ "xyzzy\n"
darren@dog:~/test> gcc a.c ; ls -li a.out ; ./a.out
1818787 -rwxr-xr-x 1 darren users 11120 2008-12-02 11:07 a.out
Hello World
darren@dog:~/test> gcc b.c ; ls -li a.out ; ./a.out
1818787 -rwxr-xr-x 1 darren users 11120 2008-12-02 11:07 a.out
Zing! World
darren@dog:~/test> cat d.c >c.c
darren@dog:~/test> gcc a.c ; ls -li a.out ; ./a.out
1818787 -rwxr-xr-x 1 darren users 11120 2008-12-02 11:08 a.out
Hello xyzzy
darren@dog:~/test> gcc b.c ; ls -li a.out ; ./a.out
1818787 -rwxr-xr-x 1 darren users 11120 2008-12-02 11:08 a.out
Zing! xyzzy
darren@dog:~/test> cat a.c >b.c
darren@dog:~/test> gcc b.c ; ls -li a.out ; ./a.out
1818787 -rwxr-xr-x 1 darren users 11120 2008-12-02 11:08 a.out
Hello xyzzy
darren@dog:~/test>

Nothing I did changed the inode number of a.out, in spite of getting 
different output every time I ran it.

-- 
   Darren New, San Diego CA, USA (PST)
   The NFL should go international. I'd pay to
   see the Detroit Lions vs the Roman Catholics.

Post a reply to this message

On Tue, 02 Dec 2008 11:30:48 -0800, Darren New wrote:

> Jim Henderson wrote:
>> <shrug>  I'm just goin' by my observation.....if you've got a better
>> explanation, I'm all ears.
> 
> Dunno. Like I said, it may be the compiler version or something. Now, if
> you're talking Java, I could understand this, because the Java
> equivalent of #include looks at .class files to see if something needs
> to be recompiled. I.e., "make" is sort of built in to Java compilers.

Might be.  What filesystem are you using?  Maybe it's a question of the 
inode allocation system, number of free inodes and the like?

Jim

Post a reply to this message

Jim Henderson wrote:
> Might be.  What filesystem are you using?  Maybe it's a question of the 
> inode allocation system, number of free inodes and the like?

ext3. I don't think that's it. If I delete the a.out and run gcc again, I 
get a different inode number.

-- 
   Darren New, San Diego CA, USA (PST)
   The NFL should go international. I'd pay to
   see the Detroit Lions vs the Roman Catholics.

Post a reply to this message

On Tue, 02 Dec 2008 12:17:52 -0800, Darren New wrote:

> Jim Henderson wrote:
>> Might be.  What filesystem are you using?  Maybe it's a question of the
>> inode allocation system, number of free inodes and the like?
> 
> ext3. I don't think that's it. If I delete the a.out and run gcc again,
> I get a different inode number.

Interesting; I also am running ext3 here....

Jim

Post a reply to this message