Now this is weird & bad...

I started a set of renders using the following shell scripts:

---------------------------------
# render_many.sh:
nohup ./render_many_sub.sh >KnightMoves_many.log 2>&1 &
.. ./status_many.sh
---------------------------------
# render_many_sub.sh:
megapov -D foo.ini +oA.hdr +J1.0 &
sleep 9000
megapov -D foo.ini +oB.hdr +J0.9 >KnightMoves_B.log 2>&1 &
sleep 120
megapov -D foo.ini +oC.hdr +J0.8 >KnightMoves_C.log 2>&1 &
sleep 120
nice megapov -D foo.ini +oD.hdr +J0.7 >KnightMoves_D.log 2>&1 &
---------------------------------

(status_many.sh basically just calls "tail -f KnightMoves_many.log")

Now I would assume that the "nohup" command basically "decouples" the render
processes from the terminal.

Render A completed successfully a few hours ago. A few minutes ago, the other
render processes terminated as well - but *not* successfully. Instead, in all
the three logfiles I find:

---------------------------------
Possible Rendering Error: Got 1 SIGINT.

 10:17:18 Aborting render!
 10:17:18 User abort
---------------------------------

(Great, just a couple of minutes before the 10-hour renders would have finished
anyway! Someone out there seems to hate me...)

How could this happen? Am I missing something here?

I think I may have stopped some "tail" that was called via the "status_many.sh"
script about that time (don't remember whether it was the one automatically
started though, or one I started later, and I don't know whether I actually
Ctrl-C'ed it before closing the terminal); I also recall having Ctrl-C'ed a
"top" command running on another terminal, and later accidently having
Ctrl-C'ed (wrong keyboard, whoops!) a shell running in some terminal, but I
think neither of these was the terminal I started the render processes from,
and I'm comparatively sure that both events were only after the processes had
already terminated. But that's about all events that to my knowledge might have
caused a SIGINT to be sent to *some* process.

Another event that happened about that time was that I connected a display to
the Unix computer's VGA output, but I can't possibly imagine that could have
caused a SIGINT.

Presuming that running render tasks "safely" in the background is a common thing
on Unix systems, I expect most people here have some experience with this issue.
How could this have happened, and most of all, how can I protect my background
processes against such accidents in future it if not via "nohup"?

I'm running Debian Linux (4.0r5 "etch") for AMD64 here, using the standard
"bash" shell and MegaPOV 1.2.1.

Post a reply to this message

> Now I would assume that the "nohup" command basically "decouples" the render
> processes from the terminal.

	'nohup' should only protect the command to which it applies from SIGHUP
signals, and nothing else.  Hence SIGINT and others will remain in effect.

	I don't know what may have happened exactly, but I'd suggest that
(i) you simplify your run script; e.g. I don't get why you sleep for many seconds
before you start the next megapov job in bg, assuming all jobs are independent.
(ii) use nohup for each megapov run, or, much better, redirect stderr for all of
them; redirecting stdout as well should be useless provided you use +oFILE.

	- NC

Post a reply to this message

clipka <nomail@nomail> wrote:
> Presuming that running render tasks "safely" in the background is a common thing
> on Unix systems, I expect most people here have some experience with this issue.
> How could this have happened, and most of all, how can I protect my background
> processes against such accidents in future it if not via "nohup"?

    man screen

  That's the proper way of running processes in the background so that they
will be kept running even if you logout. (Look for the key bindings,
especially those related to creating new screens, changing between them,
and detaching.)

-- 
                                                          - Warp

Post a reply to this message

Nicolas Calimet <pov### [at] freefr> wrote:
> (i) you simplify your run script; e.g. I don't get why you sleep for many seconds
> before you start the next megapov job in bg, assuming all jobs are independent.

Well, that's just the script I happened to be running; it was custom-tailored to
the specific "environmental conditions" and other constraints of that render:

- I had three other render processes running already, which I wanted to finish
before starting renders "B" through "D" to limit the number of "CPU-hungry"
processes to one per core.

- I had a deadline to meet, so there was not much time for developing a smarter
script, nor any room for risky experiments.

- When running a previous batch of renders for the same scene, I had started all
renders simultaneously, but checking after some minutes I had found the CPU load
highly unstable and at roughly 75% only, accompanied by heavy disk activity. I
attribute this to high physical RAM requirements (allocated memory was 1.1GB
per process), causing the render processes to waste time kicking each other to
swap space instead of peacefully crunching their numbers. After pausing one of
the processes, disk activity and CPU utilization slowly settled down to a
stable level, and re-starting the paused task later did not raise the issue
again, indicating that the high memory requirements were mainly limited to the
startup phase.

So all in all, I guess using those "sleep" statements was the smartest thing to
do.


> (ii) use nohup for each megapov run, or, much better, redirect stderr for all of
> them; redirecting stdout as well should be useless provided you use +oFILE.

I see the "just to be safe" benefit of wrapping each megapov run in its own
"nohup" statement, and redirecting stderr for the "A" render as well, both
which come at virtually no cost (although I'd have to introduce yet another
logfile, because otherwise I wouldn't be able to log errors spit out by the
script itself).

However, if I understand you correctly, this still doesn't shield my megapov
runs from SIGINTs, right?

Post a reply to this message

On Fri, 19 Dec 2008 12:45:24 -0500, clipka wrote:

> However, if I understand you correctly, this still doesn't shield my
> megapov runs from SIGINTs, right?

Within a script, I don't believe so, because the ctrl+c would interrupt 
the invoked bash shell (scripts don't run in the shell they're launched 
from IIRC, they're launched in a child shell).

Jim

Post a reply to this message

Jim Henderson <nos### [at] nospamcom> wrote:
> Within a script, I don't believe so, because the ctrl+c would interrupt
> the invoked bash shell (scripts don't run in the shell they're launched
> from IIRC, they're launched in a child shell).

Depends on whether you launch them via ". foo.sh" or just "foo.sh".

The former has the script run in your current shell (which by the way prevents
you from running it in the background), while the latter starts a new instance
of /bin/bash to run the thing.

I'm new to Linux administration, but I did quite some unser-side Unix stuff some
years a go... and I still have a reference book from back then :)
(Unfortunately it seems to be wrong, outdated or otherwise lacking on a few
details - like "nohup" not blocking SIGINT; according to the book it should,
along with SIGSTP... *sigh*)

Post a reply to this message

On Fri, 19 Dec 2008 13:06:58 -0500, clipka wrote:

> The former has the script run in your current shell (which by the way
> prevents you from running it in the background), while the latter starts
> a new instance of /bin/bash to run the thing.

Ah!  Learned something new today. :-)

Jim

Post a reply to this message

Hm - life can be a lot easier if one starts thinking along new paths:

Why try to "detach" a process from the controlling terminal, if you can have it
started without a controlling terminal in the first place...

This one does it:

-------------------------------
#!/bin/bash
at now << HERE
megapov "`pwd`/test.ini" > "`pwd`/test.log" 2>&1
HERE
sleep 5
tail -f test.log
-------------------------------

The "sleep 5" isn't the ideal solution yet to wait for the at-job to actually
start, but that can be improved.

In fact I guess I'll actually wind up using "batch" instead of "at now", because
that seems to be exactly the wheel I was trying to invent anyway. Now I only
have to figure where to specify the load average parameter I want for the "atd"
demon...

Post a reply to this message

clipka wrote:
> -------------------------------
> #!/bin/bash
> at now << HERE
> megapov "`pwd`/test.ini" > "`pwd`/test.log" 2>&1

Why the "pwd" instead of "./test.ini" and "./test.log"?


> HERE
> sleep 5
> tail -f test.log


Dan

Post a reply to this message

Dan Connelly <djc### [at] yahoocom> wrote:
> clipka wrote:
> > -------------------------------
> > #!/bin/bash
> > at now << HERE
> > megapov "`pwd`/test.ini" > "`pwd`/test.log" 2>&1
>
> Why the "pwd" instead of "./test.ini" and "./test.log"?

because I'd prefer to have the full path name in any output that might generate

Post a reply to this message