clipka wrote:
> I'd be eager to hear about it.

First, thanks for the tip on X Windows.  I verified that X Windows is not
available for compiling on the cluster.  This is probably a good thing since I
wouldn't want to accidentally leave display on when rendering a few thousand
animation frames.

As for setting up cluster jobs, the workshop was primarily an introduction to
writing SMP code.  I suppose the 'proper' way to distribute POV-Ray rendering
would be to patch it so that when it is run on a cluster one piece of the job (a
frame range or still image super-mosaic square) is rendered as usual (one mosaic
square thread per core) by each node.  Also, have another thread stitching the
pieces together as they become available.  (I'm taking the developers' word for
it that message overhead is prohibitively expensive for converting their pthread
implementation to MPI, even on clusters with massive internode bandwidth.)

But I put together a much simpler scheme that may be more appropriate for a
shared cluster using the installed job queue manager.  All I needed to write
were two scripts: one to create job submission tickets, and one to run POV-Ray
on the frame range (or image part) associated with each task number.

The job submission script creates two job tickets and submits them to the queue
manager.  The first job submits each frame range as a separate task, each of
which is pushed to a single node as one becomes available.  The second job waits
until all the tasks in the first job are finished, then stitches the frames
together in an animation.

The other script simply executes POV-Ray with the appropriate command-line
arguments to render the frame range associated with the environment variable
containing the current task number.

This strategy seems more appropriate for a shared cluster since it can start
submitting tasks as soon as a single node is available, and higher-priority jobs
can get in between separate tasks.  Note the render job ticket creates many more
tasks than there are nodes since it is likely that some frame ranges will take
much longer to render than others.  It is also very convenient to be able to
stop the entire process with just two commands, one for each node.

The only performance problem I'm having is with the delay of as much as a few
seconds between a node finishing a frame range and the queue manager pushing the
next task onto it.  I also anticipate decreased performance on images with
(single-thread) parse times approaching trace times.  I have had some success
overcoming these problems by running two tasks (two POV-Ray instances) on each
node at markedly different niceness to keep each node busy.  (Niceness is used
in an attempt decrease task-switching a bit and to help stagger render times.)
However, this works a bit too well as the processing environments I have
available watch for time-averaged node overloading, and during my tests, many
nodes stop accepting new jobs for a while when they have more than 12 active
threads on an 8-core node, then alarm when over 14.

I will probably need to contact the local cluster admins to set up a processing
environment better suited to distributed rendering with POV-Ray.  As far as I
can tell, I am the only user running multinode jobs, and very few changes beyond
the default configuration have been made to Shamu as yet.

Although I have only implemented this for animation, it should be fairly easy to
modify the scripts to break a single image into mosaic parts (rather than frame
ranges) and then stitch them together with ImageMagick (instead of ffmpeg).

Thanks for your interest.

David Wagner

P.S.  FWIW, I updated the instructions for beta 34.

Post a reply to this message