|
 |
Hi, I just wanted to share this.
Rendering a complex animation with Povray can easily take weeks if only a single
processor core is at hand. Even with Povray 3.7, which supports multiple cores,
the power of one multi-core system may not be enough to finish an animation in
acceptable time. The obivious solution would be to let several machines render
the frames in parallel. Povray doesn't have native support for this, but it can
be done by the help of scripts. In this article, I'm going to present such
scripts for parallel processing of Povray animations under Windows and Linux.
The scripts in the article are tailored for the following example situation:
We have an animation described by a scene file named landscape.pov and an .ini
file landscape.ini such that, if Povray was started with landscape.ini as input,
it would render all frames of the animation in one go. Also, we assume that the
order in which the frames are rendered does not matter, ie. there is no data
transfer from one frame to the next frame.
The animation shall be 458 seconds long at 30 frames per second. That is,
landscape.ini looks similar to this:
;
; landscape.ini
;
Input_File_Name=landscape.pov
Output_File_Type=N ; image format is png
Jitter=Off
Antialias=Off
Width=1920
Height=1080
Initial_Clock=0
Final_Clock=458
Initial_Frame=1
;
; Final_Frame = (Final_Clock - Initial_Clock) * FPS + Initial_Frame
; = 458 * 30 + 1 = 13741
;
Final_Frame=13741
Started with this .ini file, Povray would render the images landscape00001.png
till landscape13741.png. Now, to make rendering faster, we'd like to have
several instances of Povray rendering in parallel. This can be achieved by
applying the following scripts.
Windows
=======
The first script presented, builder.bat, runs on Windows. It should be placed in
the same folder as the .ini and .pov file and works like this:
1. The script searches the current folder for the image files landscape00001.png
till landscape13741.png.
2. When it detects that an image file does not exist, it starts an instance of
Povray to render that image file (only that file, none else).
3. When Povray finishes, the script looks for the next non-existing image file,
starts Povray to render it, and so forth.
To achieve parallel rendering, you simply start builder.bat several times. For
instance, when using Povray 3.6 on a machine with four CPU cores, you'd click
builder.bat four times. Or you could run several Povray instances on different
machines, working on the same folder shared through the network.
To prepare the script to your needs, you just have to define some variables at
its beginning:
----------------------------------------
@rem builder.bat
@rem
@rem Variables to be defined:
@rem povray - The path of the Povray executable.
@rem povname - The name of the scene to render, without extension.
@rem digits - The number of digits in the index of the image files.
@rem suffix - The extension of the image files, eg. png.
@rem s - Number of the initial frame.
@rem e - Number of the final frame.
@rem
@rem Author: Burkhard Reike, 2011-10-26
@echo off
setLocal EnableDelayedExpansion
set povray="C:\Users\Burk\AppData\Roaming\POV-Ray\v3.6\bin\pvengine-sse2.exe"
set povname=landscape
set digits=5
set suffix=png
set s=1
set e=13741
echo Starting: %computername% %date% %time% >> render.log
for /L %%c in (%s%,1,%e%) do (
set padcntr=0000000000%%c
set filename=!povname!!padcntr:~-%digits%!
if not exist !filename!.%suffix% (
type NUL > !filename!.%suffix%
call :GetTime
set starttime=!cseconds!
start /WAIT /min "Render" %povray% -d +SF%%c +EF%%c /exit +I %povname%.ini
call :GetTime
set endtime=!cseconds!
if !endtime! LSS !starttime! set /a endtime+=8640000
set /a elapsed=!endtime!-!starttime!
set /a elapsed/=100
echo !filename!.%suffix% !elapsed! %computername% >> render.log
)
)
echo Finished: %computername% %date% %time% >> render.log
goto:eof
:GetTime
for /f "usebackq tokens=1-4 delims=:., " %%f in (`echo.%time%`) do (
set temp=%%f %%g %%h %%i
)
for /f "usebackq tokens=1-4" %%f in (`echo %temp: 0= %`) do (
set /a cseconds=%%f*360000+%%g*6000+%%h*100+%%i
)
----------------------------------------
Along with the image files, the script generates the file render.log with
information about the rendering process, eg. when the scripts were started,
finished, and how many seconds each image file took to render.
For each script instance started, a DOS shell window will open, and it starts
Povray instances one after another. Each time a Povray instance starts, you will
see a splash screen. This cannot be avoided. Still, the Povray window does not
pop up owing to the "-d" option.
If you want to stop rendering for a longer time, you just have to kill the dos
shell windows. Each currently running Povray instance will stay alive till it
has finished rendering the current frame.
If, for some reason, you have killed Povray instances, the images they had been
rendering will remain unfinished. Starting new instances of builder.bat will not
rerender them unless the files have been removed. To identify such image files,
the following script, broken_files.bat, comes in handy.
It prints the names of all image files which exist in the current folder but
which are not mentioned in the log file. Typically, those are broken image files
whose Povray instances were killed prematurely. broken_files.bat should be
placed in the same folder as the .ini, .pov, and builder.bat file.
----------------------------------------
@rem broken_files.bat
@rem
@rem Variables to be defined:
@rem suffix - The extension of the image files, eg. png.
@rem
@rem Author: Burkhard Reike, 2011-10-26
@echo off
setlocal EnableDelayedExpansion
set suffix=png
for %%f in (*.%suffix%) do set allfiles=!allfiles! %%f
for /f "tokens=1 delims= " %%f in (render.log) do set logged=!logged! %%f
for %%g in (!allfiles!) do (
set found=0
for %%h in (!logged!) do if "%%g"=="%%h" set found=1
if !found! EQU 0 echo %%g
)
pause
----------------------------------------
Linux (Ubuntu)
==============
There are several good reasons to use Linux for Povray rendering. Linux requires
few disk space, few RAM, and renting a Linux server is usually cheaper than
renting a Windows server since you don't have to pay for a license. The
following scripts were created for the Ubuntu 8.04 Linux distribution. The
scripts builder.sh and broken_files.sh are equivalent to the Windows scripts
described above. Additionally, the script add_render_process.sh will be
presented which can automatically mount a folder from a remote machine through
SSHFS such that you can start Povray instances on several machines, all
instances rendering into the same shared folder. All operations can be executed
from a login shell, you don't need a desktop environment like KDE or so.
For sharing a remote render folder, you need to have SSH/SFTP support installed
(most Linux distributions, even the minimal ones, come with it already
installed), and the SSH port of each machine used should be open such that
remote mounting can be applied. SSH is actually the only connection you will
need since it's also the connection of choice for remote login and file transfer
(eg. with the Windows programs PuTTY and WinSCP), since it's encrypted. Apart
from SSH/SFTP support, you should have the Screen tool installed. Screen will
enable you to logout from a machine without hanging up the render and mount
processes. You just have to execute Screen to open a new shell, start the
scripts, detach from Screen, and logout. Screen will stay alive, and so will the
sshfs and render processes started from it.
To install Screen in Ubuntu, type:
$ sudo apt-get install screen
Let's assume that you have a user account called "john" on each machine. Then
you should organize the render folders and scripts as follows.
On each machine, put the add_render_process.sh script into a certain folder, eg.
a folder inside your home directory called "povray":
/home/john/povray/add_render_process.sh
One machine, which we call master, must also contain the render folder. The
render folder contains the .ini, .pov, and remaining .sh files. Besides, it is
the folder into which the image files will be rendered:
/home/john/povray/render/builder.sh
/home/john/povray/render/broken_files.sh
/home/john/povray/render/landscape.ini
/home/john/povray/render/landscape.pov
All other machines, which we call clients, do not have a render folder
preinstalled. Instead, they will mount it from the master machine by the help of
the add_render_process.sh script.
For this purpose you will need to know the IP-address of the master machine. You
can find it out while logged in on the master machine with
$ ifconfig eth0
or
$ ifconfig eth1
depending on the hardware configuration. Inspect the output of the ifconfig
command. The number right of the "inet address:" entry is your IP-address.
Possibly you have two IPs, one public IP, and one private IP to be used for
inter-server communication. If so, you should pick the private IP of the master
machine.
So much for the setup. Now for starting the render processes.
You don't necessarily have to start render processes on the master machine since
its main purpose is to keep the render folder. But if you also want to use it
for rendering, proceed like this. First, login on the master machine (as "john")
and go to the povray folder. Then run Screen. Let's assume that each machine has
four cores, so run add_render_process.sh four times (with the render folder path
and the builder script name as arguments). By this, four builder.sh scripts will
be started, which again start Povray instances to render image files. Finally,
and detach from Screen. Then you can logout. The render processes will keep
running in the Screen shell:
$ cd povray
$ screen
$ ./add_render_process.sh /home/john/povray/render builder.sh
$ ./add_render_process.sh /home/john/povray/render builder.sh
$ ./add_render_process.sh /home/john/povray/render builder.sh
$ ./add_render_process.sh /home/john/povray/render builder.sh
[Now press "Ctrl+A" and then "D" to detach from Screen.]
$ logout
Now for starting render processes on the client machines. On each client
machine, you basically perform the same steps as on the master machine. The only
difference is that the calls of the add_render_process.sh will have two
additional arguments, namely the name of the user on the master machine who owns
the render folder, and the IP-address of the master machine. Let's say the
IP-address is 12.34.56.789. Then the commands to be executed on each client
machine look like this (again, let's assume we want to run four Povray instances
in parallel):
$ cd povray
$ screen
$ ./add_render_process.sh /home/john/povray/render builder.sh john 12.34.56.789
$ ./add_render_process.sh /home/john/povray/render builder.sh john 12.34.56.789
$ ./add_render_process.sh /home/john/povray/render builder.sh john 12.34.56.789
$ ./add_render_process.sh /home/john/povray/render builder.sh john 12.34.56.789
[Now press "Ctrl+A" and then "D" to detach from Screen.]
$ logout
The add_render_process.sh script will automatically install the SSHFS package,
add the user to the fuse group such that he can start remote connections, mount
the render folder and start the builder.sh script.
You will probaly be asked two times for passwords. The first time occurs when
the first "sudo" command in the add_render_process.sh script is executed. Then
you will have to enter the root password of the client machine. The second time
you will have to enter the password of user "john" on the master machine, in
order to establish the remote connection through sshfs. When connecting for the
first time to the master machine from the client machine, you'll also have to
confirm that you want to establish the connection.
Here are the scripts:
----------------------------------------
#!/bin/bash
# add_render_process.sh <full-path-of-render-directory> <name-of-renderscript>
# [<remote-user-name> <remote-host-address>]
#
# Author: Burkhard Reike, 2011-10-21
if [ $# -ne 2 -a $# -ne 4 ]; then
echo "Usage: $0 <full-path-of-render-directory> <name-of-renderscript>"
echo " [<remote-user-name> <remote-host-address>]"
exit 1
else
RENDERDIR=$1
if [ $RENDERDIR != */ ]; then
RENDERDIR="$RENDERDIR"/
fi
RENDERSCRIPT=$2
if [ $# -eq 4 ]; then
USER=$3
HOST=$4
ISMASTER=false
else
USER=''
HOST=''
ISMASTER=true
fi
fi
if [ -d "$RENDERDIR" -a -n "$(ls -A $RENDERDIR 2> /dev/null)" ]; then
# Render directory already exists and is non-empty, so don't mount it.
if [ !$ISMASTER ]; then
echo "$0: Render directory seems already mounted. Mounting omitted."
fi
else
# Render directory doesn't exist or it's empty. Try to mount it.
if [ !$ISMASTER ]; then
if [ ! -d "$RENDERDIR" ]; then
mkdir $RENDERDIR
fi
#
# Mounting the render directory via sshfs
#
sudo chown `whoami`:`whoami` $RENDERDIR # Fix the render dir ownership
sudo apt-get install sshfs # Install the sshfs package. This may have
# to be adjusted, depending on your Linux
# distribution.
sudo modprobe fuse # Load the fuse module
sudo adduser `whoami` fuse # Add us to the fuser group
sudo chown root:fuse /dev/fuse # Fix /dev/fuse ownership
sshfs -o ServerAliveInterval=15 $USER@$HOST:$RENDERDIR $RENDERDIR
# Wait till the render directory is non-empty
echo "$0: Waiting for content in the render directory..."
until [ -d "$RENDERDIR" -a -n "$(ls -A $RENDERDIR 2> /dev/null)" ]; do
sleep 2
done
echo "$0: Render directory was mounted sucessfully."
else
# The master process needs a non-empty render directory,
# but could not find it.
echo "$0: Could not find script $RENDERDIR$RENDERSCRIPT. Exiting."
exit 1
fi
fi
echo "$0: Starting background render script $RENDERDIR$RENDERSCRIPT"
cd $RENDERDIR
../$RENDERSCRIPT &>/dev/null &
----------------------------------------
#!/bin/bash
# builder.sh
#
# Variables to be defined:
# POVNAME - The name of the scene to render, without extension.
# DIGITS - The number of digits in the index of the image files.
# SUFFIX - The extension of the image files, eg. png.
# S - Number of the initial frame.
# E - Number of the final frame.
#
# Author: Burkhard Reike, 2011-10-21
echo "Starting: `hostname` `date`" >> render.log
POVNAME=landscape
DIGITS=5
SUFFIX=png
S=1
E=13741
while [ $S -le $E ]; do
INDEX=`printf "%0"$DIGITS"d" ${S}`
FILENAME=$POVNAME$INDEX.$SUFFIX
if [ ! -f $FILENAME ]; then
touch $FILENAME
BEFORE="$(date +%s)"
povray +SF$S +EF$S -d +I $POVNAME.ini &> /dev/null
AFTER="$(date +%s)"
ELAPSED="$(expr $AFTER - $BEFORE)"
echo "$FILENAME $ELAPSED `hostname`" >> render.log
fi
S=$(($S+1))
done
echo "Finished: `hostname` `date`" >> render.log
----------------------------------------
#!/bin/bash
# broken_files.sh
#
# Author: Burkhard Reike, 2011-10-21
for F in *.png
do
GREPOUTPUT="$(grep $F render.log)"
if [ -z "$GREPOUTPUT" ]; then
echo $F
fi
done
----------------------------------------
That's it! I hope you will find all this useful.
For updates and other stuff, visit http://www.verklagekasper.de/video
Cheers,
Burkhard
Post a reply to this message
|
|
de> wrote: