Good idea, and probably long overdue too.

I agree with Warp's original idea that there should be several different
types of scenes, including one all-rounder.  This would allow those who
favour a certain type of scene to make meaningful comparisons.  For
example, if someone likes scenes with lots of plants built from
enourmous meshes, then they may well be much more interested in seeing
the benchmarks for scenes with high memory usage, and high parsing times
(if e.g. their plants were procedurally created and placed).  And of
course the all-round scene is vital for an overall comparison.

Perhaps it is not so important to accomodate those with lower spec
computers.  I would assume that one of the main uses of a benchmarking
system is to compare the performance of higher-end systems, as the
results for these would be very helpful for anyone thinking of upgrading
(e.g. Athlon/P4 comparisons).  Of course, there is the case of those
deciding which OS to run on a lower spec machine that will be used for
rendering, but perhaps with enough results publicly available, they
could get their answer without having to do the test themselves.

As regards rendering time - it might be wise to keep it under 10 hours,
but not too far under so as to allow for future developments.  This
would be handy for leaving the computer on overnight whilst rendering
the scene.  Longer times would be difficult to trust, as people may well
have used the computer for other things during the render.  Slower
computers that had been playing MP3s at the same time would return
results that were not really worth anything.  Well, unless there was a
benchmark category for rendering whilst playing music :-)

Oh, I guess we should decide pretty soon, as someone's going to have to
produce the scenes in the first place, and it would make sense for them
to be in the 3.5 distribution.

Post a reply to this message