Another two cents worth on judging procedures
and categories:

Keeping the system simple and keeping the number
of categories and options down to a reasonable
minimum is a worthy goal worth making a few
sacrifices for, but let's not go too far in that
direction and unduly constrain the judges' ability
to give constructive feedback.

Perhaps it would be a good idea to supply short
definitions for each category which are worded
specifically enough so as to eliminate

so specific as to impose an exterior set of
aesthetic priorities.  I offer the following as a


Category 1: TECHNICAL MERIT:
The degree to which the image shows mastery of
the image creation process.

Category 2: ARTISTIC MERIT
The degree to which the image evokes in you
thoughts or feelings you deem significant or
praiseworthy.

Category 3: INTERPRETATION OF TOPIC:
The degree to which you feel the image expresses
the topic idea in a particularly original or
profound way.

While it is true enough that there will be some
overlap between any such list of categories, there
are significant enough differences between each of
three categories above to merit their separate existences.

On the subject of the number of gradations for each
category, I would propose that rather than choosing an
arbitrary number, we provide a descriptive term for each
value in a given category as well as a number and see where


TECHNICAL MERIT:
[0] = Lacks both knowledge and effort
[1] = Shows BEGINNER level skills
[2] = Shows good BASIC skills
[3] = Shows ADVANCED level skills
[4] = Shows EXPERT level skills
[5] = Shows EXTRAORDINARY skills(WOW!)

ARTISTIC MERIT:
[0] = Offensively trite and/or cliche
[1] = VERY LITTLE aesthetic impact
[2] = MODERATE aesthetic impact
[3] = SIGNIFICANT aesthetic impact
[4] = HIGH aesthetic impact
[5] = PROFOUND aesthetic impact (WOW!)

INTERPRETATION OF TOPIC:
[0] = Off-topic / inappropriate
[1] = Rather OBVIOUS idea
[2] = NOT A BAD idea
[3] = VERY GOOD idea
[4] = EXCELLENT idea
[5] = BRILLIANT idea (WOW!)

Under a system like the one above, I would suggest
that the general instructions indicate that both
[0]s and [5]s indicate unusual circumstances and
should be used extremely sparingly.  Also, there
should be a very strong suggestion, if not an
absolute rule, that a [0] or a [5] should be
accompanied by an explanation in the comments.

One goal that I would suggest for the new contest
is that it actively promote the participation and
development of newcomers to the field.  I hope that it
won't be one of those contests that routinely offers
kudos to a hand full of established elite artists
and a thumbing of the nose to everyone else.  Toward
this end, I would like to offer two specific proposals:

First,the competition should employ a comment system that
emphasizes CONSTRUCTIVE criticism and feedback for ALL
participants.  Feedback, encouragement and help are
among the prime motivators that will bring new people
into ray tracing and help to develop the elite artists
of the future.

Second, the terminology for rating
should offer non-derogatory options to describe the
work of those who are still in the rather early stages
of learning.  In the system above, for example, a newbie
might actually be encouraged by a Technical Merit rating



Well, thanks for taking the time to read my rather
long winded comments. The gist of of what I have to
say can be condensed down to this: I hope that the
new IRTC will be a fun and enjoyable experience that
offers both challenge and a sense of community to new
and expert ray traces alike.

Thanks, again, to all who are making the effort to
bring the IRTC back.

Best Regards,
Mike C.

Post a reply to this message

As might have been menitioned elsewhere, the problem with one category is
thematic deviation (yes I've done it).  But what about the fact that someone
has a really great idea for their image, and just didn't develop that idea.
(Again, I've done this).  If there was one category, there would be no way to
honor that person for a great idea, but a lousy image.

In my thoughts, I think the categories should be the following, because these
are distinct, but still tie together with each other:
 -Concept/originality
 -Technical Merit
 -Impressiveness

Concept/originality is basically how the topic is developed in the image.  It
shows how well through the image was thought. e.g. In one of the Stills rounds,
called "Alien Worlds," there were some beautiful Star Trek matte paintings
rendered, but that's not really original as an idea.  So, Plus for the
artistic, minus for the orignality

Technical Merit is basically how the user approached the image s/he made.  Did
s/he use some impressive code to produce a high degree of photo-realism, or did
he simple put a few spheres next to each other, and just image-map them.
Basically, it's how much work the user put in to achieve his goal.

And Impressiveness is basically just how well the image pulls off as a good
image.  The more it makes someone go "Wow" and the more it looks good, might be
the scale.  That doesn't just mean that photo-realism is impressiveness.  I
mean, I get more "wowed" from Pixar's Toy Story, than I do from Harry Potter's
Magic Effects.  It just is how good the image looks.

By all means, don't quote me on saying that's how it should be.  I personally
think this would work well, with a bit of tweaking here or there, but that
system seems good to me.

And to David, if you use this system, or something very similar, by all means,
you may quote me in the voting guidelines, or rules, etc.

Any other ideas on those categories?

Post a reply to this message

One of the most offensive scores I got on an IRTC submission was 10 10 10. And
on that round, when I voted on others, I made sure my vote expressed exactly
what the artist were doing well (artistic, technical, theme).  This basically
means that everyone else gave me 10 10 10 for the average to be 10 10 10. No
one put any thought into the vote.

For a vote to be meaningful, it has to COST the voter something.  In the old
voting system, one could also give 20 20 20 to one's favorite, and 1 1 1 to all
the ones one didn't like.  Others might give out a strict normalized
distribution around ten, and reserve 20's for once a lifetime beauties.
Average up these two voters' votes, and you're throwing away the careful
voter's analysis.

For the Short Code Competition,  I came up with a voting scheme where you cite
your favorite six entries, and then gave 6 points to the favorite, 5 to their
2ndmost, and so on.  In this way, everyone's opinion is weighted the same.
There are more complicated schemes where folks have an IRTC Dollar to give
away, and they can divide it up any way they want-- all to one or a penny to
100 entries-- I say this is inherently fair, as every voter gets a dollar.

Analogy: My local newspaper once spent a lot of time and effort into a reader
vote on which comics to keep.   They let everyone vote yes or no on keeping
every comic. (Note there's no COST to vote here!)  The comic "Doonesbury" came
in about last, but they never dropped it.  My hunch is they knew that 55%
didn't like it but maybe 10% were passionate about it.  Because the poll didn't
measure user passion, the paper ended up ignoring the results.

As far as WHO votes,  I'd be more inclined to have real people vote than male
nerds who use freeware graphics programs. (DISCLAIMER: I'm one of the latter.)
Maybe it's impractical to line up a set of artistic, educated, and/or
interesting people, but that would be the coolest.



David Buck <dav### [at] simberoncom> wrote:


> I'd like to find a nice way to prevent or reduce voting anomalies in the
> new IRTC web site.  I don't want people to be able to vote multiple
> times in one contest and skew the votes.
>
> So I'd like to know if you have any suggestions on how to do this.
>
> 1) How do we accept new voters?
>  - automatically with e-mail confirmation?
>  - only with admin approval?
>  - other ideas?
>
> 2) Should submitters be allowed to vote in the round they submit to?
>
> I intend to allow voters to rate each image in 4 categories
>  - overall
>  - technical merit
>  - artistic merit
>  - concept and interpretation of theme
>
> Each rating is on a scale from 0 to 10.
>
> 3) How should the scores be combined
>
> 4) Should we allow partial voting?
>  - by this I mean rating some images but not all
>  - I have no assurance that they even looked at all of them
>  - if you rated the first 5 pictures but none of the others,
>   should any of your ratings be considered?
>
> Your thoughts would be appreciated.
>
> Thanks
> David Buck

Post a reply to this message

On Mon, 10 Mar 2008 06:39:00 -0700, Randal L. Schwartz wrote:

> You could let each person select a "lo" and "hi" point of their own
> scale, and then normalize the values to a floating point from 0.0 to
> 1.0.

Ewww.

Rather than do that, why not just let people pick from a 1-9 scale, and 
if they want it more coarse, they can ignore every other digit?

Letting people pick their own scale will create problems - someone thinks 
they picked a scale of 3, but the app saw it as a scale of 9, so instead 
of giving the top rating, they give one of 33%?

No way....

Jim

Post a reply to this message

On Mon, 10 Mar 2008 08:49:22 -0500, Warp wrote:

>   Does anyone have any good idea about how to avoid cross-contamination
> of categories?

If you get enough votes, cross-contamination of categories will factor 
itself out in the averages.

Jim

Post a reply to this message

47d2ce11$1@news.povray.org...
> I intend to allow voters to rate each image in 4 categories
> - overall
> - technical merit
> - artistic merit
> - concept and interpretation of theme

It should be noted that with the IRTC, the 3 notes (techical, artistic, 
concept) were always extremely correlated. In the 2006 rounds the 
correlation coefficients between these notes were in the 0.92-0.96 range: 
it's basically the same note (correlation coefficients run between 0 and 1 
where 0 = no correlation). One can run a regression and predict one note 
with another with a small margin of error. I did the same calculations in 
the past, with the same results. In a nutshell, folks who are smart enough 
to understand a concept and come up with a good idea also have decent 
artistic and technical skills. Now that's really surprising ;)

There are occasional outliers of course - for instance images with a 
moderately good artistic note and so-so concept - but on the whole having 
several voting categories is more intellectually satisfying than 
statistically meaningful.

G.

-- 
*****************************
http://www.oyonale.com
*****************************
- Graphic experiments
- POV-Ray, Cinema 4D and Poser computer images
- Posters


>
> Each rating is on a scale from 0 to 10.
>
> 3) How should the scores be combined
>
> 4) Should we allow partial voting?
> - by this I mean rating some images but not all
> - I have no assurance that they even looked at all of them
> - if you rated the first 5 pictures but none of the others,
> should any of your ratings be considered?
>
> Your thoughts would be appreciated.
>
> Thanks
> David Buck

Post a reply to this message

"gregjohn" <pte### [at] yahoocom> wrote:
> One of the most offensive scores I got on an IRTC submission was 10 10 10.

>...This basically means that ...No one put any thought into the vote.
>
> For a vote to be meaningful, it has to COST the voter something. ...
>
> For the Short Code Competition,  I came up with a voting scheme where you cite
> your favorite six entries...
.....
> As far as WHO votes,  I'd be more inclined to have real people vote than male
> nerds who use freeware graphics programs. (DISCLAIMER: I'm one of the latter.)
> Maybe it's impractical to line up a set of artistic, educated, and/or
> interesting people, but that would be the coolest.

Although I'm very sympathetic to the sentiments that you express here and
recognize that they are thoughtful and well-intentioned, I nonetheless feel
compelled to mention a few problems that I see with the particulars:

Systems under which a judge votes only for a few favorites tend to leave
everyone except a few elites out in the cold with no constructive feedback.
This bodes poorly for developing the community and building up the skills of
the up and coming.  I guess this really comes down to a question of goals and
priorities.  *ARE* we interested in building community and encouraging



Also, sadly, one of the things that must be considered is how a system might be
abused by those who do not share your good intentions and desire for fairness.
If I'm more interested in influencing the outcome in favor of my best friend or
my own little clique than in judging the entries fairly, I can do much more
damage by concentrating all 100 of my points than I can by being just one voter
with a skewed motive.  (I freely acknowledge that NO system solves the ulterior
motive problem entirely, but voting only for favorites exacerbates it. )


effort.  Not only should voters be required to vote on ALL entries, but also to
leave at least one coherent comment on each entry AND to explain why any image
was given either the highest or lowest possible rating.  To those who would
suggest that this is asking too much of judges, I would reply that it's nothing
compared to the effort of preparing a serious entry.

As for the WHO of voting, (and this is merely personal feeling with no real
objective logic behind it) I share your desire to get feedback from both inside
and outside the world of those who produce ray trace images, but I would require

don't give a rodent's rectum how people who rush home from their marketing jobs

would rate my work.  Also, a significant percentage of the judging will have to

about how one can improve.

Respectfully,
Mike C.

Post a reply to this message

I like the idea of forcing people to leave comments.  I usually try to do this,
so as to give others a chance to see why I felt the way I did.  I remember that
once,I even forgot something in the comment, and left an anonymous comment just
to add something.

Back then, I was an amateur Ray-tracer, and was soo hoping for comments on
improving my images.  In both stills competitions I entered, I recieved one
comment.  However, Animations were kinder, in giving me about 5 comments.

I think some people don't realize that there are amateurs out there who enter
partly for fun, and partly for just trying to get better at something they
really love.  When I got no comments, I thought it was because no one cared for
such low images.  Admittingly, one did get last place, but I never got to figure
out why it did.  I deduced most of it over a years worth of developing skills.
So, yeah, I agree that comments should be highly recommended, if not forced.
Someone who takes the time to write out comments for every entry, has
definitely thought each image through, and also hopefully graded each likewise.

As for other things, points need to be something the whole world understands.  I
definitely think there needs to be a guide as to what each rating means(they
used to do that for the 1, 10, and 20 marks).  Points mean a lot about how well
an image really is, and they themself can tell a contestant what there image
really is.  So, that's why I'm supporting categories.

David, have you come to some decisions on what to do, or are you still waiting
out?

Post a reply to this message

Nimish Ajmani wrote:
> David, have you come to some decisions on what to do, or are you still waiting
> out?

I've been following all the discussions here and putting together what I 
feel are the best recommendations.  I'll put together a summary of those 
later this evening and post them here.

Thanks to everyone for their feedback - it's very helpful and has 
changed my approach to the design of the site.

David Buck

Post a reply to this message

> Thanks to everyone for their feedback - it's very helpful and has
> changed my approach to the design of the site.

I think I speak for everyone when I say not a problem.

  --Nimish

Post a reply to this message