David Buck <dav### [at] simberoncom> wrote:
> 2) Should submitters be allowed to vote in the round they submit to?

  Definitely. My guess is that at least 90% of the votes will be from
the submitters themselves. Without their vote the amount of votes could
be very low.

  Natually no voting for your own submission.

-- 
                                                          - Warp

Post a reply to this message

David Buck <dav### [at] simberoncom> wrote:
> I'd like to find a nice way to prevent or reduce voting anomalies in the
> new IRTC web site.  I don't want people to be able to vote multiple
> times in one contest and skew the votes.
>
> So I'd like to know if you have any suggestions on how to do this.
>
> 1) How do we accept new voters?
>  - automatically with e-mail confirmation?
>  - only with admin approval?
>  - other ideas?
>
> 2) Should submitters be allowed to vote in the round they submit to?
>
> I intend to allow voters to rate each image in 4 categories
>  - overall
>  - technical merit
>  - artistic merit
>  - concept and interpretation of theme
>
> Each rating is on a scale from 0 to 10.
>
> 3) How should the scores be combined
>
> 4) Should we allow partial voting?
>  - by this I mean rating some images but not all
>  - I have no assurance that they even looked at all of them
>  - if you rated the first 5 pictures but none of the others,
>   should any of your ratings be considered?
>
> Your thoughts would be appreciated.
>
> Thanks
> David Buck

I'm not sure how easy it will be to program the system, but here's my idea for
you.

One, to prevent skew votes, only a panel judge (i.e. someone ruled by the IRTC
staff as a reliable voter) or an entrant can vote for a particular round.  To
ensure voting is not skewed, a voter must enter the e-mail adress they
submitted an entrant with, and a password which is associated with an account
they have created.  When a voter enters the system, they can vote, change
votes, and add coments in their name.  The system will only allow voters with
proper access.  The system allows them to vote on all images except their own,
and automatically stores their given score.

Partial voting can be allowed if a particular system is set up.  In this system,
the total is calculated by the total score earned by an entrant, divided by the
number of votes the entrant recieved.  This would make the scores comparable to
each other.

For example, entrant one had 5 votes, with scores of 35, 29, 32, 40, and 31.
Entrant two only had three votes, with scores of 28, 37, and 34.  Their final
score would be 33.4, and 33.  This way, despite vote amounts, the scores would
be close enough for a winner.

I personally would like a voter to be forced to vote for all, but that's how I
wuold deal with partial voting.

Also, if you don't mind, I suggest adding a fifth category called "Wow factor",
which is basically like, without thinking about this image at all, what did you
first intrepret it as.  I.e. a first interpretation.

I think that answers everything.  Hope I helped.

Post a reply to this message

On Sat, 08 Mar 2008 12:34:09 -0500, David Buck <dav### [at] simberoncom> wrote:

>Your thoughts would be appreciated.

I like the idea of voting without entering because I don't often have the time
to create an entry but am very interested in the contest. I think that votes
should only be counted if all entries are voted on. 
As Warp pointed out Entrants will probably make up the majority of voters.
I think that the votes/scores should be hidden until the end of the competition
to stop skewed voting.
The concept of weighted scores for different categories should be discussed and
I have always had a problem with technical merit. How would I as a Pover know
how hard it is to use another programme? Also simple techniques can make great
art.

Regards
	Stephen

Post a reply to this message

David Buck wrote:
> I'd like to find a nice way to prevent or reduce voting anomalies in the 
> new IRTC web site.  I don't want people to be able to vote multiple 
> times in one contest and skew the votes.
> 
> So I'd like to know if you have any suggestions on how to do this.
> 
> 1) How do we accept new voters?
>     - automatically with e-mail confirmation?
>     - only with admin approval?
>     - other ideas?

If they're contributors to POV-Ray V4, they can vote.  Otherwise they 
have to have an entry in the round.

Not that programmers are better judges, but I was interested in 
rewarding those who help program V4.

> 2) Should submitters be allowed to vote in the round they submit to?

Sure.

> I intend to allow voters to rate each image in 4 categories
>     - overall
>     - technical merit
>     - artistic merit
>     - concept and interpretation of theme

I'm not sure the overall category is that important.

 > Each rating is on a scale from 0 to 10.

That makes more sense than 1 to 20, but artistic merit and concept are 
still too subjective to be directly quantifiable.  On the other hand, 
whether one work is more artistically pleasing, or has a more 
interesting/entertaining topic, than another, is a question most people 
can answer without difficulty.

We can argue until doomsday about what a six out of ten means, or should 
mean, for artistic merit, and still not answer the question in any 
meaningful way.  On the other hand, while people may not agree on which 
of two renderings is more pleasing, each person will have a good idea of 
which one he/she prefers.

So we can vote on that basis, by having each voter rank the entries in 
each category, and awarding points based on the ranking.

Regards,
John

Post a reply to this message

Disclaimer:  I've never entered this contest, but I've always thought 
"someday."

"Overall" and Nimish's "Wow Factor" sound the same to me.  Either one 
would act as a way for a voter to compensate if their general impression 
is different from the average of what they think for the other 
categories.  I think the ability to use that category for such 
compensation means that the final scoring can/should be a simple sum of 
the category scores.


I'd keep voting to entrants and established panelists.  A "people's 
favorite" for anonymous voters can be an honorable mention, but not 1st, 
2nd, 3rd.

I would set it up so that each voter automatically defaults to awarding 
5 points for each category for each submission other than their own. 
Assigned a voting key, voters can change their votes and comments up 
until the deadline, except of course, for their own.


A variation:
There is a baseline of 5 to start with.  Like liquid in a vessel, if a 
voter ranks-up some entries, the baseline is displaced - it goes down 
some fraction of a point.  If the voter ranks-down some entries, the 
baseline goes up.  Un-scored submissions recieve the voter's baseline, 
as limited to the range 0 to 10.

Charles

Post a reply to this message

John VanSickle escribió:
> David Buck wrote:
>> 1) How do we accept new voters?
>>     - automatically with e-mail confirmation?
>>     - only with admin approval?
>>     - other ideas?
> 
> If they're contributors to POV-Ray V4, they can vote.  Otherwise they 
> have to have an entry in the round.
> 
> Not that programmers are better judges, but I was interested in 
> rewarding those who help program V4.

Hey, nice idea :)

Post a reply to this message

Just a question to the programmer responsible for the new site.  Will comments
be automatically added to past rounds by computer, as I always submitted
comments on the old IRTC, and they never appeared.

Post a reply to this message

"David Buck" <dav### [at] simberoncom> schreef in bericht 
news:47d2ce11$1@news.povray.org...

> 1) How do we accept new voters?
> - automatically with e-mail confirmation?
> - only with admin approval?
> - other ideas?
Submitters+ judges are accepted voters (except for own submission of 
course). A system should be in place to avoid multiple voting (not sure how 
to do that)
Non-entrants should also be able to vote, again with security in place to 
avoid multiple voting.
I am not sure if e-mail confirmation would be enough...

>
> 2) Should submitters be allowed to vote in the round they submit to?
yes, except their own submission.

>
> I intend to allow voters to rate each image in 4 categories
> - overall
> - technical merit
> - artistic merit
> - concept and interpretation of theme
Overall could be the Wow factor mentioned elsewhere. Technical merit is 
always a difficult category to rate. How does one define that really?

>
> Each rating is on a scale from 0 to 10.
Yes, much better I believe than from 0 to 20.

>
> 3) How should the scores be combined
>
> 4) Should we allow partial voting?
> - by this I mean rating some images but not all
> - I have no assurance that they even looked at all of them
> - if you rated the first 5 pictures but none of the others,
> should any of your ratings be considered?
No. I would discourage partial voting.

Thomas

Post a reply to this message

David Buck <dav### [at] simberoncom> wrote:
> I'd like to find a nice way to prevent or reduce voting anomalies in the
> new IRTC web site.  I don't want people to be able to vote multiple
> times in one contest and skew the votes.
>
> So I'd like to know if you have any suggestions on how to do this.
>
> 1) How do we accept new voters?
>  - automatically with e-mail confirmation?
>  - only with admin approval?
>  - other ideas?
>
> 2) Should submitters be allowed to vote in the round they submit to?
>
==============================================================
Opening judging to the general public is generally a bad idea
in that it invites voting fraud.   I rather liked the TC-IRTC
idea of allowing voting by both those who have submitted entries
to the current round and those who have entered within a given
period of time (say three rounds back).  Using an email
confirmed user name and password system seems the most
logical choice.
==============================================================

> I intend to allow voters to rate each image in 4 categories
>  - overall
>  - technical merit
>  - artistic merit
>  - concept and interpretation of theme
>
> Each rating is on a scale from 0 to 10.
>
==============================================================
This seems reasonable. Nimish Ajmani’s suggestion of including
a "Wow Factor" could be incorporated by fine tuning the name of
the "overall" category (perhaps to something like "overall
impact") rather than creating a fifth category.
==============================================================

> 3) How should the scores be combined
>
==============================================================
A percentage of total possible points would be pretty clear and
easy to interpret.
==============================================================

> 4) Should we allow partial voting?
>  - by this I mean rating some images but not all
>  - I have no assurance that they even looked at all of them
>  - if you rated the first 5 pictures but none of the others,
>   should any of your ratings be considered?
>
==============================================================
Personally, I feel rather strongly that only the votes of those
who have taken the time and effort to rate ALL of the submissions
should be counted.
==============================================================

> Your thoughts would be appreciated.
>
> Thanks
> David Buck

==============================================================
Thank YOU for taking the time to seek participant feedback.
==============================================================

Best Regards,
Mike C.

Post a reply to this message

Another idea occurs to me:

Allow voting by the general public.  Average all of these votes 
together.  This becomes one vote, which has weight equal to the entrant 
judges and the permanent judges.

This allows the public to vote, but minimizes the harm that a spam 
campaign can do.

Regards,
John

Post a reply to this message

John VanSickle <evi### [at] hotmailcom> wrote:
> Another idea occurs to me:
>
> Allow voting by the general public.  Average all of these votes
> together.  This becomes one vote, which has weight equal to the entrant
> judges and the permanent judges.
>
> This allows the public to vote, but minimizes the harm that a spam
> campaign can do.
>
> Regards,
> John


Although a good idea, couldn't someone just submit like, 100 perfect votes for
themself, and improve odds.

Personally, I think voting should be kept to just inside entrants.  Even there,
an entrant can give a 0 score to everyone else, but everyone is at a certain
level of fair play, and would therefore not submit to such low values (I
hope!).

I'm not making the choice however.  The idea is good, but it just doesn't seem
to work too well.

But maybe, you could use ip addresses to back check spam votes.???

-Nimish

Post a reply to this message

David Buck wrote:
> Your thoughts would be appreciated.

Might be too late to mention this, but I've always felt that a 5-point 
score system is easier for users :)

After all, pretty much everyone is familiar with it, with the values 
being "Horrible, poor, average, good, great".  Rarely do you need more 
granularity over the actual scores, as well.

With a ten point system, some people might feel that "5" is average, 
while other people feel that anything less than "9.0" stinks.  Of 
course, individual voter tendencies are averaged out over the group, but 
its still nice to have as much consistency as possible.

As far as voting goes... I think only entrants and panel judges should 
be considered "official" voters, with the caveat that entering one round 
should qualify you to vote for the next 12 months.  Partial votes (where 
not every images was rated), and votes from non-participants should be 
tracked separately, perhaps as a "populist" vote, for curiosity's sake, 
but should not affect the final score.

But then, I haven't entered since around 2001, so what do I know? :)

-- 
...Ben Chambers
www.pacificwebguy.com

Post a reply to this message

Chambers wrote:
> David Buck wrote:
>> Your thoughts would be appreciated.
> 
> Might be too late to mention this, but I've always felt that a 5-point 
> score system is easier for users :)

A 5 point system is just as easy to implement as a 10 point system.  I'm 
open to this suggestion if people think it's the best way.

David Buck

Post a reply to this message

Nimish Ajmani wrote:

> But maybe, you could use ip addresses to back check spam votes.???
> 
> -Nimish
> 

The problem with IP addresses is that many people use DHCP which 
dynamically allocates IP addresses.  The same person may get different 
IP addresses each time they logon and the IP address used by a spammer 
may later be used by a legitimate voter.  Besides, I would like the 
ability to vote from whatever IP address I happen to be using at the time.

David Buck

Post a reply to this message

David Buck <dav### [at] simberoncom> wrote:
> Nimish Ajmani wrote:
>
> > But maybe, you could use ip addresses to back check spam votes.???
> >
> > -Nimish
> >
>
> The problem with IP addresses is that many people use DHCP which
> dynamically allocates IP addresses.  The same person may get different
> IP addresses each time they logon and the IP address used by a spammer
> may later be used by a legitimate voter.  Besides, I would like the
> ability to vote from whatever IP address I happen to be using at the time.
>
> David Buck

That's true.  I hadn't thought about that.

As for points, I think 10 is better than 5, because the more points allows for a
greater degree of quality decision.
I liked the 20 point system, because, I used it much like the French grading
system, which was based on 20 points, but if I had to choose between 5 and 10,
I'd prefer 10, again, for a greater degree of quality variance.

I hope that helped.

Post a reply to this message

David Buck wrote:

> A 5 point system is just as easy to implement as a 10 point system.  I'm 
> open to this suggestion if people think it's the best way.

I think 5 points is a bit coarse, there will hopefully be many
"great" images submitted (although not by me) and the voter likely
wants to give them "great" votes but still be able to make his or
her favourite stand out a bit ;) 0-10 sounds reasonable to me.

Regarding the categories, it seems a bit redundant
to vote for the "overall" image. Like, I find it's
technically horrible and of no artistic value, but a
great picture anyway? Also, the distinction between
artistic and concept was always a bit blurred.

How about just two categories "artistic" and "technical",
where "artistic" encompasses the concept as well? If this
puts too much emphasis on the technical aspects, weighted
averages such as 2:1 or 3:2 could be used. Using fewer
categories also reduces tediousness of voting when a
successful round attracts 50 to 100 entries.

Apart from that, I'd also support keeping the voting
confined to panel judges as well as present and past
entrants - whatever the time limit. Prizes could
include winning a longer right to vote ;)

Finally, I think partial votes should not count (in fact
should not be possible to submit in an automated system),
as they might be biased by the ordering of the list, and
votes should not be disclosed until voting period ends.

Post a reply to this message

>>>>> "David" == David Buck <dav### [at] simberoncom> writes:

David> The problem with IP addresses is that many people use DHCP which
David> dynamically allocates IP addresses.  The same person may get different
David> IP addresses each time they logon and the IP address used by a spammer
David> may later be used by a legitimate voter.  Besides, I would like the
David> ability to vote from whatever IP address I happen to be using at the
David> time.

Not only that, but if you're voting from inside, say, yahoo.com, you will be
coming in via the corporate proxy, which is shared by thousands of others, and
might actually change from hit to hit, making your own votes invalid.

"IP" is not "user".  Never make that mistake.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<mer### [at] stonehengecom> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

Post a reply to this message

Christian Froeschlin wrote:
> How about just two categories "artistic" and "technical",
> where "artistic" encompasses the concept as well? If this
> puts too much emphasis on the technical aspects, weighted
> averages such as 2:1 or 3:2 could be used. Using fewer
> categories also reduces tediousness of voting when a
> successful round attracts 50 to 100 entries.


How pretty art is, is different from how clever/witty it is -- just as 
how effective a technique is in accomplishing an effect is also 
completely different from how technically involved it was to produce. 
How do you compare a well chosen/applied simple technique against an 
overly ambitious (read Rube-Goldberg-inspired) technique which showed 
better promise than first-shot results?   I think it would be nice to be 
able to credit different types of things in their own distinct 
categories so people know in what ways their work is good or not good. I 
do agree though that too many would make voting tedious.
2c,
Charles

Post a reply to this message

On Sun, 09 Mar 2008 17:42:35 -0400, David Buck wrote:

> Chambers wrote:
>> David Buck wrote:
>>> Your thoughts would be appreciated.
>> 
>> Might be too late to mention this, but I've always felt that a 5-point
>> score system is easier for users :)
> 
> A 5 point system is just as easy to implement as a 10 point system.  I'm
> open to this suggestion if people think it's the best way.

From an implementation standpoint, the difference isn't significant - but 
from the standpoint of a user, what's the difference between a "7" and an 
"8", for example, on a scale of 9 or 10 points?

This is a question I've been trying to answer for course evaluations in 
my "real life" job for a couple of years now.  From the standpoint of 
what type of scale to use, an odd-number scale works best (I find), 
because there is actually a middle number in the scale (assuming an 
integer scale).  I use a 9-point scale (because that's what's been in 
place more than anything) - 5 becomes a "neutral" score in that case.

Just something to think about.

Jim

Post a reply to this message

Charles C wrote:
> How pretty art is, is different from how clever/witty it is -- just as 
> how effective a technique is in accomplishing an effect is also 
> completely different from how technically involved it was to produce. 
> How do you compare a well chosen/applied simple technique against an 
> overly ambitious (read Rube-Goldberg-inspired) technique which showed 
> better promise than first-shot results?   I think it would be nice to be 
> able to credit different types of things in their own distinct 
> categories so people know in what ways their work is good or not good. I 
> do agree though that too many would make voting tedious.
> 2c,
> Charles

Here's a novel thought: do we really need more than one category?  Why 
not just have one score, and let that be that?  After all, is it really 
worthwhile to say, "Well, *this* picture was extremely difficult for the 
author to make because of the method he used, so the fact that he pulled 
it off makes up for it's being a lousy image"?

---

And another thought about ratings scales: what if every user's score 
were "normalized", so that the average the middle two quartiles are 
scaled from 2.5-7.5 (so the mean of the two quartiles would be a 5), and 
the two outlying quartiles are scaled from 0-2.5 and 7.5-10?

This could make up for individual voter biases (like the fact that some 
people, when using a 10 point scale, only hand out between 9 and 10 
point scores).

-- 
...Ben Chambers
www.pacificwebguy.com

Post a reply to this message

Chambers wrote:
> Here's a novel thought: do we really need more than one category?  Why 
> not just have one score, and let that be that?  After all, is it really 
> worthwhile to say, "Well, *this* picture was extremely difficult for the 
> author to make because of the method he used, so the fact that he pulled 
> it off makes up for it's being a lousy image"?

We'd get a 1st 2nd and 3rd favorite pictures and no honorable mentions. 
  It'd be a single dimensional contest which doesn't sound quite as 
interesting to me anyway.  I suppose we don't all value the same things, 
and a single category would give the most freedom to each voter to vote 
for whichever reasons they think are valid.  Keeping more categories 
could catch more participant interest though.
Charles

Post a reply to this message

Chambers wrote:

> Here's a novel thought: do we really need more than one category?  Why 
> not just have one score, and let that be that?  After all, is it really 
> worthwhile to say, "Well, *this* picture was extremely difficult for the 
> author to make because of the method he used, so the fact that he pulled 
> it off makes up for it's being a lousy image"?

Actually come to think of it, I personally think the optional comments 
people can leave are probably one of the most worthwhile parts of the 
contest.
Charles

Post a reply to this message

On Sun, 09 Mar 2008 21:50:46 -0700, Chambers <ben### [at] pacificwebguycom> wrote:

>Here's a novel thought: do we really need more than one category?  Why 
>not just have one score, and let that be that?  After all, is it really 
>worthwhile to say, "Well, *this* picture was extremely difficult for the 
>author to make because of the method he used, so the fact that he pulled 
>it off makes up for it's being a lousy image"?

This gets my vote, if you will pardon the pun :-)
Who can say what "artistic" is? It is different things to different people.
Likewise "Technical ability" is it really important in "art" ?

Regards
	Stephen

Post a reply to this message

Chambers <ben### [at] pacificwebguycom> wrote:
> Might be too late to mention this, but I've always felt that a 5-point 
> score system is easier for users :)

  IMO it's too coarse.

-- 
                                                          - Warp

Post a reply to this message

Et tu, David Buck, contra me ?

> So I'd like to know if you have any suggestions on how to do this.
> 1) How do we accept new voters?
>
Login/pass.

> 2) Should submitters be allowed to vote in the round they submit to?
>
Absolutly.

> I intend to allow voters to rate each image in 4 categories
> 	- overall
Rather a 'WOW facror', mentioned elsewhere...

> 4) Should we allow partial voting?
>
Partial voting shouldn't be allowed... (IMHO :)

And for non-submitters, some kind of rate system ( like many
'Rate my something' sites has) could be implemented...

> David Buck

Slawek (and his $0.03
-- 
  ________ BOFH excuse 143:
_/ __/ __/ had to use hammer to free stuck disk drive heads.
 \__ \__ \_______________________________________________________________
 /___/___/ S�awomir Szczyrba                          steev/AT/hot\dot\pl

Post a reply to this message

> 1) How do we accept new voters?
> 2) Should submitters be allowed to vote in the round they submit to?

I propose that submitters are the only people allowed to vote. They earn that
privilege by contributing to the competition. I propose this is cumulative in
the sense that any entrant who has successfully been  awarded a prize/place is
then a voter in perpetuity.

> I intend to allow voters to rate each image in 4 categories

I personally often found these categories difficult to distinguish.

I would tend to err on the side of simplicity and have one category ... "best
PovRay rendering of theme ...".
I haven't gone back and checked the old system but wonder if the technical and
artistic didn't generally correlate quite highly. Adearance to theme ... well
aren't submitters going to rank theme relevant entries automatically?

> 4) Should we allow partial voting?
>  - by this I mean rating some images but not all

One needs to rank all of them, in order to make this a time efficient process
then this reenforces a single category.

Or, one ranks the top N images (potentially in each category). This did seem to
work well for the scc competitions.

Personally, I think source code should be mandatory. Especially if there is a
technical category.

Post a reply to this message

>>>>> "Warp" == Warp  <war### [at] tagpovrayorg> writes:

Warp> Chambers <ben### [at] pacificwebguycom> wrote:
>> Might be too late to mention this, but I've always felt that a 5-point 
>> score system is easier for users :)

Warp>   IMO it's too coarse.

You could let each person select a "lo" and "hi" point of their own
scale, and then normalize the values to a floating point from 0.0 to 1.0.

If I want a 3-point scale, I'd get it.  If you wanted a 0-100 scale, you'd get
your own.

Best of all worlds.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<mer### [at] stonehengecom> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

Post a reply to this message

David Buck <dav### [at] simberoncom> wrote:
> I intend to allow voters to rate each image in 4 categories
>         - overall
>         - technical merit
>         - artistic merit
>         - concept and interpretation of theme

  Does anyone have any good idea about how to avoid cross-contamination
of categories?

  The only solution I can think of is to have only one category.

-- 
                                                          - Warp

Post a reply to this message

Warp <war### [at] tagpovrayorg> wrote:
> David Buck <dav### [at] simberoncom> wrote:
> > I intend to allow voters to rate each image in 4 categories
> >         - overall
> >         - technical merit
> >         - artistic merit
> >         - concept and interpretation of theme
>
>   Does anyone have any good idea about how to avoid cross-contamination
> of categories?

I think I'd rather have 2+ categories than just one. Even if we just had two:

 - quality of image (by whatever criteria one favours)
 - concept/interpretation.

My reasoning is that we need some way to penalise entrants who deviate from the
theme - I remember seeing many images that were technically and artistically
very good, but had almost no relevance to the theme. In that case, what's to
stop someone just submitting an old image that they happen to be proud of? The
idea of the compo (for me) is to create a new image inspired by the theme. A
concept/interpretation category could also incorporate the artistic score.

I always found it very easy to rate images by concept and interpretation. I used
to score common variations on an obvious interpretation lower than more original
or laterally-thought-out images.

However, I definitely agree that 'overall' is a bit hand-wavey, and that four
may be too many categories.

Just my tuppence
Bill

Post a reply to this message

Another two cents worth on judging procedures
and categories:

Keeping the system simple and keeping the number
of categories and options down to a reasonable
minimum is a worthy goal worth making a few
sacrifices for, but let's not go too far in that
direction and unduly constrain the judges' ability
to give constructive feedback.

Perhaps it would be a good idea to supply short
definitions for each category which are worded
specifically enough so as to eliminate
“I'm-not-sure-what-that means” syndrome, but not
so specific as to impose an exterior set of
aesthetic priorities.  I offer the following as a
“for instance”, rather that a finished proposal:

Category 1: TECHNICAL MERIT:
The degree to which the image shows mastery of
the image creation process.

Category 2: ARTISTIC MERIT
The degree to which the image evokes in you
thoughts or feelings you deem significant or
praiseworthy.

Category 3: INTERPRETATION OF TOPIC:
The degree to which you feel the image expresses
the topic idea in a particularly original or
profound way.

While it is true enough that there will be some
overlap between any such list of categories, there
are significant enough differences between each of
three categories above to merit their separate existences.

On the subject of the number of gradations for each
category, I would propose that rather than choosing an
arbitrary number, we provide a descriptive term for each
value in a given category as well as a number and see where
that takes us.   Again, here's a “for instance”:

TECHNICAL MERIT:
[0] = Lacks both knowledge and effort
[1] = Shows BEGINNER level skills
[2] = Shows good BASIC skills
[3] = Shows ADVANCED level skills
[4] = Shows EXPERT level skills
[5] = Shows EXTRAORDINARY skills(WOW!)

ARTISTIC MERIT:
[0] = Offensively trite and/or cliche
[1] = VERY LITTLE aesthetic impact
[2] = MODERATE aesthetic impact
[3] = SIGNIFICANT aesthetic impact
[4] = HIGH aesthetic impact
[5] = PROFOUND aesthetic impact (WOW!)

INTERPRETATION OF TOPIC:
[0] = Off-topic / inappropriate
[1] = Rather OBVIOUS idea
[2] = NOT A BAD idea
[3] = VERY GOOD idea
[4] = EXCELLENT idea
[5] = BRILLIANT idea (WOW!)

Under a system like the one above, I would suggest
that the general instructions indicate that both
[0]s and [5]s indicate unusual circumstances and
should be used extremely sparingly.  Also, there
should be a very strong suggestion, if not an
absolute rule, that a [0] or a [5] should be
accompanied by an explanation in the comments.

One goal that I would suggest for the new contest
is that it actively promote the participation and
development of newcomers to the field.  I hope that it
won't be one of those contests that routinely offers
kudos to a hand full of established elite artists
and a thumbing of the nose to everyone else.  Toward
this end, I would like to offer two specific proposals:

First,the competition should employ a comment system that
emphasizes CONSTRUCTIVE criticism and feedback for ALL
participants.  Feedback, encouragement and help are
among the prime motivators that will bring new people
into ray tracing and help to develop the elite artists
of the future.

Second, the terminology for rating
should offer non-derogatory options to describe the
work of those who are still in the rather early stages
of learning.  In the system above, for example, a newbie
might actually be encouraged by a Technical Merit rating
of [2] (“The judges thought I had 'good basic skills'
when I really think of myself as just a beginner.”).

Well, thanks for taking the time to read my rather
long winded comments. The gist of of what I have to
say can be condensed down to this: I hope that the
new IRTC will be a fun and enjoyable experience that
offers both challenge and a sense of community to new
and expert ray traces alike.

Thanks, again, to all who are making the effort to
bring the IRTC back.

Best Regards,
Mike C.

Post a reply to this message

As might have been menitioned elsewhere, the problem with one category is
thematic deviation (yes I've done it).  But what about the fact that someone
has a really great idea for their image, and just didn't develop that idea.
(Again, I've done this).  If there was one category, there would be no way to
honor that person for a great idea, but a lousy image.

In my thoughts, I think the categories should be the following, because these
are distinct, but still tie together with each other:
 -Concept/originality
 -Technical Merit
 -Impressiveness

Concept/originality is basically how the topic is developed in the image.  It
shows how well through the image was thought. e.g. In one of the Stills rounds,
called "Alien Worlds," there were some beautiful Star Trek matte paintings
rendered, but that's not really original as an idea.  So, Plus for the
artistic, minus for the orignality

Technical Merit is basically how the user approached the image s/he made.  Did
s/he use some impressive code to produce a high degree of photo-realism, or did
he simple put a few spheres next to each other, and just image-map them.
Basically, it's how much work the user put in to achieve his goal.

And Impressiveness is basically just how well the image pulls off as a good
image.  The more it makes someone go "Wow" and the more it looks good, might be
the scale.  That doesn't just mean that photo-realism is impressiveness.  I
mean, I get more "wowed" from Pixar's Toy Story, than I do from Harry Potter's
Magic Effects.  It just is how good the image looks.

By all means, don't quote me on saying that's how it should be.  I personally
think this would work well, with a bit of tweaking here or there, but that
system seems good to me.

And to David, if you use this system, or something very similar, by all means,
you may quote me in the voting guidelines, or rules, etc.

Any other ideas on those categories?

Post a reply to this message

One of the most offensive scores I got on an IRTC submission was 10 10 10. And
on that round, when I voted on others, I made sure my vote expressed exactly
what the artist were doing well (artistic, technical, theme).  This basically
means that everyone else gave me 10 10 10 for the average to be 10 10 10. No
one put any thought into the vote.

For a vote to be meaningful, it has to COST the voter something.  In the old
voting system, one could also give 20 20 20 to one's favorite, and 1 1 1 to all
the ones one didn't like.  Others might give out a strict normalized
distribution around ten, and reserve 20's for once a lifetime beauties.
Average up these two voters' votes, and you're throwing away the careful
voter's analysis.

For the Short Code Competition,  I came up with a voting scheme where you cite
your favorite six entries, and then gave 6 points to the favorite, 5 to their
2ndmost, and so on.  In this way, everyone's opinion is weighted the same.
There are more complicated schemes where folks have an IRTC Dollar to give
away, and they can divide it up any way they want-- all to one or a penny to
100 entries-- I say this is inherently fair, as every voter gets a dollar.

Analogy: My local newspaper once spent a lot of time and effort into a reader
vote on which comics to keep.   They let everyone vote yes or no on keeping
every comic. (Note there's no COST to vote here!)  The comic "Doonesbury" came
in about last, but they never dropped it.  My hunch is they knew that 55%
didn't like it but maybe 10% were passionate about it.  Because the poll didn't
measure user passion, the paper ended up ignoring the results.

As far as WHO votes,  I'd be more inclined to have real people vote than male
nerds who use freeware graphics programs. (DISCLAIMER: I'm one of the latter.)
Maybe it's impractical to line up a set of artistic, educated, and/or
interesting people, but that would be the coolest.



David Buck <dav### [at] simberoncom> wrote:


> I'd like to find a nice way to prevent or reduce voting anomalies in the
> new IRTC web site.  I don't want people to be able to vote multiple
> times in one contest and skew the votes.
>
> So I'd like to know if you have any suggestions on how to do this.
>
> 1) How do we accept new voters?
>  - automatically with e-mail confirmation?
>  - only with admin approval?
>  - other ideas?
>
> 2) Should submitters be allowed to vote in the round they submit to?
>
> I intend to allow voters to rate each image in 4 categories
>  - overall
>  - technical merit
>  - artistic merit
>  - concept and interpretation of theme
>
> Each rating is on a scale from 0 to 10.
>
> 3) How should the scores be combined
>
> 4) Should we allow partial voting?
>  - by this I mean rating some images but not all
>  - I have no assurance that they even looked at all of them
>  - if you rated the first 5 pictures but none of the others,
>   should any of your ratings be considered?
>
> Your thoughts would be appreciated.
>
> Thanks
> David Buck

Post a reply to this message

On Mon, 10 Mar 2008 06:39:00 -0700, Randal L. Schwartz wrote:

> You could let each person select a "lo" and "hi" point of their own
> scale, and then normalize the values to a floating point from 0.0 to
> 1.0.

Ewww.

Rather than do that, why not just let people pick from a 1-9 scale, and 
if they want it more coarse, they can ignore every other digit?

Letting people pick their own scale will create problems - someone thinks 
they picked a scale of 3, but the app saw it as a scale of 9, so instead 
of giving the top rating, they give one of 33%?

No way....

Jim

Post a reply to this message

On Mon, 10 Mar 2008 08:49:22 -0500, Warp wrote:

>   Does anyone have any good idea about how to avoid cross-contamination
> of categories?

If you get enough votes, cross-contamination of categories will factor 
itself out in the averages.

Jim

Post a reply to this message

"David Buck" <dav### [at] simberoncom> a écrit dans le message de news: 
47d2ce11$1@news.povray.org...
> I intend to allow voters to rate each image in 4 categories
> - overall
> - technical merit
> - artistic merit
> - concept and interpretation of theme

It should be noted that with the IRTC, the 3 notes (techical, artistic, 
concept) were always extremely correlated. In the 2006 rounds the 
correlation coefficients between these notes were in the 0.92-0.96 range: 
it's basically the same note (correlation coefficients run between 0 and 1 
where 0 = no correlation). One can run a regression and predict one note 
with another with a small margin of error. I did the same calculations in 
the past, with the same results. In a nutshell, folks who are smart enough 
to understand a concept and come up with a good idea also have decent 
artistic and technical skills. Now that's really surprising ;)

There are occasional outliers of course - for instance images with a 
moderately good artistic note and so-so concept - but on the whole having 
several voting categories is more intellectually satisfying than 
statistically meaningful.

G.

-- 
*****************************
http://www.oyonale.com
*****************************
- Graphic experiments
- POV-Ray, Cinema 4D and Poser computer images
- Posters


>
> Each rating is on a scale from 0 to 10.
>
> 3) How should the scores be combined
>
> 4) Should we allow partial voting?
> - by this I mean rating some images but not all
> - I have no assurance that they even looked at all of them
> - if you rated the first 5 pictures but none of the others,
> should any of your ratings be considered?
>
> Your thoughts would be appreciated.
>
> Thanks
> David Buck

Post a reply to this message

"gregjohn" <pte### [at] yahoocom> wrote:
> One of the most offensive scores I got on an IRTC submission was 10 10 10.

>...This basically means that ...No one put any thought into the vote.
>
> For a vote to be meaningful, it has to COST the voter something. ...
>
> For the Short Code Competition,  I came up with a voting scheme where you cite
> your favorite six entries...
.....
> As far as WHO votes,  I'd be more inclined to have real people vote than male
> nerds who use freeware graphics programs. (DISCLAIMER: I'm one of the latter.)
> Maybe it's impractical to line up a set of artistic, educated, and/or
> interesting people, but that would be the coolest.

Although I'm very sympathetic to the sentiments that you express here and
recognize that they are thoughtful and well-intentioned, I nonetheless feel
compelled to mention a few problems that I see with the particulars:

Systems under which a judge votes only for a few favorites tend to leave
everyone except a few elites out in the cold with no constructive feedback.
This bodes poorly for developing the community and building up the skills of
the up and coming.  I guess this really comes down to a question of goals and
priorities.  *ARE* we interested in building community and encouraging
newcomers or do we want a contest the purpose of which is: “Reward the best;
forget the rest”?

Also, sadly, one of the things that must be considered is how a system might be
abused by those who do not share your good intentions and desire for fairness.
If I'm more interested in influencing the outcome in favor of my best friend or
my own little clique than in judging the entries fairly, I can do much more
damage by concentrating all 100 of my points than I can by being just one voter
with a skewed motive.  (I freely acknowledge that NO system solves the ulterior
motive problem entirely, but voting only for favorites exacerbates it. )

I would suggest, alternatively, that the “cost” of voting should be time and
effort.  Not only should voters be required to vote on ALL entries, but also to
leave at least one coherent comment on each entry AND to explain why any image
was given either the highest or lowest possible rating.  To those who would
suggest that this is asking too much of judges, I would reply that it's nothing
compared to the effort of preparing a serious entry.

As for the WHO of voting, (and this is merely personal feeling with no real
objective logic behind it) I share your desire to get feedback from both inside
and outside the world of those who produce ray trace images, but I would require
some SERIOUS intellectual or artistic credentials for “guest” judges.  I really
don't give a rodent's rectum how people who rush home from their marketing jobs
to watch “Survivor” and “American Idol” (sorry for being culture specific here)
would rate my work.  Also, a significant percentage of the judging will have to
be done by “insiders” if the feedback is to contain any really useful advice
about how one can improve.

Respectfully,
Mike C.

Post a reply to this message

I like the idea of forcing people to leave comments.  I usually try to do this,
so as to give others a chance to see why I felt the way I did.  I remember that
once,I even forgot something in the comment, and left an anonymous comment just
to add something.

Back then, I was an amateur Ray-tracer, and was soo hoping for comments on
improving my images.  In both stills competitions I entered, I recieved one
comment.  However, Animations were kinder, in giving me about 5 comments.

I think some people don't realize that there are amateurs out there who enter
partly for fun, and partly for just trying to get better at something they
really love.  When I got no comments, I thought it was because no one cared for
such low images.  Admittingly, one did get last place, but I never got to figure
out why it did.  I deduced most of it over a years worth of developing skills.
So, yeah, I agree that comments should be highly recommended, if not forced.
Someone who takes the time to write out comments for every entry, has
definitely thought each image through, and also hopefully graded each likewise.

As for other things, points need to be something the whole world understands.  I
definitely think there needs to be a guide as to what each rating means(they
used to do that for the 1, 10, and 20 marks).  Points mean a lot about how well
an image really is, and they themself can tell a contestant what there image
really is.  So, that's why I'm supporting categories.

David, have you come to some decisions on what to do, or are you still waiting
out?

Post a reply to this message

Nimish Ajmani wrote:
> David, have you come to some decisions on what to do, or are you still waiting
> out?

I've been following all the discussions here and putting together what I 
feel are the best recommendations.  I'll put together a summary of those 
later this evening and post them here.

Thanks to everyone for their feedback - it's very helpful and has 
changed my approach to the design of the site.

David Buck

Post a reply to this message

> Thanks to everyone for their feedback - it's very helpful and has
> changed my approach to the design of the site.

I think I speak for everyone when I say not a problem.

  --Nimish

Post a reply to this message

Gilles Tran a écrit :
> 
> There are occasional outliers of course - for instance images with a 
> moderately good artistic note and so-so concept - but on the whole having 
> several voting categories is more intellectually satisfying than 
> statistically meaningful.

I wholeheartedly agree.

As a numerous-times IRTC participant, I've been mostly annoyed
by these useless 3 categories, and, personnally, I had the habit
to put the 3 same notes each time.

Advice : get a general ranking with a single note, and use a small jury
to attribute artistic, technical and concept merits.  Makes more sense.

Fabien.

Post a reply to this message

Paul Bourke wrote:
>> 1) How do we accept new voters?
>> 2) Should submitters be allowed to vote in the round they submit to?
> 
> I propose that submitters are the only people allowed to vote. They earn that
> privilege by contributing to the competition. I propose this is cumulative in
> the sense that any entrant who has successfully been  awarded a prize/place is
> then a voter in perpetuity.

What if they become annoyances after this point?  I don't think it will 
be a major problem, but continuing privileges should come only with 
continuing contribution.

Regards,
John

Post a reply to this message

Gilles Tran wrote:
> "David Buck" <dav### [at] simberoncom> a écrit dans le message de news: 
> 47d2ce11$1@news.povray.org...
>> I intend to allow voters to rate each image in 4 categories
>> - overall
>> - technical merit
>> - artistic merit
>> - concept and interpretation of theme
> 
> It should be noted that with the IRTC, the 3 notes (techical, artistic, 
> concept) were always extremely correlated.

It may also be that judges let their overall opinions color the 
individual category scores.  If they like the concept, judges tend to 
ignore technical flaws, for instance.

Regards,
John

Post a reply to this message

"Gilles Tran" <gil### [at] agroparistechfr> a écrit dans le message 
de news: 47d56279@news.povray.org...
> There are occasional outliers of course - for instance images with a 
> moderately good artistic note and so-so concept - but on the whole having 
> several voting categories is more intellectually satisfying than 
> statistically meaningful.

Here are the graphs with the regression results.
http://www.oyonale.com/images/divers/image001.gif Artistic vs technical
http://www.oyonale.com/images/divers/image002.gif Artistic vs conceptual
http://www.oyonale.com/images/divers/image003.gif Technical vs conceptual

The strongest fit is artistic vs technical. That's really the same notion.

G.


-- 
**********************
http://www.oyonale.com
**********************
- Graphic experiments
- POV-Ray, Cinema 4D and Poser computer art
- Posters

Post a reply to this message

John VanSickle <evi### [at] hotmailcom> wrote:
> It may also be that judges let their overall opinions color the
> individual category scores.  If they like the concept, judges tend to
> ignore technical flaws, for instance.

I bet it works that way at the other end of the spectrum too.  E.g. un-inspiring
images may get little attention paid to a clever concept etc.

I wonder, does anybody else have a tendency to open the description first before
bothering to look at the final product?  :-)

Charles

Post a reply to this message

"Charles C" <nomail@nomail> wrote:
> John VanSickle <evi### [at] hotmailcom> wrote:
> > It may also be that judges let their overall opinions color the
> > individual category scores.  If they like the concept, judges tend to
> > ignore technical flaws, for instance.
>
> I bet it works that way at the other end of the spectrum too.  E.g. un-inspiring
> images may get little attention paid to a clever concept etc.
>
> I wonder, does anybody else have a tendency to open the description first before
> bothering to look at the final product?  :-)
>
> Charles

What I usually do is first read the description.  Then I grade the image based
on its own merit.  I then adjust my given score based on what I read, and end
up with a final score.

My orignal score was based somewhat like that of the French system.  I never
gave more than an 18, and below 10 was for images that just didn't come of
good.  That's how I did it.

  --Nimish

Post a reply to this message

Gilles Tran <gitran_nospam_@wanadoo.fr> wrote:
> Here are the graphs with the regression results.
> http://www.oyonale.com/images/divers/image001.gif Artistic vs technical
> http://www.oyonale.com/images/divers/image002.gif Artistic vs conceptual
> http://www.oyonale.com/images/divers/image003.gif Technical vs conceptual

  I find the complete lack of dots on the upper left and lower right
corners rather telling.
  Either nobody ever made an image which was superb in one category and
completely sucked in another (which I really find hard to believe), or
cross-contamination of categories is just an inherent property of the
human mind and there's nothing that can be done about that.

-- 
                                                          - Warp

Post a reply to this message

Nimish Ajmani wrote:
> "Charles C" <nomail@nomail> wrote:
>> John VanSickle <evi### [at] hotmailcom> wrote:
>>> It may also be that judges let their overall opinions color the
>>> individual category scores.  If they like the concept, judges tend to
>>> ignore technical flaws, for instance.
>> I bet it works that way at the other end of the spectrum too.  E.g. un-inspiring
>> images may get little attention paid to a clever concept etc.
>>
>> I wonder, does anybody else have a tendency to open the description first before
>> bothering to look at the final product?  :-)
> 
> What I usually do is first read the description.  Then I grade the image based
> on its own merit.  I then adjust my given score based on what I read, and end
> up with a final score.

I generally view the entry first, and then read the description.  This 
is for the animations round (which is the side I enter).  If I need the 
description in order to understand the animation, then there is 
something wrong with the execution of the concept.

Regards,
John

Post a reply to this message

"Charles C" <nomail@nomail> schreef in bericht 
news:web.47d5d90b32e069a82f046a7d0@news.povray.org...
>
> I wonder, does anybody else have a tendency to open the description first 
> before
> bothering to look at the final product?  :-)
>
No. Always image first, description later.

Thomas

Post a reply to this message

"Thomas de Groot" <t.d### [at] internlDOTnet> wrote:
> "Charles C" <nomail@nomail> schreef in bericht
> news:web.47d5d90b32e069a82f046a7d0@news.povray.org...
> >
> > I wonder, does anybody else have a tendency to open the description first
> > before
> > bothering to look at the final product?  :-)
> >
> No. Always image first, description later.
>
> Thomas

Perhaps I should change my style of judging (granted I've only done it twice...)

  --Nimish

Post a reply to this message

"Nimish Ajmani" <nomail@nomail> schreef in bericht 
news:web.47dfcc5e32e069a88c12d7010@news.povray.org...
>
> Perhaps I should change my style of judging (granted I've only done it 
> twice...)
>

<grin>
I think it doesn't matter too much what technique or which procedure you 
follow, as long as you get a balanced judgment in the end. That said, 
however, personally I feel that the competition is about *images* or 
*animations*, so my very first move is precisely towards the *image* or the 
*animation*. That determines my first impression and, in the end, the 
overall level of judgment. Description and so on, come second.

Thomas

Post a reply to this message