[Followups set to povray.general]

1 February 2007
---------------

The following is a joint announcement from Persistence of Vision Raytracer
Pty. Ltd. ('POV'), Lutz Kretzschmar (Lutz) and Markus Lutz (Markus).

Persistence of Vision Raytracer Pty. Ltd. is pleased to advise that it has
acquired the rights to the 3d modeller 'Moray' from Lutz and Markus. Moray is
a Windows program specifically designed to create 3d models and scenes to be
used with the POV-Ray raytracer, and was originally sold as trialware.

POV will, in due course, re-issue Moray under a new name and new license. The
license has not been decided on with finality, but will be OSI-approved. [POV
is planning to release it under the GPLv3 but this decision is subject of
course to review of the final version of that license.]

The POV-Team is continuing development on version 3.7, which is the last
planned major release of the current POV-Ray codebase prior to version 4.0.

Version 4.0 will be released under an OSI-approved license. POV has not
decided on which license will be chosen at this stage. Those portions of the
codebase that are not already re-releasable under an OSI license will, prior
to the release of 4.0, be re-written so as to permit them to be released
under one. Some of the 'new to version 3.7' codebase is already able to be
released under both the old and newly-chosen license.

The current version 3.7 development codebase will be made publicly available
at around the same time as the Moray codebase. This will allow the public to
assist, if they wish, in completing version 3.7 of POV-Ray and will also
assist in the transition to 4.0. POV hopes to be able to announce the final
4.0 license choice at this time.

There is no schedule for the release of 4.0. It will primarily depend on the
amount of public assistance that is received in re-writing non-OSI-licensable
code.

Lutz and Markus have their own announcement available on their news page,
which can be found at http://www.stmuc.com/moray/menews.html.

The text of this announcement itself can be found on the POV-Ray site at
http://www.povray.org/news/moray-announcement.php.

Post a reply to this message

Nice!

Chris Cason schrieb:
> [Followups set to povray.general]
> 
> 1 February 2007
> ---------------
> 
> The following is a joint announcement from Persistence of Vision Raytracer
> Pty. Ltd. ('POV'), Lutz Kretzschmar (Lutz) and Markus Lutz (Markus).
> 
> Persistence of Vision Raytracer Pty. Ltd. is pleased to advise that it has
> acquired the rights to the 3d modeller 'Moray' from Lutz and Markus. Moray is
> a Windows program specifically designed to create 3d models and scenes to be
> used with the POV-Ray raytracer, and was originally sold as trialware.
> 
> POV will, in due course, re-issue Moray under a new name and new license. The
> license has not been decided on with finality, but will be OSI-approved. [POV
> is planning to release it under the GPLv3 but this decision is subject of
> course to review of the final version of that license.]
> 
> The POV-Team is continuing development on version 3.7, which is the last
> planned major release of the current POV-Ray codebase prior to version 4.0.
> 
> Version 4.0 will be released under an OSI-approved license. POV has not
> decided on which license will be chosen at this stage. Those portions of the
> codebase that are not already re-releasable under an OSI license will, prior
> to the release of 4.0, be re-written so as to permit them to be released
> under one. Some of the 'new to version 3.7' codebase is already able to be
> released under both the old and newly-chosen license.
> 
> The current version 3.7 development codebase will be made publicly available
> at around the same time as the Moray codebase. This will allow the public to
> assist, if they wish, in completing version 3.7 of POV-Ray and will also
> assist in the transition to 4.0. POV hopes to be able to announce the final
> 4.0 license choice at this time.
> 
> There is no schedule for the release of 4.0. It will primarily depend on the
> amount of public assistance that is received in re-writing non-OSI-licensable
> code.
> 
> Lutz and Markus have their own announcement available on their news page,
> which can be found at http://www.stmuc.com/moray/menews.html.
> 
> The text of this announcement itself can be found on the POV-Ray site at
> http://www.povray.org/news/moray-announcement.php.
> 
>

Post a reply to this message

"Chris Cason" <del### [at] deletethistoopovrayorg> a écrit dans 
le message de news: 45c1c908$1@news.povray.org...
> [Followups set to povray.general]
>
> 1 February 2007
> ---------------
>
That's a great new indeed!
Thanks Lutz and Markus for your work.
It must be somewhat heartbreaking to let go a child but let's hope he wil 
grow healthy :-)

Marc

Post a reply to this message

"Marc" <jac### [at] wanadoofr> wrote:
> "Chris Cason" <del### [at] deletethistoopovrayorg> a écrit dans
> le message de news: 45c1c908$1@news.povray.org...
> > [Followups set to povray.general]
> >
> > 1 February 2007
> > ---------------
> >
> That's a great new indeed!
> Thanks Lutz and Markus for your work.
> It must be somewhat heartbreaking to let go a child but let's hope he wil
> grow healthy :-)
>
> Marc

I too am pleased that Moray will continue to live and be developed. I join
Marc in thanking Lutz and Markus for their work. Moray has allowed me to
use Pov-Ray despite being challenged in the code-writing department.
Selfishly I wonder if there will still be a repository for Moray modals and
textures?

PS The fish were our pleasure :-)



Stephen

Post a reply to this message

This is great news indeed!

May I repeat others in deeply thanking Lutz and Markus for their excellent 
work over the years? Without Moray, I would not be doing what I am doing 
now, and I sincerely hope and believe that will be also the case with the 
future avatars of this great program.

Thomas

Post a reply to this message

Wow.

I (try) to be a coder but have always loved Moray for it's beauty and
elegance.

Thanks so much to Lutz and gang for their major contribution to the Pov
community.

And no more occasional guilt.

Peter SeaWA

Post a reply to this message

"Stephen" <mcavoys_AT_aolDOT.com> a écrit dans le message de news: 
web.45c1d8014a1d3fd8f1cb1e660@news.povray.org...

> PS The fish were our pleasure :-)
I vote for a fishbowl with these words engraved  "So long and thanks 
for...."

Marc

Post a reply to this message

Sweet!  Thanks to Lutz and Marcus as well as the POV-Ray team for making
this possible!

Jim

Post a reply to this message

On Thu, 1 Feb 2007 20:14:08 +0100, "Marc" <jac### [at] wanadoofr>
wrote:

>
>"Stephen" <mcavoys_AT_aolDOT.com> a écrit dans le message de news: 
>web.45c1d8014a1d3fd8f1cb1e660@news.povray.org...
>
>> PS The fish were our pleasure :-)
>I vote for a fishbowl with these words engraved  "So long and thanks 
>for...."
>
>Marc 
>
Do you want to start a tribute?

Regards
	Stephen

Post a reply to this message

"Stephen" <mcavoysATaolDOTcom@> a écrit dans le message de news: 
3jj4s2hsj7gmt20tegm21mqnk77c85eeo3@4ax.com...
> On Thu, 1 Feb 2007 20:14:08 +0100, "Marc" <jac### [at] wanadoofr>
> wrote:
>
>>
>>"Stephen" <mcavoys_AT_aolDOT.com> a écrit dans le message de news:
>>web.45c1d8014a1d3fd8f1cb1e660@news.povray.org...
>>
>>> PS The fish were our pleasure :-)
>>I vote for a fishbowl with these words engraved  "So long and thanks
>>for...."
>>
>>Marc
>>
> Do you want to start a tribute?

Why not... though this one is rather simple if you don't want to model 
Arthur Dent's home :-)


Marc

Post a reply to this message

woot!

-- 
Tim Cook
http://home.bellsouth.net/p/PWP-empyrean

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GFA dpu- s: a?-- C++(++++) U P? L E--- W++(+++)>$
N++ o? K- w(+) O? M-(--) V? PS+(+++) PE(--) Y(--)
PGP-(--) t* 5++>+++++ X+ R* tv+ b++(+++) DI
D++(---) G(++) e*>++ h+ !r--- !y--
------END GEEK CODE BLOCK------

Post a reply to this message

Waw...such happy news are only few in humain life :)
Taha

Post a reply to this message

Hi Stephen, you recently wrote in povray.general:

> I too am pleased that Moray will continue to live and be developed. I join
> Marc in thanking Lutz and Markus for their work. Moray has allowed me to
> use Pov-Ray despite being challenged in the code-writing department.
Thank you all for your kind words. As I wrote on our website, we are
very happy that development will (hopefully) pick up and continue.

> Selfishly I wonder if there will still be a repository for Moray modals and
> textures?
While we haven't talked about the website with the POV-Team, I won't
be able to administer the current repository. Besides, I haven't
received a single submission in the last three years.... maybe moving
it to povray.org will reinvigorate it and I'm sure someone with good
web/database skills can create a good interface.
 
> PS The fish were our pleasure :-)
:-)

- Lutz
  email : lut### [at] stmuccom
  Web   : http://www.stmuc.com/moray

Post a reply to this message

Hi Marc, you recently wrote in povray.general:

> It must be somewhat heartbreaking to let go a child but let's hope he wil 
> grow healthy :-)
Yes, it is with a crying and a laughing eye (sorry, bad literal
translation of a german saying) that we did this. It is hard to let
our "baby" go, but we do hope that it'll grow up faster.....

- Lutz
  email : lut### [at] stmuccom
  Web   : http://www.stmuc.com/moray

Post a reply to this message

"Stephen" <mcavoysATaolDOTcom@> schreef in bericht 
news:3jj4s2hsj7gmt20tegm21mqnk77c85eeo3@4ax.com...
>>
> Do you want to start a tribute?
>

Our earlier tribute just stopped along the way...:-(

Thomas

Post a reply to this message

"Thomas de Groot" <t.d### [at] internlDOTnet> wrote:
> "Stephen" <mcavoysATaolDOTcom@> schreef in bericht
> news:3jj4s2hsj7gmt20tegm21mqnk77c85eeo3@4ax.com...
> >>
> > Do you want to start a tribute?
> >
>
> Our earlier tribute just stopped along the way...:-(
>
> Thomas

That’s because we were too ambitious, I think. It ended up too recourse
hungry. Why not let Marc do the basic fishbowl scene and anyone who wants
to, can add a signature?
I’m good at volunteering other people to do the work :-)


Stephen

Post a reply to this message

Stephen schrieb:
> "Thomas de Groot" <t.d### [at] internlDOTnet> wrote:
>> "Stephen" <mcavoysATaolDOTcom@> schreef in bericht
>> news:3jj4s2hsj7gmt20tegm21mqnk77c85eeo3@4ax.com...
>>> Do you want to start a tribute?
>>>
>> Our earlier tribute just stopped along the way...:-(
>>
>> Thomas
> 
> That’s because we were too ambitious, I think. It ended up too recourse
> hungry. Why not let Marc do the basic fishbowl scene and anyone who wants
> to, can add a signature?
> I’m good at volunteering other people to do the work :-)
> 
> 
> Stephen
> 
> 
> 
  Sounds like a good Idea to me, but maybe Povray.general is not the 
right group for it?

Greets,
Arne

p.S. Thanks, Lutz, for letting your baby go. I have a registered version 
of Moray since 1993 and even though i almost solely used it for hobby 
reasons I've never felt the money wasted.

I hope to see a bright future for moray and hope you are still in the 
boat with it.

Post a reply to this message

Lutz Kretzschmar wrote:
> While we haven't talked about the website with the POV-Team, I won't
> be able to administer the current repository. Besides, I haven't
> received a single submission in the last three years.... maybe moving
> it to povray.org will reinvigorate it and I'm sure someone with good
> web/database skills can create a good interface.

Actually this is happening now - see the long threads on pov-ray includes.
The site will be flexible enough to accommodate moray models.

-- Chris

Post a reply to this message

> Persistence of Vision Raytracer Pty. Ltd. is pleased to advise that it has
> acquired the rights to the 3d modeller 'Moray' from Lutz and Markus......

Wow, that's great news. I'm really happy that Moray will not only survive 
but will undoubtably prosper and improve. Thanks Lutz and Markus for all the 
years you gave.

Steve Shelby
http://www.shelbyvision.com

Post a reply to this message

"Chris Cason" <del### [at] deletethistoopovrayorg> a écrit dans 
le message de news: 45c31221@news.povray.org...
> Actually this is happening now - see the long threads on pov-ray includes.
> The site will be flexible enough to accommodate moray models.
>
> -- Chris

pov-ray includes? is that a group that I am not aware of?
can't find anything like it in the group list :-(

Marc

Post a reply to this message

"Marc" <jac### [at] wanadoofr> wrote:
> "Chris Cason" <del### [at] deletethistoopovrayorg> a écrit dans
> le message de news: 45c31221@news.povray.org...
> > Actually this is happening now - see the long threads on pov-ray includes.
> > The site will be flexible enough to accommodate moray models.
> >
> > -- Chris
>
> pov-ray includes? is that a group that I am not aware of?
> can't find anything like it in the group list :-(
>
> Marc

http://news.povray.org/povray.general/thread/%3C456b29c7$1@news.povray.org%3E/?ttop=237611&toff=50

Stephen

Post a reply to this message

"Stephen" <mcavoys_AT_aolDOT.com> a écrit dans le message de news: 
web.45c334164a1d3fd8f1cb1e660@news.povray.org...
>
>
http://news.povray.org/povray.general/thread/%3C456b29c7$1@news.povray.org%3E/?ttop=237611&toff=50
>
> Stephen
>
Thank you :-)

Marc (Flushes ashamed and feels dumb)

Post a reply to this message

In article <45c1c908$1@news.povray.org>, deletethis-newsadmin2
@deletethistoo.povray.org says...
> [Followups set to povray.general]
> 
> 1 February 2007
> ---------------
> 
> The following is a joint announcement from Persistence of Vision Raytrace
r
> Pty. Ltd. ('POV'), Lutz Kretzschmar (Lutz) and Markus Lutz (Markus).
> 
> Persistence of Vision Raytracer Pty. Ltd. is pleased to advise that it ha
s
> acquired the rights to the 3d modeller 'Moray' from Lutz and Markus.
> 
Wow...!!

Does this mean it might finally include, at minimum, a setting to make 
the handedness work the same between them? lol But seriously, this is 
unbelievable news.

-- 
void main () {
  If Schrödingers_cat is alive  
    call functional_code()
  else
    call crash_windows();
}

<A HREF='http://www.daz3d.com/index.php?refid=16130551'>Get 3D Models,
 
3D Content, and 3D Software at DAZ3D!</A>

Post a reply to this message

Wow!
Moray was my first approach to 3d - and to POV-Ray.
In spite of the little time I'v used it, I can declare that it is a very
nice application.
Many thanks to Lutz and Markus for the work (and the fish, of course).
;-)
Paolo

> "Chris Cason" wrote:
> [Followups set to povray.general]
>
> 1 February 2007
> ---------------
>
> The following is a joint announcement from Persistence of Vision Raytracer
> Pty. Ltd. ('POV'), Lutz Kretzschmar (Lutz) and Markus Lutz (Markus).
>
> Persistence of Vision Raytracer Pty. Ltd. is pleased to advise that it has
> acquired the rights to the 3d modeller 'Moray' from Lutz and Markus. Moray
is
> a Windows program specifically designed to create 3d models and scenes to
be
> used with the POV-Ray raytracer, and was originally sold as trialware.
[...]

Post a reply to this message

Chris Cason wrote:
> The current version 3.7 development codebase will be made publicly available
> at around the same time as the Moray codebase. This will allow the public to
> assist, if they wish, in completing version 3.7 of POV-Ray and will also
> assist in the transition to 4.0.

Did noone else notice this?  So we'll get access to the 3.7 codebase 
before it's finished?

Cool :)

...Chambers

Post a reply to this message

On Sun, 11 Feb 2007 23:01:39 -0800, Ben Chambers <ben### [at] pacificwebguycom> wrote:

>Chris Cason wrote:
>> The current version 3.7 development codebase will be made publicly available
>> at around the same time as the Moray codebase. This will allow the public to
>> assist, if they wish, in completing version 3.7 of POV-Ray and will also
>> assist in the transition to 4.0.
>
>Did noone else notice this?  So we'll get access to the 3.7 codebase 
>before it's finished?

Hmm, yeah, I read right over that point.  I'll be interested to look at the code,
however, I won't
likely have much to add.  I have studied the 3.6 code quite a bit and, from what I've
read,  it
seems that the 3.7 code is an extensive rewrite towards better management and
extensibility.  Should
be interesting...



Kyle

Post a reply to this message

Kyle <hob### [at] gatenet> wrote:
> I have studied the 3.6 code quite a bit and, from what I've read,  it
> seems that the 3.7 code is an extensive rewrite towards better management
> and extensibility.

  That's indeed true. While 3.7 is not a complete rewrite from scratch,
it's still a vast improvement on the 3.6 code. The latter uses tons and
tons of ugly hacks such as global variables and other solutions which are
thread-unsafe, most of which are needlessly so.

  Making code thread-safe automatically makes code better because
thread-safeness forces a higher degree of modularity. :)

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:
> Kyle <hob### [at] gatenet> wrote:
> 
>>I have studied the 3.6 code quite a bit and, from what I've read,  it
>>seems that the 3.7 code is an extensive rewrite towards better management
>>and extensibility.
> 
> 
>   That's indeed true. While 3.7 is not a complete rewrite from scratch,
> it's still a vast improvement on the 3.6 code. The latter uses tons and
> tons of ugly hacks such as global variables and other solutions which are
> thread-unsafe, most of which are needlessly so.
> 
>   Making code thread-safe automatically makes code better because
> thread-safeness forces a higher degree of modularity. :)

How far along is the conversion to C++ code?  I've been converting my 
modeler from C to C++, and I keep wincing at the evil code that I first 
wrote.

I wonder how many times the POV-Ray coders look at something and say 
"Good lord, did *I* write that?"

Regards,
John

Post a reply to this message

John VanSickle wrote:
> How far along is the conversion to C++ code?  I've been converting my
> modeler from C to C++, and I keep wincing at the evil code that I first
> wrote.

POV-Ray code has been C++ since 3.5 and required a C++ compiler since 3.6.

> I wonder how many times the POV-Ray coders look at something and say
> "Good lord, did *I* write that?"

Not the "I", but "anybody". Especially the lighting code was rather
revealing when it came to the "many cooks" problem. In same cases analysis
showed that perfectly logical looking code into which a lot of time went
maintaining it was either unreachable or conditions were always true. While
in many cases a compiler could probably have figured them out, they just
added to the complexity of the code. In other cases there were really
inefficient linear searches that were not needed at all.

Also, in particular the media code is known to not handle its interval
sorting efficiently, but lack of time did prevent us from rolling out the
replacement code because it had various funny bugs (that was why early media
in 3.7 did not work right). So what was done was simply adding the old
inefficient but flawlessly working code back. In fact, many such instances
in many places are the reason for the various strange bugs that did appear
in early betas. And they do take long to fix because usually the code does
work on the test scenes originally used to check the changes.

	Thorsten

Post a reply to this message

Thorsten Froehlich <tho### [at] trfde> wrote:
> POV-Ray code has been C++ since 3.5 and required a C++ compiler since 3.6.

  You know, "requires a C++ compiler" and "is C++" (and not C) are two
different things. ;)

  Those gigantic switch-case blocks are the most typical C-style code
in POV-Ray. (Sure, technically speaking switch-cases are C++ too, but
in this case those gigantic switch-cases are actually C code emulating
dynamic binding, something which could be done more directly and much
more easily using C++-specific code.)

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:
>   You know, "requires a C++ compiler" and "is C++" (and not C) are two
> different things. ;)
> 
>   Those gigantic switch-case blocks are the most typical C-style code
> in POV-Ray. 

But those are in the parser and not used for dynamic binding!?!

	Thorsten

Post a reply to this message

Thorsten Froehlich <tho### [at] trfde> wrote:
> But those are in the parser and not used for dynamic binding!?!

  They are not doing dynamic binding because they are written in C.
However, the functionality they are performing is something quite
typical of what dynamic binding is for. In typical OO code you don't
write such gigantic switch-case blocks but instead you inherit from
a base class which has the proper virtual functions to do the same
task.

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:
> Thorsten Froehlich <tho### [at] trfde> wrote:
>> But those are in the parser and not used for dynamic binding!?!
> 
>   They are not doing dynamic binding because they are written in C.
> However, the functionality they are performing is something quite
> typical of what dynamic binding is for. In typical OO code you don't
> write such gigantic switch-case blocks but instead you inherit from
> a base class which has the proper virtual functions to do the same
> task.

I think we are talking about two different things here. The switch-case
statements in the parser do neither emulate nor replace or in any other way
simulate anything even remotely like dynamic binding: Remember, the POV-Ray
scanner and parser are a standard recursive decent implementation, and
interpretation of a language requires staged conditional code execution.
There is no place for dynamic binding in a parser for those tasks, it is
just a completely different technique which has no application here.

	Thorsten

Post a reply to this message

Thorsten Froehlich wrote:
> Warp wrote:
>> Thorsten Froehlich <tho### [at] trfde> wrote:
>>> But those are in the parser and not used for dynamic binding!?!
>>   They are not doing dynamic binding because they are written in C.
>> However, the functionality they are performing is something quite
>> typical of what dynamic binding is for. In typical OO code you don't
>> write such gigantic switch-case blocks but instead you inherit from
>> a base class which has the proper virtual functions to do the same
>> task.
> 
> I think we are talking about two different things here.

To elaborate some more:

I do guess that your view is of the SDL as just that, a static description
(in the layer above preprocessing via macros and so on) of objects that is
read from a file and replicated in memory. While this perspective my appear
correct at first, second and third sight, it does not actually represent the
nature of a scene description.

Pressing a SDL (I did this at work for VRML 97 a few years ago, so I cannot
share the code that would illustrate it) in a completely get/set driven
manner, with object descriptions created by binding the get/set methods as
parser rules, may be appealing. Unfortunately, implementing even with the
most advanced template structures to support it, is not all that easy. Apart
from the approximately two abstraction layers in form of intermediate
template functions (which cost nothing, but *are* visible when debugging),
to define an object almost always involves some terminal rules to be
executed that depend on other information. In essence, you gain
interdependencies of data on previously parsed data, or (more difficult)
data that is still to be parsed or may be entirely optional.

Dealing with these cases is certainly possible, but you end up with more
than just get/set rules. Instead, you end up with rules as object creation
to set default values, rules for setting a parsed data, rules for
post-processing results after parsing everything else, and rules for setting
data after that. All those do share a lot of dependencies. Factoring each
into separate methods is possible. Factoring each special handling case into
methods is possible as well. However, suddenly what would have been 10 case
statements and some leading and trailing code has turned into 10 methods,
each calling two or three other methods that contain the shared code.

Each such method contains only a handful of lines of code. And conventional
wisdom holds that small methods are easy to maintain, but if you have a
swarm of 10 methods, depending on about 20 shared methods, all working on
the same object, you suddenly have decomposed sequential conditional code
into a complex tree of method calls. The code will be easy to read for sure,
*but* it will no longer be as easily understood.

Does this mean the get/set pattern does not work for an SDL? No, absolutely
not. It just means that a case-switch pattern will still be very useful: The
parser needs to be abstracted such that all possible values are sequential
starting at zero, and then you get a very efficient switch-case statement.
Even better, you also retain easy to maintain code that others can
understand quickly. Call it a tradeoff between maintainability and elegance
if you want to, it certainly is, but it also is a rather pragmatic approach
and has no performance drawbacks whatsoever. It may not be the most elegant,
but it will certainly be fast and easy to maintain. In the end, that is what
counts the most...

	Thorsten

Post a reply to this message

Thorsten Froehlich <tho### [at] trfde> wrote:
> However, suddenly what would have been 10 case
> statements and some leading and trailing code has turned into 10 methods,
> each calling two or three other methods that contain the shared code.

> Each such method contains only a handful of lines of code. And conventional
> wisdom holds that small methods are easy to maintain, but if you have a
> swarm of 10 methods, depending on about 20 shared methods, all working on
> the same object, you suddenly have decomposed sequential conditional code
> into a complex tree of method calls. The code will be easy to read for sure,
> *but* it will no longer be as easily understood.

  The question is not how many methods or case branches are needed, but
how the code is organized. The main point with OO modularization is that
everything related to one element of the input will be located in the
same place, and each such implementation will be similar in structure.
Once you know how such element parser objects are implemented and what
is their structure it's easy to understand any given one.

  I remember once wanting to add a new keyword to the POV-Ray 3.5 SDL.
I don't remember exact details anylonger, but I had to add at least two
separate 'case' lines in two separate files, as well as at least two
new elements to two arrays in to separate files. Plus the implementation
of that feature in its own separate file. (As I said, I don't remember
the details well anymore, so I may remember something wrongly, but it
felt needlessly laborious back then.)
  The information about that new token was scattered into many separate
files, none of which were specific to that feature. This is not very
modular.

  A well-implemented modular parser does not need this. Instead, you
create *one* new class in one new file, and then add the name of that
class into *one* existing array (or whatever). There's no need to make
modifications to existing code anywhere else than in this one place.
The class you created will contain all the necessary info for parsing
the new input element that it was created for. The existing parser file
where you add the name of this class will be completely abstract: It only
contains generic code for parsing, no token-specific code at all (as
pov3.6 and earlier do).

  The advantage of this is, of course, that all the code related to a
specific token/feature is contained in its own module instead of being
scattered among several gigantic files. If you need to eg. change this
somehow you don't need to hunt for all the files where it is mentioned
but you change only that one module.

  Will this cause some kind of speed penalty in parsing? Maybe, but I bet
it's nothing radical. Probably negligible compared to the most time-consuming
tasks done during parsing (such as allocating and initializing objects).

> It may not be the most elegant,
> but it will certainly be fast and easy to maintain. In the end, that is what
> counts the most...

  It's easy to maintain only if you know the parser and all the gigantic
switch-case-blocks by heart, and you can cite from memory all the files
which you need to modify if you want to eg. add a new token.

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:
>   A well-implemented modular parser does not need this. Instead, you
> create *one* new class in one new file, and then add the name of that
> class into *one* existing array (or whatever). 

That may or may not be desirable. Usually it actually is not, as it has a
serious negative effect on performance. Not to mention that is is not
actually possible if the parser is part of a public interface, where you of
course always have header and source file that need to be modified.

	Thorsten

Post a reply to this message

Warp wrote:
>> It may not be the most elegant,
>> but it will certainly be fast and easy to maintain. In the end, that is what
>> counts the most...
> 
>   It's easy to maintain only if you know the parser and all the gigantic
> switch-case-blocks by heart, and you can cite from memory all the files
> which you need to modify if you want to eg. add a new token.

I made no statement that a parser would or should be implemented exactly
like the current one, did I?

Even the current one does not have only "gigantic switch-case-blocks"
because there simply isn't that much to handle in 90% of cases. Here are
exceptions, but they are just that (i.e. texture parsing).

	Thorsten

Post a reply to this message

Thorsten Froehlich <tho### [at] trfde> wrote:
> That may or may not be desirable. Usually it actually is not, as it has a
> serious negative effect on performance.

  Serious negative performance? Like eg. parsing goes 1% slower?

  Switch-case blocks are not that much faster compared to dynamic binding.
Besides I clearly remember you saying in the past that the parsing of the
input file itself is not by far the slowest operation done at the parsing
stage (but allocating and initializing objects).

> Not to mention that is is not
> actually possible if the parser is part of a public interface, where you of
> course always have header and source file that need to be modified.

  Why would the public interface of the parser have any info on the
implementation details of the input file?

  A well-implemented parser *abstracts* away these implementation details.
The user of the parser is not interested in knowing whether a sphere is
created with the keyword "sphere", "Sphere" or "ball". The user of the
parser is only interested in getting the objects (in an abstract way).
Putting implementation details in the public interface is only going
to cause problems: Code around the entire application may start being
dependant on those implementation details, making it harder to modify.

  I see no reason why adding a new token could not be done by simply
adding the name of a new class in one .cpp file (besides implementing
that new class, of course).

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:
>   Switch-case blocks are not that much faster compared to dynamic binding.

They are not by code, but by design. As I outlined, you get an impenetrable
mess of dependent methods. That is slow. How slow? Well, it depends on the
object's details. Expect up to twice the time required for processing. Does
that translate to 20% or 50% or more slowdown? Implementing would tell, but
no need to implement something already known to be slower. As I said, I have
done it before. How fast was it to begin with - the VRML 97 I worked on was
faster than the equivalent generated by a standard C parser generator. For
some unfortunate VRML 97 object types abstraction on a method level would
cause needlessly complicated detached processing that ate most of the
parser's performance compared to the parser generator version.

>   A well-implemented parser *abstracts* away these implementation details.

That is pure theory and completely inappropriate for maintainable code.

>   I see no reason why adding a new token could not be done by simply
> adding the name of a new class in one .cpp file (besides implementing
> that new class, of course).

Well, from experience I know there are good reasons to make certain choices
for speed and maintainability. So I guess in the end it just comes down to
believing in my experience in building very fast and maintainable parsers,
including all the lessons learnt from problems caused by overdoing the
design abstraction. The best theory simply isn't always the best (in all
dimensions) implementation.

	Thorsten

Post a reply to this message

Thorsten Froehlich <tho### [at] trfde> wrote:
> >   A well-implemented parser *abstracts* away these implementation details.

> That is pure theory and completely inappropriate for maintainable code.

  Are you basically saying that abstract code is unmaintainable?

  Are you next going to say that modules make code hard to reuse? ;)

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:
> Thorsten Froehlich <tho### [at] trfde> wrote:
>>>   A well-implemented parser *abstracts* away these implementation details.
> 
>> That is pure theory and completely inappropriate for maintainable code.
> 
>   Are you basically saying that abstract code is unmaintainable?

In the context of a parser, yes. It is also why parser generators are
getting out of fashion for complex languages if said code has to be
maintained by large(r) groups of developers.

>   Are you next going to say that modules make code hard to reuse? ;)

Hmm ;-)

	Thorsten

Post a reply to this message

Thorsten Froehlich <tho### [at] trfde> wrote:
> >>>   A well-implemented parser *abstracts* away these implementation details.
> > 
> >> That is pure theory and completely inappropriate for maintainable code.
> > 
> >   Are you basically saying that abstract code is unmaintainable?

> In the context of a parser, yes. It is also why parser generators are
> getting out of fashion for complex languages if said code has to be
> maintained by large(r) groups of developers.

  I don't really understand why.

  A typical program which has a complex input file format consists
basically of two separate parts: The parser, which reads the input
file and interprets what is in there and creates appropriate data
from it, and the rest of the program which handles that data.

  There's a clear wall between these two things: The rest of the
program is only interested in the data. It's not interested how the
data was created. Usually (and in fact optimally) it's not even interested
in the fact that the data came from a file. The data generator is
*abstract* to the rest of the program. It's just a black box which
spits data out of it. It doesn't matter where that data is coming from.
(This kind of abstraction allows for this "black box" to generate data
from alternative sources which may be completely different from an
input file, or may be generated by different input file formats.)

  Usually, although perhaps not as importantly, the same applies to
the other direction: The parser just generates the data and is not
interested in what that data is used for (even though there may be
rational exceptions to this).

  As I said above, this kind of abstraction makes it easier to enhance
the parser. For example, support for a new type of input file format
could be added completely transparently. The rest of the code doesn't
even need to know this.
  However, if the interface between the parser and the rest of the
program "leaks" implementation details, it immediately creates a
dependency on a specific implementation, making it a lot more difficult
to actually change that implementation to something else. For example
adding support for a new file format may be impossible if the rest of
the program makes assumptions about the input being in the current
format.

  The parser itself may be divided into separate parts too: The parser's
main body may contain only a generic parser for files in a certain format
(for a quick&dirty example, let's say that the input file is in XML format):
It just parses any file in that structural format but doesn't specify (and
thus hardcode) what information that structure might contain.

  If this is well-made, it allows for modules to be "plugged in" the
parser. These modules may eg. handle blocks of certain type (for example
XML blocks named in a certain way). The main parser doesn't know what
that block is for, but it knows that if a block with that name appears,
it should be given to the module which tells it that it can handle it.

  Adding support for new blocks is easy: Just create a new module and
"plug it in" the parser. In practice this "plugging in" may be as simple
as adding the name of the module to some array or such.

  This makes the parser flexible. Instead of having hard-coded values,
new blocks can be added "dynamically" (even if it happens at compile time).

  The price to pay for this is some additional virtual function calls.
Given that a virtual function call is a double indirect function call
(ie. take a pointer, take another pointer at an offset from that, and
then call the function pointed by that) it's not all that slower than
a gigantic switch-case block (which might actually internally have a
very similar implementation).

  You say that this kind of system would require a large amount of
chained virtual function calls. I don't see why this would be so.
I can't think of any reason why this would be so.

  The parser can do some *preprocessing* using data specified by the
modules. For example, if each module says what is the name of the tag
it supports, the parser can build a state machine from all the names,
with a pointer to each corresponding module in the leaf nodes. This
shouldn't be any slower than having this kind of state machine hard-coded
in the parser.

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:
>> In the context of a parser, yes. It is also why parser generators are
>> getting out of fashion for complex languages if said code has to be
>> maintained by large(r) groups of developers.
> 
>   I don't really understand why.

I did not expect parser implementations to be so annoyingly resisting an
abstract implementation either, but I had to learn that they are. I agree it
is surprising.

>   There's a clear wall between these two things: The rest of the
> program is only interested in the data. It's not interested how the
> data was created. Usually (and in fact optimally) it's not even interested
> in the fact that the data came from a file. The data generator is
> *abstract* to the rest of the program. It's just a black box which
> spits data out of it. It doesn't matter where that data is coming from.

And this is exactly what will doom any implementation. If you follow through
with this for anything but a trivial language with trivial (read: no) rules
applied to statements, you set yourself up for getting unmaintainable code
that will also be much slower assuming you at least want to still have a
chance of comprehending it as author.

>   The price to pay for this is some additional virtual function calls.

It is not about speed, it is about a maintainable implementation. The cost
of abstraction is much higher as I previously outlined, as you get a maze of
method calls, which don't eat time for the call, but for the operations they
need to perform and the order in which they need to perform them. You just
add complexity at a point where you don't want it assuming you want the same
efficiency as a much simpler switch-based implementation.

Once you have sorted out how to implement rules, you end up having to
distribute the rule logic between several methods. Inevitably you have to
consider all possible conditions in every method, add checks, and even do
additional processing because you have no control over when you get to
process the data. In trivial cases this adds no overhead, but you will soon
find that many rules are not of the trivial kind. Instead, they depend on a
lot more context, which might even be a context only computed on the fly. So
you have to compute that every time.

For example, consider the POV-ray camera statement. There are distinct rules
 on what effect every keyword has, and how these keywords interact.
Intermediate values, like distances need to be computed, and applied. If you
 allow setting values via a strictly external interface, you soon face the
fact that you can never know if more data will be set. Yet, behavior of the
camera changes. So each time you set a value, you have to compute a valid
set of all camera parameters, even if the user will set them next. The set
method simply cannot know what will come next. However, the current parser
does know all this, and keeps states between applying rules.

Of course, for a camera there is little data and hence not much time is
lost. But just consider a mesh. Suddenly you are dealing with potentially
gigabytes of data. Any operation on it will be costly.

In short, you can get some abstraction, but there is a serious price to pay
for too much abstraction. in an academic product you may be willing to pay
it, together with the added cost of maintaining it by a tiny group of highly
qualified experts that know everything in detail (even though I have to
admit that even that is impossible for an abstract implementation, even for
the author himself/herself if there is just one). That is a price you do not
want to pay for code developed by more than one or two people. It will just
be a waste of time, for designers, programmers, debuggers, and users.

	Thorsten

Post a reply to this message

Warp wrote:
>   As I said above, this kind of abstraction makes it easier to enhance
> the parser. For example, support for a new type of input file format
> could be added completely transparently. The rest of the code doesn't
> even need to know this.

Abstraction on the syntactic analysis level says nothing about abstraction
on the lexical analysis level. As those two are distinct, passes, and hence
design choices of the parser do not influence the design of the lexical
analysis. I thought it was clear that when we talk about "parser" here, we
talk about it in the was it is used in POV-Ray, which means syntactic and
semantic analysis.

>   Adding support for new blocks is easy: Just create a new module and
> "plug it in" the parser.

This already works with the current parser. Only token names are kept in a
central file, because they are needed external to the parser, i.e. to
support syntax coloring. This could be handled differently though, so it
does not influence the level of "abstraction".

>   You say that this kind of system would require a large amount of
> chained virtual function calls. I don't see why this would be so.
> I can't think of any reason why this would be so.

I never said this at all. I said you need to divide each rule into methods,
which in and by itself is the problem. The overhead when they called is not
a problem I am talking about.

	Thorsten

Post a reply to this message

Thorsten Froehlich <tho### [at] trfde> wrote:
> >   There's a clear wall between these two things: The rest of the
> > program is only interested in the data. It's not interested how the
> > data was created. Usually (and in fact optimally) it's not even interested
> > in the fact that the data came from a file. The data generator is
> > *abstract* to the rest of the program. It's just a black box which
> > spits data out of it. It doesn't matter where that data is coming from.

> And this is exactly what will doom any implementation. If you follow through
> with this for anything but a trivial language with trivial (read: no) rules
> applied to statements, you set yourself up for getting unmaintainable code
> that will also be much slower assuming you at least want to still have a
> chance of comprehending it as author.

  If you do otherwise you doom yourself to hardcode the input format in
the program, making it painfully difficult to change the format, add support
for alternative formats and create alternative ways of generating the data.

  You still insist that abstracting the format from the data makes the
program incomprehensible, yet you don't give any reasoning for this.

  I have worked in a medium-size project which used a relatively complex
file format, and let me assure you that it was a blessing that I didn't
have to even know what the format looked like in order to implement the
programs in that project. (Of course I knew the format because I worked
so closely with it, and I had to create test input files and such, but
from the programming point of view I didn't have to care because the
parser was not my job.)

  Abstracting the input file parser, and additionally the command-line
parser, gave many benefits. One good example was when the project
managers wanted a "history" section in all the result files: This history
section contained information about every single step (in this case
every single program and its parameters) that data had been run through,
so it would be possible to repeat the whole process again if necessary.
Thanks to the abstraction level of both parsers we could implement this
history section completely transparently by only modifying the command-line
parser and the file format parser. Not one single line of the program
themselves had to be modified, yet we got a perfectly-working history
section in all result files. After the modification and recompiling of
these parser libraries it was enough to relink all the programs (over
a dozen of them) and by magic every single one of them started supporting
automatic reading and writing of this new history section (adding the
program's own data to the output).

  This was, in my opinion, rather illustrative. All programs started
supporting and creating a new block of data in the file format without
even one single modification in their source code. Only by modifying
two parsers.

  Another example: At some point another idea came up: It would be nice
if part of the data could be in a separate file and a data block could
simply say "read the data from this file". Also it should be possible
to tell any program to write the data of any given block to a given
file instead of the regular output file.
  Again: No modifications whatsoever were necessary in any of the
programs in order to achieve this. Again, just adding this feature
to the two parsers was enough. Now, completely transparently, all the
programs suddenly started supporting reading and writing of these
separate "include" files.

  The teamwork between the two parsers was just awesome. Most things
could be done between them behind the scenes, transparently, without the
programs themselves even knowing, and it worked like a charm.

  On the other hand, each time we wanted to add a new feature common
to all the programs such as these, if we would had to modify all the
over a dozen programs, I wouldn't call that "maintainable".

> Of course, for a camera there is little data and hence not much time is
> lost. But just consider a mesh. Suddenly you are dealing with potentially
> gigabytes of data. Any operation on it will be costly.

  We had hundreds of megabytes of data to read/write at times in our
project. Didn't seem to cause any problem.
  (The reading and writing was pretty fast, in fact. It was possible to
hacker-optimize it thanks to (or regardless of) the abstraction layer.)

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:
>   If you do otherwise you doom yourself to hardcode the input format in
> the program, making it painfully difficult to change the format, add support
> for alternative formats and create alternative ways of generating the data.

There is no sense in changing the format for POV-Ray. As I pointed out, it
is more than static data. I am not denying that with completely static data,
i.e. reading in a database format, you have little loss with abstraction.
However, as you are well aware, POV-Ray is way more than a static format. It
is much closer to a programming language, and it is a well demonstrated fact
that neither parser generators nor excessive abstraction are a gain for a
programming language. To the contrary, they seriously hurt maintainability.

>   Abstracting the input file parser, and additionally the command-line
> parser, gave many benefits.

I am not denying that there are potential benefits at first sight, what I am
saying is they are completely outweight by the drawbacks.

> yet you don't give any reasoning for this.

As I said, I have substantial experience with exactly what we are talking
about, parsers for SDLs. Naturally I cannot disclose what I did at my
employer, hence, as I also said before, you either take my word for it or
you don't. -- But please consider this: Given that POV-Ray 3.7 scales
extremely well (according to Intel's measurements) thanks to the work of
Chris Cason and me (I did the design, he profiled and tweaked it), I think I
have successfully demonstrated I know really well how to deliver
high-performance code. So in the context of POV-Ray, I would say just wait
and see.

	Thorsten

Post a reply to this message

Thorsten Froehlich <tho### [at] trfde> wrote:
> >   Abstracting the input file parser, and additionally the command-line
> > parser, gave many benefits.

> I am not denying that there are potential benefits at first sight, what I am
> saying is they are completely outweight by the drawbacks.

  That contradicts completely my experience. Our parser system was
flexible, enhanceable and fast.
  A more unabstract hardcoded solution would have been rigid and very
laborious to enhance, most probably giving no measurable speed benefits.

> Chris Cason and me (I did the design, he profiled and tweaked it), I think I
> have successfully demonstrated I know really well how to deliver
> high-performance code.

  You said elsewhere that the point was not the performance but the
manageability.

  "I used this solution and it proved to be very fast" is not a proof
of anything relevant. It only proves that that specific solution is fast.
It doesn't prove that other more flexible solutions are not equally fast.

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:
>   If you do otherwise you doom yourself to hardcode the input format in
> the program, making it painfully difficult to change the format, add support
> for alternative formats and create alternative ways of generating the data.

If the data structures resulting from the parse are very similar to the 
input language, and you're parsing from a "meaningless" stream of bytes 
into essentially a native representation of the parse tree, this can work.

If you're doing something like generating code as you parse, the 
intermediate level is not as useful.  I've worked with OO parsers and 
structured editors that generate data structures for interpreters. When 
you have 300 mutually-recursive methods each maybe one to three lines 
long, it gets very difficult to figure out what's going on when there's 
a problem.

Plus, the more generic the parser, the more difficult it is to generate 
clear error messages. For example, if your "union" refers to an object 
that isn't allowed to be inside a union, generating the appropriate 
error message can get very messy. You *can* make it work, mind, but it 
can be harder than if you're using a less-data-driven type of parser.

>   On the other hand, each time we wanted to add a new feature common
> to all the programs such as these, if we would had to modify all the
> over a dozen programs, I wouldn't call that "maintainable".

Questions like this always depend on which direction you see a program 
growing over time. It's what makes software architecturing fun.

>   Adding support for new blocks is easy: Just create a new module and
> "plug it in" the parser. In practice this "plugging in" may be as simple
> as adding the name of the module to some array or such.

Or not even that, if you're using a language with reflection. ;-) It's 
exactly the kind of extensibility that Warp is talking about that 
frustrates me so much trying to do big programs in C or C++, with no 
programmatic access to the code I wrote myself. Why should I have to 
tell the compiler *and* the application what the name of the class I 
just added is? :-)

-- 
   Darren New / San Diego, CA, USA (PST)
     New horror show, now only in Las Vegas:
        HANNIBAL LUXOR!

Post a reply to this message

Warp wrote:
>   "I used this solution and it proved to be very fast" is not a proof
> of anything relevant. It only proves that that specific solution is fast.
> It doesn't prove that other more flexible solutions are not equally fast.

If you think scalability is not a sign of quality, I can't help you.

	Thorsten

Post a reply to this message

Darren New wrote:
> You *can* make it work, mind, but it
> can be harder than if you're using a less-data-driven type of parser.

Exactly my point.

	Thorsten

Post a reply to this message