I have written about this years ago already, but here's a quick
recapitulation of what I think would be good for the new scripting
language (and perhaps a few new ideas):

- It must be easily byte-compilable. Preferably to a byte code which may
  be easily JIT-compiled to native machine code in some future version.
  Even without JIT-compiling just the fact of having it in bytecode
  format can make parsing an order of magnitude faster.

- The bytecode will also allow saving scenes in bytecode format for
  faster parsing. (Basically the need to parse the scene is removed
  completely, except for some sanity checks of the bytecode.)

- Currently there's a distinction between parse-time scripting (the SDL)
  and rendertime scripting (user-defined functions, used eg. in isosurfaces).
  Rather obviously no such distinction should be made in the new language.
  Anything you write should be runnable at parse time and at rendertime
  (and also at the post-processing stage).

- Object properties, such as textures, should be fully specifiable using
  the scripting language itself (facilitated by the previous point). In
  other words, you should be able to write, for example, "shaders" with
  the scripting language. This "shader" is evaluated every time that
  certain property of the object (eg. its texture) is needed.
    This would allow eg. writing an AOI patter with the scripting language,
  completely removing the need to add such a pattern to the core renderer.

- Most core renderer features should be available to the scripting language.
  As an example, you should be able to launch a ray from a given point and
  retrieve the color it returns.
    This will allow, among many many things, writing eg. a "portal" shader,
  ie. a texture which shows the image as seen by a secondary "camera"
  somewhere else. I believe megapov has currently such an option. Rather
  than being a core feature, it would be easily implementable with scripting.

- Related to that, the scripting language should be so flexible that many
  current core features can be removed from the core renderer and put into
  libraries.
    Currently there are tons of features in the core renderer which are there
  for the sole reason that you can't implement them with the SDL, and there
  wouldn't be any other reason for them to them to be there (eg. efficiency).
    The core renderer should be as minimal as possible, and only contain
  things which have to be there because of efficiency (it would be way too
  slow to try to implement them in scripting). The smaller the core renderer
  is, the lesser bugs it will have.
    (Another advantage is that bugs in libraries are much easier to fix
  than bux in the core renderer. Also libraries can be modified by the
  user much more easily. Modifying the core renderer is way more laborious
  and error-prone.)

  Related to that last point, an interesting question is how to provide
standard data containers ? la STL. By the previous principle they should
be provided as libraries. This makes them more flexible, easier to modify,
and removes clutter from the core renderer.

  However, there's also a negative side: The STL in modern C++ compilers
is very stable, very well tested and almost certainly bug-free, while
creating equivalents with the new scripting language, no matter how
flexible it might be, is very laborious and will inevitable be ridden
with bugs for years to come. Providing a "wrapper" to the STL in the
scripting as a core feature could be way less work and less error-prone
than trying to re-implement them from scratch using the scripting language.
They will probably also be faster that way (because the container functions
will be compiled, rather than bytecode-interpreted).

  If the scripting language is designed smartly enough, it will actually
not make any difference whether the containers are implemented as libraries
written in the scripting language itself, or whether they are a core
feature (acting as STL wrappers). This allows changing the implementation
of any of them from one approach to the other without breaking any existing
scenes. This is a goal to aim for.

- The design principle of the new scripting language should have some kind
  of "unified object model" approach. With this I mean that core objects
  and user-created objects should work and be usable in the exact same way.
  That is, it doesn't matter whether a "sphere" object is a core feature
  or something which came from a library. In either case you would use that
  "sphere" object in the exact same way (as long as you know its interface,
  of course).
    For example, if you wanted to access (read, change) the texture of the
  object, it would be done with an identical syntax regardless of whether
  the object is a core feature or defined in a library.
    In object-oriented terms, they would have a common base class (even if
  the scripting language wouldn't really use literal inheritance) and you
  can handle any object in the same way.

    And by "object" I don't necessarily mean literally object primitives
  (spheres, boxes), but basically anything povray features (lights, cameras,
  textures, arrays...). In other words, regardless of what it is, it should
  be usable in the same way regardless of whether it's a core feature or
  defined in a library. Of course different types of objects will have
  different interfaces (in OO terms, they will have different base classes),
  and you have to know the "type" of object you have in order to use it
  (eg. handling a sphere is different from handling a texture, for instace,
  as they have different type of data), but the point is that the user
  doesn't have to know if it's a core feature or a library object in order
  to use it (he only has to know its interface).

- As for memory management: Garbage collection (as in the one in Java or C#)
  might be nice, but VERY hard to implement efficiently. If possible the
  scripting language should be designed in a way which makes creating a GC
  engine unnecessary.
    This might mean that the only way to use dynamic memory is through the
  dynamic data containers provided. However, that might be enough for
  almost everything. (After all, most people have done cool things with
  the current array feature, even though it's *really* rigid. The new
  data containers will be much more flexible.)

- The scripting language should also be designed so that it's easy to
  flexibly read, parse and write data files (as well as handle the data
  itself). This would allow writing things like 3DS or OBJ file format
  importers as libraries. (This is an example of something which should
  definitely be a library rather than a core feature.)

-- 
                                                          - Warp

Post a reply to this message

Chris B wrote:
> Backwards Compatibility
> There's a lot of time been invested in tutorials, interfaces, macros and
> objects.
> Much of the written information out on the Internet (including the years
> of newsgroup postings) doesn't specify which version it relates to, so
> if this material is rendered largely redundant it's probably going to be
> wise to totally change the product name. Otherwise there's a risk of
> years of confusion that may create difficult barriers to new users.

	I doubt a product name change is necessary. However, it should be made
clear all over the place (docs, Web site, etc) that the latest Pov will
break compatibility with anything before v4.

	I personally don't think there's a great need to maintain backward
compatibility as far as syntax/code goes. I think the hope is that the
new SDL will provide "nicer"/"better" ways to do stuff than the old SDL
- and so we wouldn't want people being hampered by tutorials from the
past - but force them to learn the new way of thinking.

	One possibility is to try to find lots of good tutorials, put them onto
the Wiki (if permission can be acquired), and before POV4 is released,
update the tutorials to show how to get each effect using POV4.

-- 
A man attempting to walk around the world DROWNED today...


                    /\  /\               /\  /
                   /  \/  \ u e e n     /  \/  a w a z
                       >>>>>>mue### [at] nawazorg<<<<<<
                                   anl

Post a reply to this message

Warp <war### [at] tagpovrayorg> wrote:
> - It must be easily byte-compilable. Preferably to a byte code which may
>   be easily JIT-compiled to native machine code in some future version.
>   Even without JIT-compiling just the fact of having it in bytecode
>   format can make parsing an order of magnitude faster.
>
> - The bytecode will also allow saving scenes in bytecode format for
>   faster parsing. (Basically the need to parse the scene is removed
>   completely, except for some sanity checks of the bytecode.)

I guess this boils down to about the same as what I had in mind about separating
what would basically be the tokenizer, and the actual parser. Except that what
you call bytecode, I'd call a serialized "syntax tree"

I don't think, however, that JIT-compiling would make any sense or be an option:

- A JIT-compiler integrated into POV-Ray itself is, quite obviously, no more
than a distant dream.

- Using an external JIT-compiler would require another piece of software (namely
the JIT-compiler) to run POV; I don't think this is acceptable.

- Integration of an external JIT-compiler would pose major architectural
problems, which I guess are better left unaddressed, given the number of other
ideas pending implementation and the desire of users to have them all ASAP.

> - Currently there's a distinction between parse-time scripting (the SDL)
>   and rendertime scripting (user-defined functions, used eg. in isosurfaces).
>   Rather obviously no such distinction should be made in the new language.
>   Anything you write should be runnable at parse time and at rendertime
>   (and also at the post-processing stage).

Yes and no. Obviously the render-time scripting should have exactly the same
syntax as the parse-time scripting, most notably access to the same integrated
identifiers. However, things like generating additional geometric objects, or
even just modifying existing ones, is a definite no-go for render-time code.

Or, to put it in other words: Render-time code should be restricted to read-only
ops. Temp variables would come in handy, but their lifetime and scope should be
limited to the current invocation of the function for a whole bunch of reasons
- the most important being that many POV internals rely on user functions to be
stateless. Throwing this overboard would open up a can of really nasty worms.

> - Object properties, such as textures, should be fully specifiable using
>   the scripting language itself (facilitated by the previous point). In
>   other words, you should be able to write, for example, "shaders" with
>   the scripting language. This "shader" is evaluated every time that
>   certain property of the object (eg. its texture) is needed.
....
> - Most core renderer features should be available to the scripting language.
>   As an example, you should be able to launch a ray from a given point and
>   retrieve the color it returns.

I think these ideas are not realistic in near future, as they require yet
another major architectural change of POV-Ray, and I guess we want a new SDL
earlier. However, it is a good point to keep in mind when designing that
language.

But note also that these shaders, being render-time code, would not be able to
use data caches or stuff alike, due to restriction on statelessness. I'd
advocate that shaders that require stateful operation be written as binary
plug-ins of sort.

> - Related to that, the scripting language should be so flexible that many
>   current core features can be removed from the core renderer and put into
>   libraries.

Given how complex the render engine currently is regarding even seemingly simple
tasks, such as shooting a ray and seeing what color you get, I must conclude
that (a) heading for that goal would definitely be of benefit for the internal
architecture of the render engine, but (b) I'd be prepared for that goal to be
never actually reached.

In addition, I'd advocate not to implement those functions in script code, but
instead have them remain binary modules, albeit separated from the main engine
code more strictly, e.g. as a kind of plug-in.

> - The design principle of the new scripting language should have some kind
>   of "unified object model" approach. With this I mean that core objects
>   and user-created objects should work and be usable in the exact same way.
>   That is, it doesn't matter whether a "sphere" object is a core feature
>   or something which came from a library. In either case you would use that
>   "sphere" object in the exact same way (as long as you know its interface,
>   of course).

This would presume that the SDL in fact provide an interface to define new
geometric entities. These would have to be render-time code.

I don't really think there is much use for such objects, and that instead giving
more flexibility to functions to be used in isosurfaces is the way to go.

>     For example, if you wanted to access (read, change) the texture of the
>   object, it would be done with an identical syntax regardless of whether
>   the object is a core feature or defined in a library.

Yes, that's an important consistency thing. It will probably also make the
parser more lightweight. At the moment, every property for every thing has its
own code snippet in the parser.

> - As for memory management: Garbage collection (as in the one in Java or C#)
>   might be nice, but VERY hard to implement efficiently. If possible the
>   scripting language should be designed in a way which makes creating a GC
>   engine unnecessary.
>     This might mean that the only way to use dynamic memory is through the
>   dynamic data containers provided. However, that might be enough for
>   almost everything. (After all, most people have done cool things with
>   the current array feature, even though it's *really* rigid. The new
>   data containers will be much more flexible.)

Yes, I guess this is the way to go. It makes for clear memory handling
responsibilities: The built-in containers do all that's needed; they'll also
care for either copying items, or keeping a reference counter to know when a
shared entity is no longer needed. That should do for most practical purposes
(after all, we're not writing complex business applications in the order of
magnitude of several man-years ;))

Post a reply to this message

"Chris B" <nom### [at] nomailcom> wrote:
> Backwards Compatibility
> There's a lot of time been invested in tutorials, interfaces, macros and
> objects.
> Much of the written information out on the Internet (including the years of
> newsgroup postings) doesn't specify which version it relates to, so if this
> material is rendered largely redundant it's probably going to be wise to
> totally change the product name. Otherwise there's a risk of years of
> confusion that may create difficult barriers to new users.

There is some point to this; however, that's why it would be "POV-Ray 4.0", and
not "POV-Ray 3.8".

I don't know anything about POV-Ray 2.0, but I bet tutorials on it would be far
from usable nowadays, too.

Backward compatibility could also be maintained by designing the new language
such that it will be compatible to POV 3.x SDL up to (and including) the
"#version" statement, at which point a legacy parser may kick in.

> Inventing New Languages
> There are a lot of mature languages about in common usage with comprehensive
> and well designed structures and support channels. It may be better to adopt
> one existing language, providing API/DOM access to the C++ core with
> official support for that language. Enthusiast for other languages could
> then add support for other languages as trends change.

This would (a) generate a major problem with POV being unusable stand-alone, (b)
require a major effort to implement a whole mature language (and all its
features, because that's what people would expect), or (c) open up the can of
worms called GPL by integrating an existing language interpreter into the POV
package.

> Maybe a solution would be to create an integration layer with native support
> for an enhanced version of the established POV-Ray SDL and, at the same time
> expose the API/DOM. As a particular programming language becomes a popular
> choice for the POV-Ray community in years to come it could become the
> defacto standard and could be added to the short list of officially
> supported languages.

In my opinion the problem with the current SDL is not only that it is showing
its age regarding the features it offers, but also the syntax: It is far from
consistent with itself (just look at #fopen vs. #fwrite: Why does one have the
parentheses, while the other has not?), and needs an overhaul even in that
respect.

I think a redesigned language would actually be of benefit for new users, as it
will be much easier to learn.

If the basic syntax is sufficiently different, but the scene stuff parameter
names are not changed, I guess there will not be much confusion: It will just
be an (obviously) different syntax to access the same properties.

Those tutorials not dealing with scene stuff properties but instead elaborating
on how to work around the shortcomings of the current SDL will be rendered
obsolete anyway by a next generation SDL, or we'd better not talk about a new
SDL at all.

Re established general-purpose language, note also that Perl is currently
undergoing a major syntax overhaul, so even going for established languages
does not save you from incompatibilities between language versions.

Plus, I'd expect API/DOM access from an established language to usually be a
good deal more cumbersome. So instead of writing something like this:

for(i = 0; i < n; i ++) {
  sphere (<0,i*2,0>, 1) {
    texture {
      ...
    }
  }
}

you would for instance need to write something like this:

for(int i = 0; i < n; i ++) {
  Sphere mySphere = new sphere(new Vector3d(0,i*2,0), 1);
  mySphere.texture  = new Texture(
    ...
  );
  pov.addToScene(mySphere);
}

So, as a scene designer, which would you prefer?

POV scene description is a highly specialized application, and therefore IMO
advocates a special language. Even if it's only for the native support of 3D
vectors and colors: They're just too commonplace to use general-purpose
languages' typically comparatively cumbersome ways of expressing custom-type
"literals".

Instead, the SDL can fairly well go without various other common features of
all-purpose languages, like event handling, exception handling, inheritance,
sophisticated regular expressions, dynamic memory handling (provided the basic
data containers are good enough), reflection (except maybe some rudimentary
stuff), template classes, interface declarations, and a good deal more, that
really make those languages complicated to implement.

To summarize: No, I don't think it would be prudent to continue extending the
existing SDL after POV 3.9999.

Post a reply to this message

clipka <nomail@nomail> wrote:
> I guess this boils down to about the same as what I had in mind about separating
> what would basically be the tokenizer, and the actual parser. Except that what
> you call bytecode, I'd call a serialized "syntax tree"

  I don't think it's necessary to keep any kind of "syntax tree" data in
any form once the scene file has been parsed and compiled into bytecode.
The bytecode can be very straightforward asm-like series of opcodes and
their parameters. The advantage of this is that it's very fast to interpret
(which can be important at rendertime), and even JIT-compile.

> I don't think, however, that JIT-compiling would make any sense or be an option:

  I don't see why not. Many scripting languages have that.

> > - Currently there's a distinction between parse-time scripting (the SDL)
> >   and rendertime scripting (user-defined functions, used eg. in isosurfaces).
> >   Rather obviously no such distinction should be made in the new language.
> >   Anything you write should be runnable at parse time and at rendertime
> >   (and also at the post-processing stage).

> Yes and no. Obviously the render-time scripting should have exactly the same
> syntax as the parse-time scripting, most notably access to the same integrated
> identifiers. However, things like generating additional geometric objects, or
> even just modifying existing ones, is a definite no-go for render-time code.

  Not "same syntax". The one and only scripting language, down to the bytecode
compiler and bytecode interpreter.

  There may be additional limitations for the rendertime code, but
that doesn't mean it has to be separate from the compile-time code.

  It makes no sense to create two scripting languages with two compilers
and two interpreters. That will only be more work and very confusing for
the users (even more than the current SDL / user-defined function
distinction).

> > - Object properties, such as textures, should be fully specifiable using
> >   the scripting language itself (facilitated by the previous point). In
> >   other words, you should be able to write, for example, "shaders" with
> >   the scripting language. This "shader" is evaluated every time that
> >   certain property of the object (eg. its texture) is needed.
> ....
> > - Most core renderer features should be available to the scripting language.
> >   As an example, you should be able to launch a ray from a given point and
> >   retrieve the color it returns.

> I think these ideas are not realistic in near future, as they require yet
> another major architectural change of POV-Ray, and I guess we want a new SDL
> earlier.

  POV-Ray 4 was supposed to be a complete rewrite anyways. If it is, then
it should be done properly.

> In addition, I'd advocate not to implement those functions in script code, but
> instead have them remain binary modules, albeit separated from the main engine
> code more strictly, e.g. as a kind of plug-in.

  That sounds to me like it will only make the whole system more complicated,
not simpler.

  A simple core engine (only things which require speed should be there),
a simple script interpreter. Everything else as libraries written in the
scripting language. Let the complexities be inside the libraries, not in
the core engine.

> This would presume that the SDL in fact provide an interface to define new
> geometric entities. These would have to be render-time code.

> I don't really think there is much use for such objects, and that instead giving
> more flexibility to functions to be used in isosurfaces is the way to go.

  I consider those sentences contradictory.

  "There's no use for user-defined objects. Instead, we should enhance
the properties of usef-defined objects (isosurfaces)."

  I'd say no: Don't clutter the already cluttered features with even more
clutter. Instead, make a clean simple implementation which can be used for
the same thing, but with more flexibility.

-- 
                                                          - Warp

Post a reply to this message

Warp <war### [at] tagpovrayorg> wrote:
> > I don't think, however, that JIT-compiling would make any sense or be an option:
>
>   I don't see why not. Many scripting languages have that.

Damn it, Jim, POV is a raytracer, not a general-purpose IDE - and our human
resources are limited.

>   POV-Ray 4 was supposed to be a complete rewrite anyways. If it is, then
> it should be done properly.

If the intention is to do a *complete* rewrite, then why waste time on a 3.7,
which already is a major architectural overhaul?

>   A simple core engine (only things which require speed should be there),
> a simple script interpreter. Everything else as libraries written in the
> scripting language. Let the complexities be inside the libraries, not in
> the core engine.

No, Sir. We're talking about an application that is performance-hungry anyway,
so there should be a way to implement certain things in binary code, yet
separate from the core engine to keep that simple. Otherwise, we will sooner or
later end up with stuff that *does* need speed, and gets hacked into the core
just because there's no clearly defined way to interface binary stuff to it
properly.

Rather do it the other way round: Provide an interface for binary add-ons, then
implement the script language as such an add-on.

Scripting most parts of the software is ok when performance is not paramount,
like e.g. a web browser: You want it to be fast, but more importantly you want
it to be safe, comfortable etc., and you probably can't tell the difference
whether a page takes 0.5 or 0.6 seconds to load. You can tell the difference,
however, whether an animation takes 5 or 6 days to render.

>
> > This would presume that the SDL in fact provide an interface to define new
> > geometric entities. These would have to be render-time code.
>
> > I don't really think there is much use for such objects, and that instead giving
> > more flexibility to functions to be used in isosurfaces is the way to go.
>
>   I consider those sentences contradictory.
>
>   "There's no use for user-defined objects. Instead, we should enhance
> the properties of usef-defined objects (isosurfaces)."

Isosurfaces provide a simple, single-function interface to define objects.
Internally in POV, however, the interface is much different, and consists of
multiple methods: One to compute ray-object intersections, one to perform an
inside test, one to compute the surface normal for a given point, and so forth.
So when I hear "user-defined objects", I think of exposing *that* interface.

To me, isosurfaces are just that: Isosurfaces. A hard-coded shape that just
happens to be very flexibly parameterizable.

Post a reply to this message

"clipka" <nomail@nomail> wrote:

> * For Loops
>
> Loops like #local i = 0; #while (i < n) ... #local i = i + 1; #end are
> commonplace, and they work, but they suck. I want something like a #for i = 0
> to n-1 ... #end.
>

I hope I'm not weighing in with some 'simpleton' comments here (I'm definitely
NO computer scientist), but the way I approach discussions of a proposed new
SDL language is strictly from the standpoint of 'comfortable usability.' As
basically a non-programmer (except with the SDL), I would want to see a
language that made 'simple sense' as to its syntax. For example: The #while
loop is easily understood, simply from the fact of its name. Using #for instead
seems a bit 'obtuse' as to its meaning (except among programmers, of course!)
But that's the point: If the new language is to have 'non-programmer'
adherents, then its syntax should be 'obvious' as to what it does, and easily
understood. (Of course, there are plenty of current SDL terms that probably
don't fall into that category, yet we've all learned to use them.) I guess my
basic concern is not to alienate new users (or even old ones) with new syntax
that seems strange, or rather, non-obvious as to its meaning.

Of course, I could be the 'odd-man-out'; perhaps the majority of POV-Ray users
*do* actually come from some kind of programming background, where 'odd' syntax
terms are of no consequence.

Ken W.

Post a reply to this message

Kenneth <kdw### [at] earthlinknet> wrote:
> The #while loop is easily understood, simply from the fact of its name.

  I beg to differ. I once wrote a long page dedicated solely to explaining
how the #while loop does and doesn't work, because misunderstandings are
so common:

http://tag.povray.org/povQandT/whileloops.html

  For many non-programmers it's not at all obvious how they work, which
is the reason why mistakes are common, and often a source of obscure errors
for which questions get posted to the newsgroups. Especially nested loops
tend to cause confusion. For a programmer and for people who have used the
while loops it's simple, but for someone who has never used them it might
not be.

  A simple for-loop removes most if not all the confusion.

  And it's not like the for-loop would *replace* while-loops.

-- 
                                                          - Warp

Post a reply to this message

Warp <war### [at] tagpovrayorg> wrote:
>   A simple for-loop removes most if not all the confusion.

I'm not sure. The opinions of people like Kenneth are quite valuable here,
because they are still at a place that those of us who write code fluently can
only dimly remember. I honestly can't remember if I had more trouble with for
loops than while loops when I was learning to program. From where I'm standing,
it could easily be argued that for loops are the more challenging.

>   And it's not like the for-loop would *replace* while-loops.

Well, true. If we have both, then it's not an issue :-)


As an aside, I think there *has* to be a legacy parser (as I believe clipka
suggested). There is just too much useful stuff that will break, and users who
will either stick to pre-4.0 versions or stop using POV completely. Also,
having a simpler, more 'approachable' SDL will encourage users in, as it does
now. I see no problem in the old-style option being more restrictive; this
will, in turn, encourage the more confident user to move on to the new. Of
course, we will need plenty of code and tutorials posting here in the early
days to help everyone out!

(Don't get me wrong, I'm all in favour of a new SDL - indeed, I'm salivating at
the possibilities!)

Bill

Post a reply to this message

Warp <war### [at] tagpovrayorg> wrote:
> Kenneth <kdw### [at] earthlinknet> wrote:
> > The #while loop is easily understood, simply from the fact of its name.
>
>   I beg to differ. I once wrote a long page dedicated solely to explaining
> how the #while loop does and doesn't work, because misunderstandings are
> so common:
>
> http://tag.povray.org/povQandT/whileloops.html

My main concern is actually with 'semantics' (if that's the right word.) I.e.,
the term #while seems to be quite logical and understandable in its meaning:
"while (this is true) do this stuff" etc. Same thing with #if. But--using #for
as an example again--I'm not so sure I understand what "for" signifies, in an
English syntax sense. (Actually, I do seem to recall 'for-next' being a part of
Fortran IV that I learned long ago in college. I think. Which has been my only
experience with 'real' computer programming, aside from Basic or one of its
iterations on an old Commodore computer. But #for doesn't immediately create a
mental picture of what it does.) So my own desire would be to have SDL syntax
that is as easy (or even easier) to 'quickly grasp' than the current version's.

KW

Post a reply to this message