I would like to promote the idea of a PoV-Ray binary scene format.
I know the idea isn't orginal, I have even found something like that in the
VFAQ, but I hope to give here one or two new ideas...

Actually, I am not a big fan of binary file formats, because it is easier to
hack (and break :-) textual files within a text editor.
But binary files have the advantage of being compact and easy/fast to parse,
ie. no conversion of textual representation of numbers to their binary
values, read a 16bit or 32bit token instead of several characters for a
keyword to hash and lookup, etc.

Note: I have seen in the PoV-Ray VFAQ that binary format (it was for mesh
files) was out of question, partly because of portability issues, ie. not
the same binary representation of float numbers between platforms. Some
programs, like the Lua language, are very portable and still manage to write
and read binary data with float number (Lua's bytecode, for example). So it
is not an impossible task.

The base idea relies on the idea I have on the way PoV-Ray parses the scene
files. I can be wrong (I haven't checked the code) and this whole message
can be pointless :-)

I believe the parser reads the text file, tokenises it (replacing keywords
and identifiers by numbers) and then the engine executes this bytecode
(running loops, etc.), creating object data in memory. When the scene
creates an object, the engine allocates a structure representing the
attributes of the object, sets defaults values, and fill in the indicated
values. The final result is a in-memory binary representation of the scene.

Of course, this data isn't suitable for raw serialisation (output to a
file). Pointers have to be relative, etc., plus it is non-portable, as the
internal representation of objects is subject to change from one release to
the next.

We need then a robust file format, probably using chunks like [T]IFF or PNG
formats.
Older engines will ignore unkown data while newer ones will provide default
values where missing data.

So, we could ask the engine to output the scene data after parsing, or to
load one and render it.
Note that this would be a secondary file format, I don't suggest to drop the
current format!

What would be the use of such file?
Among numerous ideas:
- To give a model without revealing some tricks used to make it. So it still
can be rendered with different point of view, lighting, etc.
Of course, this format can be disassembled, but in "unrolled" format, ie.
with only the raw list of generated objects, without the coding wizardry
that could have been used to create it.
- To ease the creation of PoV-Ray to other 3D formats converters. They are
quite rare currently, because you either have to rip the parser from
PoV-Ray, or write your own, with probably incompleteness. With this unrolled
format, you avoid at least the task of interpreting the procedural language,
of checking syntax, etc.
- In the same spirit, to ease the creation of PoV-Ray pre-viewers, eg.
converting objects to OpenGL to quickly see a scene under various points of
view. See VOP
<http://www.kfunigraz.ac.at/imawww/thaller/wolfgang/vop-intro.html> for an
example.
- To provide detailed/complex mesh models with files of reasonable size.
- To write other scene description languages. If an API to read and write
this file format is released (like libpng), we can write programs to
generate such files. So programers could use their favorite language to
create scenes: C[++], Perl, Python, Lua (my choice :-), etc.
The current SDL is complete but a bit primitive (on the procedural side),
and using higher level languages (and perhaps faster) could be a bonus.
Of course, we can already write programs that output PoV-Ray scene files,
but you still have to parse them, which is a waste of time, particularly for
large scenes.

-- #=--=#=--=#=--=#=--=#=--=#=--=#=--=#=--=# --
Philippe Lhoste (Paris -- France)
Professional programmer and amateur artist
http://jove.prohosting.com/~philho/

Post a reply to this message

In VRML the files are text.  However, there are also "packed" versions of
VRML files that are just gziped VRML text files. VRML viewers are designed
to detect "packed VRML" files and render them properly.  It would seem that
this capability could be added to POV-Ray as well.  Gzip is also supported
across all major platforms (Windows, Linux, etc).

This kind of approach might help provide the functionality you seek.

Jim

Post a reply to this message

"Jim Kress" <nos### [at] kressworkscom> wrote:
> In VRML the files are text.  However, there are also "packed" versions of
> VRML files that are just gziped VRML text files. VRML viewers are designed
> to detect "packed VRML" files and render them properly.  It would seem
that
> this capability could be added to POV-Ray as well.  Gzip is also supported
> across all major platforms (Windows, Linux, etc).
>
> This kind of approach might help provide the functionality you seek.

I doubt so. Allowing PoV-Ray to read gzipped scene files can be useful for
faster reading of huge mesh files, but it still has to parse them, using a
lot of memory and time in the process.
In my idea, this data would have been already parsed, saving a lot of time,
and a bit of memory.

-- #=--=#=--=#=--=#=--=#=--=#=--=#=--=#=--=# --
Philippe Lhoste (Paris -- France)
Professional programmer and amateur artist
http://jove.prohosting.com/~philho/

Post a reply to this message

I believe this was discussed about a year back, but I can't find the
thread.  Warp had quite a lot to say on the matter (surprised? ;)

IIRC, it was considered too difficult to implement now, but reasonable
to do in the V4 rewrite.  IIRC of course...



"Philippe Lhoste" <Phi### [at] GMXnet> wrote in message
news:3d4958b8@news.povray.org...
> I would like to promote the idea of a PoV-Ray binary scene format.
> I know the idea isn't orginal, I have even found something like that
in the
> VFAQ, but I hope to give here one or two new ideas...
>
> Actually, I am not a big fan of binary file formats, because it is
easier to
> hack (and break :-) textual files within a text editor.
> But binary files have the advantage of being compact and easy/fast to
parse,
> ie. no conversion of textual representation of numbers to their binary
> values, read a 16bit or 32bit token instead of several characters for
a
> keyword to hash and lookup, etc.
>
> Note: I have seen in the PoV-Ray VFAQ that binary format (it was for
mesh
> files) was out of question, partly because of portability issues, ie.
not
> the same binary representation of float numbers between platforms.
Some
> programs, like the Lua language, are very portable and still manage to
write
> and read binary data with float number (Lua's bytecode, for example).
So it
> is not an impossible task.
>
> The base idea relies on the idea I have on the way PoV-Ray parses the
scene
> files. I can be wrong (I haven't checked the code) and this whole
message
> can be pointless :-)
>
> I believe the parser reads the text file, tokenises it (replacing
keywords
> and identifiers by numbers) and then the engine executes this bytecode
> (running loops, etc.), creating object data in memory. When the scene
> creates an object, the engine allocates a structure representing the
> attributes of the object, sets defaults values, and fill in the
indicated
> values. The final result is a in-memory binary representation of the
scene.
>
> Of course, this data isn't suitable for raw serialisation (output to a
> file). Pointers have to be relative, etc., plus it is non-portable, as
the
> internal representation of objects is subject to change from one
release to
> the next.
>
> We need then a robust file format, probably using chunks like [T]IFF
or PNG
> formats.
> Older engines will ignore unkown data while newer ones will provide
default
> values where missing data.
>
> So, we could ask the engine to output the scene data after parsing, or
to
> load one and render it.
> Note that this would be a secondary file format, I don't suggest to
drop the
> current format!
>
> What would be the use of such file?
> Among numerous ideas:
> - To give a model without revealing some tricks used to make it. So it
still
> can be rendered with different point of view, lighting, etc.
> Of course, this format can be disassembled, but in "unrolled" format,
ie.
> with only the raw list of generated objects, without the coding
wizardry
> that could have been used to create it.
> - To ease the creation of PoV-Ray to other 3D formats converters. They
are
> quite rare currently, because you either have to rip the parser from
> PoV-Ray, or write your own, with probably incompleteness. With this
unrolled
> format, you avoid at least the task of interpreting the procedural
language,
> of checking syntax, etc.
> - In the same spirit, to ease the creation of PoV-Ray pre-viewers, eg.
> converting objects to OpenGL to quickly see a scene under various
points of
> view. See VOP
> <http://www.kfunigraz.ac.at/imawww/thaller/wolfgang/vop-intro.html>
for an
> example.
> - To provide detailed/complex mesh models with files of reasonable
size.
> - To write other scene description languages. If an API to read and
write
> this file format is released (like libpng), we can write programs to
> generate such files. So programers could use their favorite language
to
> create scenes: C[++], Perl, Python, Lua (my choice :-), etc.
> The current SDL is complete but a bit primitive (on the procedural
side),
> and using higher level languages (and perhaps faster) could be a
bonus.
> Of course, we can already write programs that output PoV-Ray scene
files,
> but you still have to parse them, which is a waste of time,
particularly for
> large scenes.
>
> -- #=--=#=--=#=--=#=--=#=--=#=--=#=--=#=--=# --
> Philippe Lhoste (Paris -- France)
> Professional programmer and amateur artist
> http://jove.prohosting.com/~philho/
>
>

Post a reply to this message

"Philippe Lhoste" <Phi### [at] GMXnet> wrote in
news:3d4961d7@news.povray.org 

> I doubt so. Allowing PoV-Ray to read gzipped scene files can be useful
> for faster reading of huge mesh files, but it still has to parse them,
> using a lot of memory and time in the process.
> In my idea, this data would have been already parsed, saving a lot of
> time, and a bit of memory.

I had simmilar idea, exacly idea was to allow Pov to dump / read all it's 
data.

It would be useful for resuming rendering scenes that take long to 
parse/pre-process (many photons, radiosity etc).

I suggest to add i.e. +SA (Save All) option / +LA (Load all).
 
File used with +SA (or data_save_all=on in .ini) will create i.e. 
file_name.pbi (Pov Binary ?) containing  _all_ data - parsed scene, 
radiosity data, photons data.

File can ba later immediatly loaded.



-- 
#macro g(U,V)(.4*abs(sin(9*sqrt(pow(x-U,2)+pow(y-V,2))))*pow(1-min(1,(sqrt(
pow(x-U,2)+pow(y-V,2))*.3)),2)+.9)#end#macro p(c)#if(c>1)#local l=mod(c,100
);g(2*div(l,10)-8,2*mod(l,10)-8)*p(div(c,100))#else 1#end#end light_source{
y 2}sphere{z*20 9pigment{function{p(26252423)*p(36455644)*p(66656463)}}}//M

Post a reply to this message

Xns### [at] 204213191226...

> It would be useful for resuming rendering scenes that take long to
> parse/pre-process (many photons, radiosity etc).

You can already save and reload your photons and radiosity data so this is
not an issue.

Actually, I tend to think that the use of a binary format to save on parsing
time is a non-issue with modern computers. In my experience, large parsing
times are more often caused by insufficient RAM than by insufficient
processing speed. A contemporary 1-2 Ghz PC can gobble very large meshes and
process long loops very quickly. Clever scene design using switches and
dummies can remove most of the parsing-related problem during the testing
phase. In any case the parsing time is likely to be minimal when compared to
the rendering time, particularly now that we have the mesh2 format and the
saving/loading of photons and radiosity data.
If there's a complicated routine involved, it may be easier to output the
results to a text file once for all (see the demos in the "advanced/blocks"
directory).

This could be more interesting in animation, but then, since the purpose of
the animation is usually to change the scene geometry between frames, the
gain would be minimal in many cases.  (perhaps the IMP people use such a
binary format, I'm not sure).

A binary format could have its uses, but the absence of it is not currently
an actual limitation of Povray. I tend to believe that, given the nature of
Povray, implementing it in a useful way would be quite difficult.

G.


--

**********************
http://www.oyonale.com
**********************
- Graphic experiments
- POV-Ray and Poser computer images
- Posters

Post a reply to this message

Philippe Lhoste <Phi### [at] gmxnet> wrote:
> Note: I have seen in the PoV-Ray VFAQ that binary format (it was for mesh
> files) was out of question

  It's not out of question. It's just difficult to make it so that it will
work without problems in any system where POV-Ray has been ported. The
biggest problem is endianess, and a minor problem is different floating
point formats (which is actually a major problem in the case that the
format is different; but I consider it minor here because virtually all
practical computers use the same floating point format).

  As for the binary povray format, it's not good to make a kludged format
which is just a pov-file in binary format. It just makes the files smaller,
without any other benefits.
  The correct way to go is the to change the POV-Ray parser completely.
It should work like perl: It first byte-codes the source file to memory
and then it interpets this byte code (the function parsing already does
this; it just needs to be extended to everything). The main reason for
doing this is speed: If the source has lots of loops, macros, etc, the
speed increase could be enormous compared to the current speed.
  After this is done, it should be easy to just make a raw dump of the
bytecode to a file. This file will then be the binary format. Reading
this file will then be extremely easy: Just read the entire file to memory
and start interpreting. This saves the parsing stage, which can be really
slow.

-- 
#macro N(D)#if(D>99)cylinder{M()#local D=div(D,104);M().5,2pigment{rgb M()}}
N(D)#end#end#macro M()<mod(D,13)-6mod(div(D,13)8)-3,10>#end blob{
N(11117333955)N(4254934330)N(3900569407)N(7382340)N(3358)N(970)}//  - Warp -

Post a reply to this message

[description of a POV-Ray binary format]

>
>What would be the use of such file?
>Among numerous ideas:
>- To give a model without revealing some tricks used to make it. So it still
>can be rendered with different point of view, lighting, etc.
>Of course, this format can be disassembled, but in "unrolled" format, ie.
>with only the raw list of generated objects, without the coding wizardry
>that could have been used to create it.

    I might be in the minority, but I feel this is actually a good reason
NOT to do it.  The POV-Ray community is the main reason for using POV-Ray,
in my mind; and I _like_ the fact that people share code as well as ideas.
I'm just an old-time coot, I suppose, but I liked the Net in 1994, when
everyone used transparent, standard HTML and you could "view source" on a
cool page and learn something.  And I've always liked that about POV-Ray,
too.

    I think the POVteam should avoid anything that encourages people to wrap
their coding in impenetrable layers.  I think they should avoid anything
that allows one to "give a model without revealing some tricks used to make
it".

Post a reply to this message

Cool... bytecodes and a POV-Ray Virtual Machine...

As a programmer, I'm thrilled with the possibilities. You could 
theoretically compile Nvidia's cg to POV bytecodes and then render it... 
or have a POVMan interface...

Well, back to real life. :-)

Warp wrote:
> Philippe Lhoste <Phi### [at] gmxnet> wrote:
> 
>>Note: I have seen in the PoV-Ray VFAQ that binary format (it was for mesh
>>files) was out of question
> 
> 
>   It's not out of question. It's just difficult to make it so that it will
> work without problems in any system where POV-Ray has been ported. The
> biggest problem is endianess, and a minor problem is different floating
> point formats (which is actually a major problem in the case that the
> format is different; but I consider it minor here because virtually all
> practical computers use the same floating point format).
> 
>   As for the binary povray format, it's not good to make a kludged format
> which is just a pov-file in binary format. It just makes the files smaller,
> without any other benefits.
>   The correct way to go is the to change the POV-Ray parser completely.
> It should work like perl: It first byte-codes the source file to memory
> and then it interpets this byte code (the function parsing already does
> this; it just needs to be extended to everything). The main reason for
> doing this is speed: If the source has lots of loops, macros, etc, the
> speed increase could be enormous compared to the current speed.
>   After this is done, it should be easy to just make a raw dump of the
> bytecode to a file. This file will then be the binary format. Reading
> this file will then be extremely easy: Just read the entire file to memory
> and start interpreting. This saves the parsing stage, which can be really
> slow.
>

Post a reply to this message

In article <3d4989dc@news.povray.org>, Warp <war### [at] tagpovrayorg> 
wrote:

>   It's not out of question. It's just difficult to make it so that it will
> work without problems in any system where POV-Ray has been ported. The
> biggest problem is endianess, and a minor problem is different floating
> point formats (which is actually a major problem in the case that the
> format is different; but I consider it minor here because virtually all
> practical computers use the same floating point format).

You could just standardize on one floating point format for the POV data 
files, and write platform indpendant code that reads it byte by byte and 
converts it to a float or takes a float and writes it in the correct 
format, but the format might have different precision from what is 
native on some platforms. I don't think this would cause any serious 
problems...it's a pain, but not an insurmountable obstacle.


>   After this is done, it should be easy to just make a raw dump of the
> bytecode to a file. This file will then be the binary format. Reading
> this file will then be extremely easy: Just read the entire file to memory
> and start interpreting. This saves the parsing stage, which can be really
> slow.

Actually, the compilation stage isn't necessarily that slow...you could 
get a huge speedup over the present parser by just compiling to 
bytecodes and running on the VM every time. Combine this with a compact 
mesh format (maybe a custom binary format compressed with gzip 
compression), and you have taken care of most of the reasons for a 
binary scene format.

Another possibility: instead of saving the compiled bytecode program or 
the internal data structures of the raytracer, save the commands the 
program makes to the tracer. These commands could be used to rebuild the 
scene without compiling/loading and interpreting the scene file, but 
would be easier to serialize to a file than the complex internal data 
structures: they would already be pretty serial in nature. I don't see 
much reason to go through the effort though...whatever you do, you end 
up with a non-human-readable file that is smaller than the text file 
(but not significantly smaller than a compressed version of the text 
file would be) and another input file format to worry about with 
transferring between unofficial versions and updating the official 
version.

-- 
Christopher James Huff <chr### [at] maccom>
POV-Ray TAG e-mail: chr### [at] tagpovrayorg
TAG web site: http://tag.povray.org/

Post a reply to this message