a macro for reading and writing the POV-Ray variant of CSV (Comma Separated Values) data files.
a typical CSV file is organised as one record per line/row, but while POV-Ray provides the primitives ("directives") for file handling, it has no concept of record(s), one reads value after value from a file until done or end of file. every file which differs in the number and or type of columns ("fields"), and or the number of rows, requires, basically, writing custom file handling code in SDL (Scene Description Language). obviously, that makes using CSV data in scene files a bit of a "chore", with many opportunities for .. mistakes.
the Filed() macro has been designed
to hide the "gory details"
of dealing with data files,
and uses modern POV-Ray array syntax
to provide a flexible,
fairly intuitive I hope,
user interface.
with Filed() one defines an array of
"fields" (aka a "record"),
then all reading and writing (and appending) is
done in record units.
a 'dictionary {}'
packages all the information needed by the macro,
it has only a small number of self-explanatory,
mostly, keys.
regarding the code design, data files are always read whole before processing, meaning that, for a short time, approximately twice as much memory is used as is occupied by the data. which should not be a problem unless the file in question is really big. there is, of course, no support for writing binary files, nor for the associated big/little-endian data types.
note – there is also a "philosophical"
difference with regard to the behaviour of the macro.
with POV-Ray, when you open a non-existing file for
appending it will be created, and if you open an
existing file for writing it will be overwritten.
with Filed() you have to explicitly
set a key
('.Strict: false')
to get it to work like POV-Ray.
the default is not to overwrite any existing file,
nor try to append to one which does not exist.
the SDL code in the include file, version
202108.2,
is written for the
Persistence Of Vision Raytracer version 3.8
or later.
seen from the user's perspective, the include file
only contains the macro and the variable described below.
all other, undocumented, globally visible identifiers
in the file are prefixed 'fild_'
to reduce "pollution" in the global namespace.
fild_workingDirthis optional variable can be declared to provide the string name of a working directory, which will be prefixed to the file names in subsequent macro calls; the value is used "as is", and must end with the correct (for the OS) path separator.
Filed(dict)
the Filed() macro takes a single
'dictionary {}' argument,
and returns ("expands to") nothing.
when reading a file those data become available in the corresponding dictionary keys, '.Data' and perhaps '.Names'. when writing or appending a file, that data must be provided, in the form of arrays, via the same key(s). all dictionary keys are required, unless described as "optional", and depending on the type of file access.
array mixed' elements
when records are heterogeneous.
for examples of each see the examples section below.
note that although POV-Ray permits individual array
elements to be "uninitialised",
empty elements, whether whole records or just a field,
are not allowed in the file context.
false when omitted.
1,
irrespective of an existing column name/label row.
true
when omitted.
use .. sparingly.
#warning's.
the default for this optional key is false
when omitted, ie run "silent".
regarding records and columns per line/row in a file – because POV-Ray simply returns one value after another, and the user defines what constitutes a record, files can be read when there are multiple, complete, records per line. indeed, if all fields have the same type, or form a suitable "pattern", line boundaries are of no importance and it only matters whether the last record was read in full.
Filed()
is compatible with the
Tabulated() macro,
in that the
'.Fields',
'.Data', and
'.Names'
keys can be used as
'.DataColumns',
'.DataTable', and
'.ColLabels', respectively;
see the examples section.
however, remember that
Tabulated()
cannot deal with more than twenty columns/fields.
note – all sorts of errors can occur which are
not handled by the Filed() code.
forgetting to specify '.hasNames' when
there is a column names row, for instance,
a missing field or an empty field
(ie consecutive commas),
or accidentally trying to read an empty file,
all these (and more ;-))
result in some POV-Ray "Parse Error"
being thrown.
the archive contains a number of demo and example scene
files which, between them, illustrate usage of all of
Filed()'s features.
the three files named 'ex?.pov' are very
simple instances of reading and writing the three
different "shapes" of array accepted in
a '.Data' key.
the three scenes share a common structure:
reading a data file,
followed by outputting the data,
followed by writing the data.
the second and third part both are "guarded"
by conditionals,
#if blocks,
where only the value needs changing
(to 'true') when wanted.
part two, output, is just a choice between text only
output on the #debug stream (the default),
and a rendered image,
while part three is there for checking, comparing,
a newly written file to an original.
everything is, of course, fully commented.
these examples can be rendered as is,
the snippet shows a typical command-line.
$ cd /path/to/filed/demo $ povray ex1.pov +l.. ...
the '+l..' option let's
POV-Ray find its include file.
for the rest of the command-line,
indicated by the ellipsis,
see the recommendation in the header lines
of each respective scene.
it is evident from the examples that "selling" something as inherently not-visual as a file handling macro is .. difficult :-). still, thanks to the help of a "pro-active" beta-tester, I have managed to include not one but two, um, colourful uses of the macro. the first demo, introduced in Thomas de Groot's own words:
" The flower planting demo randomly plants genetically-tailored flowers on an asteroid. It requires two scene files and one ini file. "flowers.pov" has to be run first, in order to plant a first batch of 'red' flowers, the data of which is written to 'flowers.txt'. It is followed by an 'append' section of 'yellow' flowers, and - optionally - one of 'blue' flowers. The user is encouraged to play with the random seeds, and the number of flowers in each batch. The scene file renders the asteroid complete with planted flowers, in the background a table of the flower data, from which a range of data can be provided. Obviously this can only be achieved if the user has 'tabulated.inc' available. "flowers_read.pov" has been specially written to demonstrate how "filed.inc" works to 'read' a previously generated 'flowers.txt' data file. An animation of the asteroid can be generated with the help of the "flowers.ini" file. "
the second demo accompanying
Filed(), discussed below,
illustrates a per-frame "state" file
and keeping a (very) basic log file,
in an animation which is a little OTT perhaps
(it took on a life of its own :-)).
the idea for the example is to create a small
"arena" in which two
"robots" are placed.
in each frame each robot in turn gets one move,
which it can choose from three,
and a check is carried out to see whether the move
resulted in one robot occupying the same square/tile
as the other. such a collision causes
"damage" to the robot which moved,
and is shown by a change of colour.
when either robot runs out of colours (six),
the animation ends at the next collision caused by it.
the "two objects animated" (TOA) animation consists of three files, two housing the SDL code, split into an arena ('.pov') and an include file with macros and some constant data, and an '.ini' file to drive the demo. TOA could have been named "bumble bots", more accurately, as they cannot "see" and are unaware of their surround. however, the bots are "individuals", implemented as separate macros, and choose their moves using different strategies. it would be fun to see something like TOA developing to the point where "aware" bots, written by different authors, could be entered into a common arena for set tasks.
a small number of things can be adjusted
for every run of TOA:
the size of the arena,
where in the arena to place each bot,
and per bot random number generator seeds;
both robots will be created the same colour,
facing "south",
ie facing the camera.
changing the size of the arena will, usually, also
require adjusting the camera's settings.
it is important to note that the code has not been
"bullet-proofed" in any sense, that is
supplying, for instance, a
'#declare area_ = <-5,0.28>;'
will produce interesting behaviour, at best.
the 'toa.pov' settings, as shipped,
are shown in the second snippet.
the arena is made quite small to increase the potential
for collisions,
hence this animation will run for just forty-odd frames.
only the working directory declaration (line 21)
and the output_file_name in the .ini
file need checking/adjusting.
/* size of arena: <cols, rows> */
#declare area_ = <5,3>;
/* initial positions of objects: <col, row> */
#declare init_pos_ = array [2] {<1,2>,<4,1>};
/* per object rng seeds */
#declare init_seeds_ = array [2] {54321,45678};
the first frame ('0') of the animation
is used to set up,
create the "infrastructure".
a check that neither state nor log file exist comes
first.
then logging is prepared to create a file,
and the current system date/time is used
to make the first entry.
next, the two robots are created as the
'array mixed' elements
of a simple array; a XY position, a direction,
a damage level, and a seed value make a bot.
the array data is then used to create the state file,
and the new state is logged to file.
finally, the render of the frame shows a simple
visualisation of the initial state.
the second and subsequent frames of the animation also begin with a test for files, this time expecting to find existing state and log files. the state file is read and a check is made to see if either robot has reached the maximum damage level, the animation ends if that is the case. now each robot, in turn, gets to make one move, and each outcome is checked to see whether a collison occurred. the updated state is then written back to file, and logged, too. again, the current state is rendered.
after an animation has completed, it is useful to have a good image viewer to hand, one that makes stepping through the frame sequence easy. reading the log file in a text viewer while stepping through the corresponding frames allows one to re-trace the moves.
#declare fild_workingDir = "/tmp/toa/";
the TOA code creates two files, appends and re-writes these and creates an image for every frame, every time it runs. it is a good idea™ therefore to direct all files to a "scratch" directory, ideally located on a RAM disk, the snippet shows a typical (on Linux) setting for a working directory.
I am grateful for Thomas de Groot's contributions
to the project.
not only gave Thomas freely of his time testing the macro
and helping improve this documentation,
but he also offered
– kudos –
to write a code example,
included in the demo directory;
note I have made a couple of cosmetic edits,
and added (as comment) an alternative
'rotate' to
'flowers_read.pov',
as suggested by Thomas in an email.
while the 'filed.inc' include is stand-alone, that is, requires no other include file(s), the demos and example scenes are not, unfortunately. links for the macros needed below.
Ruled()
Tabulated()
"The End." enjoy.