Please give this version a thorough shakedown, with focus on parsing:

https://github.com/POV-Ray/povray/releases/tag/v3.8.0-x.tokenizer.9999

Besides testing functionality, please also look at parsing speed.


Known issues:

- `#read` currently disabled
- macro caching currently disabled
- system-specific character encoding in strings currently not supported
(but utf8 should work)
- signature BOM in utf8-encoded files currently not supported
- probably one or two things I'm forgetting to mention

- Backward compatibility with scenes that use single backslashes in
literal filenames has been sacrificed, and will most likely not be restored.


The parser overhaul has been extended to cover not just the "scanner"
stage, but also "raw tokenization", whereby literals are already
evaluated (e.g. digit sequences converted to numbers, and escape
sequences in strings resolved) and reserved words identified. Pretty
much the only tokenization-related task remaining in the monolithic bulk
of the parser is identifying variables, which requires knowledge and
understanding of context.

In passing, a few internal limitations of the parser have also been
lifted; for instance, there is no longer any fixed limit to the nesting
depth of include files, or the nesting depth of parentheses, braces etc.

Post a reply to this message