POV-Ray v3.7 (for Windows) behaves as follows with respect to different
#version/charset settings (as tested with German locale):


Common
======

- "\uXXXX" escape sequences are technically always interpreted as UCS-2
character codes. Note however that depending on the context in which a
string is used, the /effective/ interpretation may vary.

- `asc()` and `chr()` functions technically always operate according to
UCS-2 character encoding. Note however that depending on the context in
which a string is used, the /effective/ encoding may vary.


3.0 / ascii
===========

- Non-ASCII octets in strings are technically decoded according to ISO
8859-1 (Latin-1; matching the display in the editor for many characters,
except for codes hex 80-9F, though I presume the editor display may vary
with the system's locale, while the Latin-1 decoding is invariant). Note
however that depending on the context in which a string is used, the
/effective/ decoding may vary.

- When used in file names or debug output, strings are effectively
subject to re-interpretation of the UCS-2 character codes as
Windows-1252 codes (matching the display in the editor; presumably this
varies with the system's locale), with character codes above hex FF
interpreted modulo 256.

- When used in text primitives, non-ASCII characters in strings are
typically garbled, depending on the font used; for instace, with
Microsoft's Arial font the text appears to be subject to
re-interpretation of the UCS-2 character codes as Macintosh Roman codes,
while with POV-Ray's `cyrvetic.ttf` the re-interpretation seems to be as
Windows-1251 (Cyrillic) codes. Character codes above hex FF are treated
according to obscure rules in all cases. (In some cases,
re-interpretation may happen to match the display in the editor.)


3.0 / utf8
==========

- Non-ASCII octets in strings are technically decoded according to
UTF-8. Note however that depending on the context in which a string is
used, non-ASCII characters may effectively become garbled.

- When used in file names or debug output, strings are effectively
subject to re-interpretation of the UCS-2 codes as Windows-1252 codes
(presumably this varies with the system's locale), with codes above hex
FF interpreted modulo 256.

- When used in text primitives, non-ASCII characters in strings may or
may not be garbled, depending on the font used; for instace, with
Microsoft's Arial font the text is displayed as expected for UCS-2
encoded text, while with POV-Ray's `cyrvetic.ttf` it appears to be
subject to re-interpretation of the UCS-2 codes as Macintosh Cyrillic
codes with codes above hex FF being treated according to obscure rules.


3.7 / ascii
===========

- Non-ASCII octets in strings are decoded as blanks (ASCII hex 20;
non-ASCII characters can still be inserted via `\uXXXX` escape sequences
or the `chr()` function though).

- When used in file names or debug output, non-ASCII characters in
strings (entered via "\uXXXX" or `chr()`) are substituted with blanks.

- When used in text primitives, non-ASCII characters in strings (entered
via "\uXXXX" or `chr()`) are typically garbled, depending on the font
used; for instace, with Microsoft's Arial font the text appears to be
subject to re-interpretation of the codes as Macintosh Roman codes,
while with POV-Ray's `cyrvetic.ttf` the re-interpretation seems to be as
Windows-1251 (Cyrillic) codes. Character codes above hex FF are treated
according to obscure rules in all cases.


3.7 / utf8
==========

- Non-ASCII octets in strings are decoded according to UTF-8.

- When used in file names or debug output, non-ASCII characters in
strings are substituted with blanks.

- When used in text primitives, non-ASCII characters in strings may or
may not be garbled, depending on the font used; for instace, with
Microsoft's Arial font the text is displayed as expected for UCS-2
encoded text, while with POV-Ray's `cyrvetic.ttf` it appears to be
subject to re-interpretation of the UCS-2 codes as Windows-1251
(Cyrillic) codes, again with codes above hex FF being treated according
to obscure rules.

Post a reply to this message

Note that the TrueType font decoding used to be rather buggy before 3.6 (or was
it 3.5?), which is why i.e. in 3.0 it matched the MacRoman font tables before
ever getting to the other tables.

clipka <ano### [at] anonymousorg> wrote:
> POV-Ray v3.7 (for Windows) behaves as follows with respect to different
> #version/charset settings (as tested with German locale):
>
>
> Common
> ======
>
> - "\uXXXX" escape sequences are technically always interpreted as UCS-2
> character codes. Note however that depending on the context in which a
> string is used, the /effective/ interpretation may vary.
>
> - `asc()` and `chr()` functions technically always operate according to
> UCS-2 character encoding. Note however that depending on the context in
> which a string is used, the /effective/ encoding may vary.
>
>
> 3.0 / ascii
> ===========
>
> - Non-ASCII octets in strings are technically decoded according to ISO
> 8859-1 (Latin-1; matching the display in the editor for many characters,
> except for codes hex 80-9F, though I presume the editor display may vary
> with the system's locale, while the Latin-1 decoding is invariant). Note
> however that depending on the context in which a string is used, the
> /effective/ decoding may vary.
>
> - When used in file names or debug output, strings are effectively
> subject to re-interpretation of the UCS-2 character codes as
> Windows-1252 codes (matching the display in the editor; presumably this
> varies with the system's locale), with character codes above hex FF
> interpreted modulo 256.
>
> - When used in text primitives, non-ASCII characters in strings are
> typically garbled, depending on the font used; for instace, with
> Microsoft's Arial font the text appears to be subject to
> re-interpretation of the UCS-2 character codes as Macintosh Roman codes,
> while with POV-Ray's `cyrvetic.ttf` the re-interpretation seems to be as
> Windows-1251 (Cyrillic) codes. Character codes above hex FF are treated
> according to obscure rules in all cases. (In some cases,
> re-interpretation may happen to match the display in the editor.)
>
>
> 3.0 / utf8
> ==========
>
> - Non-ASCII octets in strings are technically decoded according to
> UTF-8. Note however that depending on the context in which a string is
> used, non-ASCII characters may effectively become garbled.
>
> - When used in file names or debug output, strings are effectively
> subject to re-interpretation of the UCS-2 codes as Windows-1252 codes
> (presumably this varies with the system's locale), with codes above hex
> FF interpreted modulo 256.
>
> - When used in text primitives, non-ASCII characters in strings may or
> may not be garbled, depending on the font used; for instace, with
> Microsoft's Arial font the text is displayed as expected for UCS-2
> encoded text, while with POV-Ray's `cyrvetic.ttf` it appears to be
> subject to re-interpretation of the UCS-2 codes as Macintosh Cyrillic
> codes with codes above hex FF being treated according to obscure rules.
>
>
> 3.7 / ascii
> ===========
>
> - Non-ASCII octets in strings are decoded as blanks (ASCII hex 20;
> non-ASCII characters can still be inserted via `\uXXXX` escape sequences
> or the `chr()` function though).
>
> - When used in file names or debug output, non-ASCII characters in
> strings (entered via "\uXXXX" or `chr()`) are substituted with blanks.
>
> - When used in text primitives, non-ASCII characters in strings (entered
> via "\uXXXX" or `chr()`) are typically garbled, depending on the font
> used; for instace, with Microsoft's Arial font the text appears to be
> subject to re-interpretation of the codes as Macintosh Roman codes,
> while with POV-Ray's `cyrvetic.ttf` the re-interpretation seems to be as
> Windows-1251 (Cyrillic) codes. Character codes above hex FF are treated
> according to obscure rules in all cases.
>
>
> 3.7 / utf8
> ==========
>
> - Non-ASCII octets in strings are decoded according to UTF-8.
>
> - When used in file names or debug output, non-ASCII characters in
> strings are substituted with blanks.
>
> - When used in text primitives, non-ASCII characters in strings may or
> may not be garbled, depending on the font used; for instace, with
> Microsoft's Arial font the text is displayed as expected for UCS-2
> encoded text, while with POV-Ray's `cyrvetic.ttf` it appears to be
> subject to re-interpretation of the UCS-2 codes as Windows-1251
> (Cyrillic) codes, again with codes above hex FF being treated according
> to obscure rules.

Post a reply to this message

clipka <ano### [at] anonymousorg> wrote:
> 3.7 / ascii
> ===========
>
> - Non-ASCII octets in strings are decoded as blanks (ASCII hex 20;
> non-ASCII characters can still be inserted via `\uXXXX` escape sequences
> or the `chr()` function though).
>
> - When used in file names or debug output, non-ASCII characters in
> strings (entered via "\uXXXX" or `chr()`) are substituted with blanks.
>
> - When used in text primitives, non-ASCII characters in strings (entered
> via "\uXXXX" or `chr()`) are typically garbled, depending on the font
> used; for instace, with Microsoft's Arial font the text appears to be
> subject to re-interpretation of the codes as Macintosh Roman codes,
> while with POV-Ray's `cyrvetic.ttf` the re-interpretation seems to be as
> Windows-1251 (Cyrillic) codes. Character codes above hex FF are treated
> according to obscure rules in all cases.
>
>
> 3.7 / utf8
> ==========
>
> - Non-ASCII octets in strings are decoded according to UTF-8.
>
> - When used in file names or debug output, non-ASCII characters in
> strings are substituted with blanks.
>
> - When used in text primitives, non-ASCII characters in strings may or
> may not be garbled, depending on the font used; for instace, with
> Microsoft's Arial font the text is displayed as expected for UCS-2
> encoded text, while with POV-Ray's `cyrvetic.ttf` it appears to be
> subject to re-interpretation of the UCS-2 codes as Windows-1251
> (Cyrillic) codes, again with codes above hex FF being treated according
> to obscure rules.

I won't dispute your results because they don't surprise me, but I dispute the
fact that the code behaves unpredictably: The code is in OpenFontFile, and it
does pick the Unicode tables as top priority if Unicode is specified as text
format. It only falls back to MacRoman (which next to all TTFs contain for
legacy reasons) if it cannot find any of the Unicode formats that it does
support. So the real question is if maybe in the 18 years since this code was
introduced in 3.5 based on TTFs shipped with Mac OS 9 and Windows 2000, some
other TTF Unicode formats were introduced that the code does not support...

Post a reply to this message

"Thorsten Froehlich" <nomail@nomail> wrote:
> I won't dispute your results because they don't surprise me, but I dispute the
> fact that the code behaves unpredictably: The code is in OpenFontFile, and it
> does pick the Unicode tables as top priority if Unicode is specified as text
> format. It only falls back to MacRoman (which next to all TTFs contain for
> legacy reasons) if it cannot find any of the Unicode formats that it does
> support. So the real question is if maybe in the 18 years since this code was
> introduced in 3.5 based on TTFs shipped with Mac OS 9 and Windows 2000, some
> other TTF Unicode formats were introduced that the code does not support...

To answer my own questions, the formats did change...
https://developer.apple.com/fonts/TrueType-Reference-Manual/RM06/Chap6cmap.html

So what you see is the code falling back to a ASCII (MacRoman) code tables as a
fail-safe because it cannot use the newer Unicode tables most current fonts seem
to contain.

Post a reply to this message

Am 06.06.2018 um 19:32 schrieb Thorsten Froehlich:
> "Thorsten Froehlich" <nomail@nomail> wrote:
>> I won't dispute your results because they don't surprise me, but I dispute the
>> fact that the code behaves unpredictably:

I never used the word "unpredictably" - I used the phrase "according to
obscure rules". As in, "pretty difficult to guess without looking at the
actual code and maybe the font file".

>> The code is in OpenFontFile, and it
>> does pick the Unicode tables as top priority if Unicode is specified as text
>> format. It only falls back to MacRoman (which next to all TTFs contain for
>> legacy reasons) if it cannot find any of the Unicode formats that it does
>> support. So the real question is if maybe in the 18 years since this code was
>> introduced in 3.5 based on TTFs shipped with Mac OS 9 and Windows 2000, some
>> other TTF Unicode formats were introduced that the code does not support...
> 
> To answer my own questions, the formats did change...
> https://developer.apple.com/fonts/TrueType-Reference-Manual/RM06/Chap6cmap.html
> 
> So what you see is the code falling back to a ASCII (MacRoman) code tables as a
> fail-safe because it cannot use the newer Unicode tables most current fonts seem
> to contain.

Um... I do /not/ really think so... you know, the font that I found to
be ok (with `charset utf8`) is the Microsoft Arial font on my Windows 10
machine, while the other is POV-Ray's cyrvetic.ttf. So I have a hunch
that the ok font /may/ be the newer one.

So I'd guess ye olde cyrvetic.ttf never had a Unicode table to begin with.

Also, I don't see a fallback to Mac Roman in the cyrvetic font, but
rather a fallback to Mac Cyrillic.


(As it turns out, the POV-Ray TrueType font gives a rat's arse about the
platformSpecificID when looking for a code table it can use; platformID
is all it tests for.)

Post a reply to this message

Am 06.06.2018 um 19:13 schrieb Thorsten Froehlich:
> Note that the TrueType font decoding used to be rather buggy before 3.6 (or was
> it 3.5?), which is why i.e. in 3.0 it matched the MacRoman font tables before
> ever getting to the other tables.

Presumably before v3.5, because that's the version number threshold
POV-Ray v3.7 tests for.

But right now I don't really care much /why/ POV-Ray is behaving the way
it does; at the moment I'm just taking stock of /how/ it currently
behaves, so that I can mimick it as accurately as reasonably possible
within the framework of the v3.8 parser changes.

Post a reply to this message

clipka <ano### [at] anonymousorg> wrote:
> Am 06.06.2018 um 19:13 schrieb Thorsten Froehlich:
> > Note that the TrueType font decoding used to be rather buggy before 3.6 (or was
> > it 3.5?), which is why i.e. in 3.0 it matched the MacRoman font tables before
> > ever getting to the other tables.
>
> Presumably before v3.5, because that's the version number threshold
> POV-Ray v3.7 tests for.

Yes, there should be a compatibility warning in there somewhere because it was
decided back then that the previous behavior was so badly broken that only ASCII
ever worked. The other problem found back then was that especially many (old)
free fonts used to be not exactly conforming to the specifications available at
the time (i.e. some only worked on Macs, others only on Windows), which might
also be due to the specs not being easily accessible when the freeware that some
people used to create those fonts was published. So there is very little point
trying to recreate all that odd behavior.

> But right now I don't really care much /why/ POV-Ray is behaving the way
> it does; at the moment I'm just taking stock of /how/ it currently
> behaves, so that I can mimick it as accurately as reasonably possible
> within the framework of the v3.8 parser changes.

I would suggest to go with the documented behavior, which is ASCII support for
charset ASCII and UCS-2 support (because that is all what fonts had back then)
for UTF-8 input. And after that maybe consider adding the missing cmap table
support for the extended Unicode stuff beyond the 16 bit codes. You will also
need to fiddle with color fonts and the like then ... but at least nowadays you
can find those specs on the internet. Adding Unicode support to 3.5 meant buying
the Unicode 3.0 book...

Btw., I found that the best way to determine what happens inside the TrueType
parser code is enabling the debug code in there. Otherwise you can never be
certain there isn't some problem with the input file rather than the code.

Thorsten

Post a reply to this message

Living in the U.S, I've never paid much attention to text encodings other than
ASCII ("US-ASCII" I suppose)-- although I've seen "UTF-8" etc. show up from time
to time in others' scene files or include files.

When writing my own include files manually (for my own use, and saved as 'plain
text'), I've always used either Window's NOTEPAD or WORDPAD-- only because they
are simple and available. But I'm having a problem saving even a *simple* UTF-8
file.

WORDPAD (in my Win7 installation) can encode text in different ways:
plain text (.txt)-- is that the same as ANSI?
Rich Text Format (.rtf)-- one of Microsoft's own file-types
Unicode (.txt ?)
Unicode big endian (.txt ?)
UTF-8 (.txt ?)
.... plus a few others like XML and such.

[I assume that the various Unicode files are output as .txt file types.]

The thing is, WORDPAD's  'plain text file' is the ONLY one of its own encodings
that can be successfully read by POV-Ray as a text include file; all the others
produce various error messages. Most of those errors are expected-- but even a
UTF-8 file doesn't work. This is... odd. Perhaps I don't understand how to use
Unicode files. OR, WORDPAD isn't writng the file correctly??

Code example:
#version 3.71;  // using 3.7.1 beta 9
global_settings {assumed_gamma 1.0 charset utf8}
#include "text file as UTF8.txt" // Saved as UTF-8. No strings in
           // the contents, just a single line--  #local R = 45;

Fatal error result:
"illegal character in input file, value is ef"
This happens whether global_settings has charset utf8 or no charset at all.
(BTW, I can't 'see' the 'ef' value when I open the file.) So It appears that
WORDPAD is appending a small header-- a BOM?-- which may not conform to UTF-8
specs(?)

So I tried it a different way, just to be complete:
#version 3.71;
global_settings {assumed_gamma 1.0} // NO charset here
#include "text file as UTF8.txt" // Saved as UTF-8 again, but this one
// has  global_settings{charset utf8}, plus   #local R = 45;

.... which has the same fatal result.

I looked at various Wikipedia pages ( "US-ASCII", Windows WordPad app", "UTF-8",
"Comparison of Unicode encodings" ), but I *still* don't have a full grasp of
the facts:

"WordPad for Windows XP [and later?] added full Unicode support, enabling
WordPad to support multiple languages..."

"[Windows] files saved as Unicode text are encoded as UTF-16 LE.  [not UTF-8,
unless that is specified] ...Such [Unicode] files normally begin with Byte Order
Mark (BOM), which communicates the endianness of the file content. Although
UTF-8 does not suffer from endianness problems, many Windows programs --i.e.
Notepad-- prepend the contents of UTF-8-encoded files with BOM, to differentiate
UTF-8 encoding from other 8-bit encodings." [Other 8-bit encodings meaning
"plain ACSII"?]

[However...]
"The Unicode Standard neither requires nor recommends the use of the BOM for
UTF-8, but warns that it may be encountered at the start of a file as a
transcoding artifact. The presence of the UTF-8 BOM may cause problems with
existing software that can [otherwise] handle UTF-8..."

"UTF-16 [is] incompatible with ASCII files, and thus requires Unicode-aware
programs to display, print and manipulate them, even if the file is known to
contain only characters in the ASCII subset...”

"UTF-16 does not have endianness defined, [but] this may be achieved by using a
byte-order mark [BOM] at the start of the text, or assuming big-endian... [but]
UTF-8 is standardised on a single byte order and does not have this problem."

"If any stored data is in UTF-8 (such as file contents or names), it is very
difficult to write a system that uses UTF-16 or UTF-32 as an API. [However,] it
is trivial to translate invalid UTF-16 to a unique (though technically invalid)
UTF-8 string, so a UTF-8 API can control both UTF-8 and UTF-16 files and names,
making UTF-8 preferred in any such mixed environment."

Post a reply to this message

On 25/06/2018 10:41, Kenneth wrote:
> Living in the U.S, I've never paid much attention to text encodings other than
> ASCII ("US-ASCII" I suppose)-- although I've seen "UTF-8" etc. show up from time
> to time in others' scene files or include files.
> 
Similarly, living in the UK I use the encoding used in the King James 
bible. ;-)
And if foreigners want to understand. I will shout. :-)


> When writing my own include files manually (for my own use, and saved as 'plain
> text'), I've always used either Window's NOTEPAD or WORDPAD-- only because they
> are simple and available. But I'm having a problem saving even a*simple*  UTF-8
> file.

Have you come across Notepad ++ ?
httos://noteoadplusplus.oro/

It is free, simple to use and jam packed with features such as Encoding, 
Syntax Highlighting for lots of languages, Macro recording etc.



-- 

Regards
     Stephen

Post a reply to this message

hi,

"Kenneth" <kdw### [at] gmailcom> wrote:
> ... text encodings other than ASCII ...
> When writing my own include files manually (for my own use, and saved as 'plain
> text'), I've always used either Window's NOTEPAD or WORDPAD-- only because they
> are simple and available.

many years ago I used to use 'Qedit'* but there will be any number of other,
often free editors.  the aBOMination is purely Microsoft, a "programmer's
editor" should not have issues.

https://en.wikipedia.org/wiki/The_SemWare_Editor
free for personal use but it only cost 30 USD (I think) to license, back then.


regards, jr.

Post a reply to this message

Stephen <mca### [at] aolcom> wrote:
>
> Have you come across Notepad ++ ?
> httos://noteoadplusplus.oro/
>

Ya know, I actually *did* have that app at one time-- on a hard drive that
failed-- but then I completely forgot about it! Thanks for the memory boost ;-)
I'll download it again. (I never expected Windows' own long-time 'core' apps to
have such a basic text-encoding flaw; apparently neither NOTEPAD nor WORDPAD
conform to the UTF-8 published standards, as far as I can tell. Maybe the
standards came later(!)

Post a reply to this message

Am 25.06.2018 um 11:41 schrieb Kenneth:
> Living in the U.S, I've never paid much attention to text encodings other than
> ASCII ("US-ASCII" I suppose)-- although I've seen "UTF-8" etc. show up from time
> to time in others' scene files or include files.
> 
> When writing my own include files manually (for my own use, and saved as 'plain
> text'), I've always used either Window's NOTEPAD or WORDPAD-- only because they
> are simple and available. But I'm having a problem saving even a *simple* UTF-8
> file.

Actually, no, you don't (if you do indeed use plain ASCII): Every ASCII
text file is also a perfectly valid UTF-8 file ;)


> WORDPAD (in my Win7 installation) can encode text in different ways:
> plain text (.txt)-- is that the same as ANSI?

It's the same as whatever codepage you happen to be using; probably
Windows-1252, also erroneously referred to as "ISO-8859-1" (of which
codepage 1252 happens to be a superset) or "ANSI" (presumably shorthand
for "ANSI/ISO-8859-1").

> Rich Text Format (.rtf)-- one of Microsoft's own file-types

Entirely different thing: This is not just a character encoding, but an
entirely different file format.

> Unicode (.txt ?)

That would be UTF-16 with Little Endian encoding (*).

> Unicode big endian (.txt ?)

That would presumably (the Windows 10 version doesn't have this) be
UTF-16 with Big Endian encoding (*).

(* Not to be confused with UTF-16LE or UTF-16BE, respectively; UTF-16
always has a signature (aka BOM = Byte Order Mark), UTF-16LE and
UTF-16BE never have.)

> UTF-8 (.txt ?)

That would presumably (again the Windows 10 version doesn't have this)
be UTF-8 with signature (aka BOM = Byte Order Mark", though in this
context that term might be misleading; see below).

> The thing is, WORDPAD's  'plain text file' is the ONLY one of its own encodings
> that can be successfully read by POV-Ray as a text include file; all the others
> produce various error messages. Most of those errors are expected-- but even a
> UTF-8 file doesn't work. This is... odd. Perhaps I don't understand how to use
> Unicode files. OR, WORDPAD isn't writng the file correctly??
> 
> Code example:
> #version 3.71;  // using 3.7.1 beta 9
> global_settings {assumed_gamma 1.0 charset utf8}
> #include "text file as UTF8.txt" // Saved as UTF-8. No strings in
>            // the contents, just a single line--  #local R = 45;
> 
> Fatal error result:
> "illegal character in input file, value is ef"
> This happens whether global_settings has charset utf8 or no charset at all.
> (BTW, I can't 'see' the 'ef' value when I open the file.) So It appears that
> WORDPAD is appending a small header-- a BOM?-- which may not conform to UTF-8
> specs(?)

The UCS specification explicitly and indiscriminately allows for UTF-8
files both with and without a signature. The Unicode specification is
more partial in that it discourages the presence of a signature, but
still allows it.

If present, the signature is (hex) `EF BB BF`, which "happens" to match
the UTF-8 encoding of U+FEFF, the UTF-16 and UTF-32 encodings of which
"happen" to match the signatures of UTF-16 and UTF-32 indicating byte order.

So Wordpad is without fault here (these days, at any rate). It is
POV-Ray that is to blame -- or, rather, whoever added support for
signatures in UTF-8 files, because while they did add such support for
the scene file proper, they completely forgot to do the same for include
files.

The overhauled parser will address this.


> I looked at various Wikipedia pages ( "US-ASCII", Windows WordPad app", "UTF-8",
> "Comparison of Unicode encodings" ), but I *still* don't have a full grasp of
> the facts:
> 
> "WordPad for Windows XP [and later?] added full Unicode support, enabling
> WordPad to support multiple languages..."

Yup. It's a pity they've apparently thrown part of this support
overboard again between Windows 7 and Windows 10 (saving as Big-Endian
UTF-16 and saving as UTF-8).

> "[Windows] files saved as Unicode text are encoded as UTF-16 LE.  [not UTF-8,

Nope. It's not UTF-16LE (that would be a file using UTF-16 little-endian
encoding /without/ a signature BOM), but UTF-16 with Little Endian byte
order (carrying a signature BOM).

> unless that is specified] ...Such [Unicode] files normally begin with Byte Order
> Mark (BOM), which communicates the endianness of the file content. Although
> UTF-8 does not suffer from endianness problems, many Windows programs --i.e.
> Notepad-- prepend the contents of UTF-8-encoded files with BOM, to differentiate
> UTF-8 encoding from other 8-bit encodings." [Other 8-bit encodings meaning
> "plain ACSII"?]

No, "other 8-bit encodings" means /any/ encodings where each character
is encoded as a sequence of one or more 8-bit /code units/.

Many such encodings are /compatible/ with ASCII, in that they are
indistinguishable from plain ASCII if the text uses only the 128
printable and control characters from the ASCII character set. However,
such encodings (commonly referred to as "extended ASCII") are legion,
differing in how the remaining 128 code unit values (80 through FF) are
interpreted.

There are also 8-bit encodings that are incompatible even with plain
ASCII, the most important examples still in use these days certainly
being EBCDIC and its extensions.

Post a reply to this message

clipka <ano### [at] anonymousorg> wrote:
> Am 25.06.2018 um 11:41 schrieb Kenneth:
>
> > Rich Text Format (.rtf)-- one of Microsoft's own file-types
>
> Entirely different thing: This is not just a character encoding, but an
> entirely different file format.

Yeah, and I even tried #including such a file, just to see what would happen:
After the fatal error, the file contents actually showed up-- a bunch of
Microsoft 'rtf' gibberish as a header.

>
> The UCS specification explicitly and indiscriminately allows for UTF-8
> files both with and without a signature. The Unicode specification is
> more partial in that it discourages the presence of a signature, but
> still allows it.
>
> If present, the signature is (hex) `EF BB BF`...

.... so I assume that the error message I see (the "illegal ef value") is the
first part of that hex code. It appears to be, anyway.
>
> So Wordpad is without fault here (these days, at any rate). It is
> POV-Ray that is to blame -- or, rather, whoever added support for
> signatures in UTF-8 files, because while they did add such support for
> the scene file proper, they completely forgot to do the same for include
> files.
>

That's a surprise! Thanks for clarifying. And I was all set to write a
multi-page screed to Mr. Microsoft, detailing my grievances with NOTEPAD :-P

So I assume that NO one has had any luck trying to #include a UTF-8 -encoded
file, regardless of the text-editing app that was used.

Just from my own research into all this stuff, it seems that dealing with text
encoding is a real headache for software developers. What a complex mess.

Post a reply to this message

On 25/06/2018 14:53, Kenneth wrote:
> Stephen <mca### [at] aolcom> wrote:
>>
>> Have you come across Notepad ++ ?
>> httos://noteoadplusplus.oro/
>>
> 
> Ya know, I actually *did* have that app at one time-- on a hard drive that
> failed-- but then I completely forgot about it! Thanks for the memory boost ;-)

Fish oils help. ;-)

If you use XML files. You might find that XML Notepad is not bad.
https://www.microsoft.com/en-gb/download/details.aspx?id=7973



> I'll download it again. (I never expected Windows' own long-time 'core' apps to
> have such a basic text-encoding flaw; apparently neither NOTEPAD nor WORDPAD
> conform to the UTF-8 published standards, as far as I can tell. Maybe the
> standards came later(!)
> 
> 

If at all, at all. :-)

-- 

Regards
     Stephen

Post a reply to this message

Am 25.06.2018 um 17:12 schrieb Kenneth:

>> If present, the signature is (hex) `EF BB BF`...
> 
> ..... so I assume that the error message I see (the "illegal ef value") is the
> first part of that hex code. It appears to be, anyway.

Exactly.

>> So Wordpad is without fault here (these days, at any rate). It is
>> POV-Ray that is to blame -- or, rather, whoever added support for
>> signatures in UTF-8 files, because while they did add such support for
>> the scene file proper, they completely forgot to do the same for include
>> files.
...
> So I assume that NO one has had any luck trying to #include a UTF-8 -encoded
> file, regardless of the text-editing app that was used.

UTF-8 encoded files /without/ a signature are fine(*). Unfortunately,
Notepad and WordPad can't write those.

(* For certain definitions of "fine", that is; the `global_settings {
charset FOO }` mechanism isn't really useful for mixing files with
different encodings. Also, the editor component of POV-Ray for Windows
doesn't do UTF-8.)

> Just from my own research into all this stuff, it seems that dealing with text
> encoding is a real headache for software developers. What a complex mess.

It's gotten better. The apparent increase in brain-wrecking is just due
to the fact that nowadays it has become worth /trying/. Historically,
the only way to stay sane in a small software project was to just
pretend that there wasn't such a thing as different file encodings,
hoping that the users could work around the fallout. Now proper handling
of text encoding can be as simple as supporting UTF-8 with and without
signature - no need to even worry about plain ASCII, being merely a
special case of UTF-8 without signature.

It is only when you want to implement /some/ support for legacy
encodings, and do it in a clean way, that things still get a bit tricky.

In POV-Ray for Windows, I consider it pretty evident that besides ASCII
and UTF-8 the parser should also support whatever encoding the editor
module is effectively using. On my machine that's Windows-1252, but I
wouldn't be surprised if that depended on the operating system's locale
settings.

Post a reply to this message

clipka <ano### [at] anonymousorg> wrote:
> Am 25.06.2018 um 17:12 schrieb Kenneth:

> > So I assume that NO one has had any luck trying to #include a UTF-8 -encoded
> > file, regardless of the text-editing app that was used.
>
> UTF-8 encoded files /without/ a signature are fine(*). Unfortunately,
> Notepad and WordPad can't write those.

Ah yes, of course. (I did see that tidbit of information in my research, but it
didn't sink in.) All clear now.
>
> (* For certain definitions of "fine", that is; the `global_settings {
> charset FOO }` mechanism isn't really useful for mixing files with
> different encodings. Also, the editor component of POV-Ray for Windows
> doesn't do UTF-8.)
[snip]
> Now proper handling
> of text encoding can be as simple as supporting UTF-8 with and without
> signature - no need to even worry about plain ASCII, being merely a
> special case of UTF-8 without signature.
>

Going forward:
When you have finished your work on restructuring the parser, and someone wants
to write an #include file using UTF-8 encoding (with or without a BOM): Which of
the following two contructs is the proper way to code the scene file/#include
file combo:

A)
Scene file:
global_settings{... charset utf8}
#include "MY FILE.txt" // encoded as UTF-8 but with no charset keyword

OR B):
scene file:
global_settings{...) // no charset keyword
#include "MY FILE.txt" // encoded as UTF-8 and with its *own*
                       // global_settings{charset utf8}

I'm still a bit confused as to which is correct-- although B) looks like the
logical choice(?). The documentation about 'charset' seems to imply this.

Post a reply to this message

Am 26.06.2018 um 05:12 schrieb Kenneth:

> Going forward:
> When you have finished your work on restructuring the parser, and someone wants
> to write an #include file using UTF-8 encoding (with or without a BOM): Which of
> the following two contructs is the proper way to code the scene file/#include
> file combo:
> 
> A)
> Scene file:
> global_settings{... charset utf8}
> #include "MY FILE.txt" // encoded as UTF-8 but with no charset keyword
> 
> OR B):
> scene file:
> global_settings{...) // no charset keyword
> #include "MY FILE.txt" // encoded as UTF-8 and with its *own*
>                        // global_settings{charset utf8}
> 
> I'm still a bit confused as to which is correct-- although B) looks like the
> logical choice(?). The documentation about 'charset' seems to imply this.

When I have finished my work?

Probably neither. The `global_settings { charset FOO }` mechanism isn't
really ideal, and I'm pretty sure I'll be deprecating it and introducing
something different, possibly along the following lines:

(1) A signature-based mechanism to auto-detect UTF-8 with signature, and
maybe also UTF-16 and/or UTF-32 (either endian variant).

(2) A `#charset STRING_LITERAL` directive to explicitly specify the
encoding on a per-file basis. This setting would explicitly apply only
to the respective file itself, and would probably have to appear at the
top of the file (right alongside the initial `#version` directive).

(3a.1) An INI setting `Charset_Autodetect=BOOL` to specify whether
POV-Ray should attempt to auto-detect UTF-8 without signature (and maybe
certain other encodings) based on the first non-ASCII byte sequence in
the file.

(3a.2) An INI setting `Charset_Default=STRING` to specify what character
set should be presumed for files that have neither a signature, nor a
`#charset` statement, nor can be recognized based on the first non-ASCII
byte sequence.

-or-

(3b) An INI setting `Charset_Autodetect=STRING_LIST` to specify a list
of character sets, in order of descending preference, to try to
auto-detect based on the first non-ASCII byte sequence in the file.

Post a reply to this message

clipka <ano### [at] anonymousorg> wrote:
> Am 26.06.2018 um 05:12 schrieb Kenneth:
>
> > Going forward:
> > When you have finished your work on restructuring the parser, and someone wants
> > to write an #include file using UTF-8 encoding (with or without a BOM): Which of
> > the following two contructs is the proper way to code the scene file/#include
> > file combo:
> >
> > A)
> > Scene file:
> > global_settings{... charset utf8}
> > #include "MY FILE.txt" // encoded as UTF-8 but with no charset keyword
> >
> > OR B):
> > scene file:
> > global_settings{...) // no charset keyword
> > #include "MY FILE.txt" // encoded as UTF-8 and with its *own*
> >                        // global_settings{charset utf8}
> >
> > I'm still a bit confused as to which is correct-- although B) looks like the
> > logical choice(?). The documentation about 'charset' seems to imply this.
>
> When I have finished my work?
>
> Probably neither. The `global_settings { charset FOO }` mechanism isn't
> really ideal, and I'm pretty sure I'll be deprecating it and introducing
> something different, possibly along the following lines:

Well, the idea of it being inside a file was really simple: To have the encoding
of a file's strings inside the file.

Of course, the more meaningful question - nowadays that even Windows 10 Notepad
supports UTF-8 properly - is if there is any non-legacy (aka Windows editor)
reason not to require input to be UTF-8. It is extremely unlikely that anything
will replace UTF-8 any time soon...

Post a reply to this message

On 28/06/2018 18:26, Thorsten Froehlich wrote:
> It is extremely unlikely that anything
> will replace UTF-8 any time soon...

Who in their right mind would ever need more than 640k of ram?

;-)

-- 

Regards
     Stephen

Post a reply to this message

Stephen <mca### [at] aolcom> wrote:
> On 28/06/2018 18:26, Thorsten Froehlich wrote:
> > It is extremely unlikely that anything
> > will replace UTF-8 any time soon...
>
> Who in their right mind would ever need more than 640k of ram?
>
> ;-)

ROFL - at least designers of standards got smarter and Unicode is pretty
extensible, though the focus seems to be on more and more emoticons...

Post a reply to this message

Am 28.06.2018 um 19:26 schrieb Thorsten Froehlich:

>> Probably neither. The `global_settings { charset FOO }` mechanism isn't
>> really ideal, and I'm pretty sure I'll be deprecating it and introducing
>> something different, possibly along the following lines:
> 
> Well, the idea of it being inside a file was really simple: To have the encoding
> of a file's strings inside the file.

The problem there is that the setting in the file also governs the
presumed encoding of strings inside /other/ files, namely include files.


> Of course, the more meaningful question - nowadays that even Windows 10 Notepad
> supports UTF-8 properly - is if there is any non-legacy (aka Windows editor)
> reason not to require input to be UTF-8. It is extremely unlikely that anything
> will replace UTF-8 any time soon...

I pretty much agree there. But as long as we're rolling out legacy tools
(aka Windows editor) ourselves, we ought to support it properly.

Post a reply to this message

On 29/06/2018 13:10, Thorsten Froehlich wrote:
> Stephen <mca### [at] aolcom> wrote:
>> On 28/06/2018 18:26, Thorsten Froehlich wrote:
>>> It is extremely unlikely that anything
>>> will replace UTF-8 any time soon...
>>
>> Who in their right mind would ever need more than 640k of ram?
>>
>> ;-)
> 
> ROFL - at least designers of standards got smarter and Unicode is pretty
> extensible, though the focus seems to be on more and more emoticons...
> 
> 
> 
> 
I could not resist it. It is a classic (rebuff).

I think that in English, emoticons are not a bad idea*. As English can 
be quite ambiguous.
Warp seldom used them and I was always misunderstanding his meaning. :-(

For instance. Replying "interesting". Does not mean "It is interesting" 
but "Not worth perusing, it is not a good idea".


*
I don't mean crying cats and the like.


-- 

Regards
     Stephen

Post a reply to this message