Ron Parker wrote:

> On Mon, 13 Sep 1999 01:04:10 -0700, Jon A. Cruz wrote:
> >There was some talk of wanting to at least allow different languages in the
> >comments. The main problem is that the text display is fairly isolated, whereas
> >the parsing is more of all over.
>
> The part you'd want to change, though, is mostly a smallish case statement
> in tokenize.c.  At the moment, it treats a-z, A-Z, and _ as the beginning of
> a symbol.  You'd want to make it treat anything with the high bit set as a
> symbol as well.  You'd also need to modify the Read_Symbol code to recognize
> and parse correctly characters with the high bit set.  This would allow
> high-bit characters to be used inside of declared symbols, which includes
> macro names and arguments.  There'd probably also be a few small modifications
> needed to some error-reporting code, unless you're comfortable with sending
> UTF-8 to the error stream in the event of an undefined symbol or the like.
>
> Believe it or not, the tokenizer can already deal with high-bit UTF-8 characters
> inside of comments and literal strings.  The built-in editor in the Windows
> version can't, of course, but I'd expect someone using UTF-8 to use something
> more suited to Unicode, such as Unipad.
>
> Parsing UCS-2 or UCS-4 would be a lot more difficult, of course.

That's a good start, thanks. But here are few more problems than that :-(

I'd need to go through things to make sure that the "right" thing is done for such
as chr(), asc(), etc. There are several different logical approaches to the
problem, but backwards compatibility is one of the biggies. Also need to handle
invalid UTF-8, etc.

Guess I'd better get on it.

--
"My new computer's got the clocks, it rocks
But it was obsolete before I opened the box" - W.A.Y.

Post a reply to this message