|
![](/i/fill.gif) |
> scott wrote:
>
>> Do you need to have some sort of grammar rules though before you can
>> tokenise?
>
> Yes, definitely.
To see this, consider that in many programming languages, "foo_bar" is a
single identifier. However, in TeX source code, "_" is [usually] a
command name and hence should be parsed as a seperate token.
So what constitutes a "token" completely depends on exactly what you're
trying to tokenise/parse.
Post a reply to this message
|
![](/i/fill.gif) |