> scott wrote:
> 
>> Do you need to have some sort of grammar rules though before you can 
>> tokenise?
> 
> Yes, definitely.

To see this, consider that in many programming languages, "foo_bar" is a 
single identifier. However, in TeX source code, "_" is [usually] a 
command name and hence should be parsed as a seperate token.

So what constitutes a "token" completely depends on exactly what you're 
trying to tokenise/parse.

Post a reply to this message