> 1. A "tokeniser" takes the input string and splits it into "tokens", 
> possibly decorating them slightly. So now you have a flat stream of tokens 
> instead of just characters.

Do you need to have some sort of grammar rules though before you can 
tokenise?  I mean what if you have the string "-3*(4-3e-2)+5*-3", the 
tokeniser (if I understand right) needs to correctly decide how to interpret 
those minus signs, needing some rules about what is surrouding it etc.  What 
would be the best way to convert an ASCII string like that into a list of 
tokens?

Post a reply to this message