> 1. A "tokeniser" takes the input string and splits it into "tokens",
> possibly decorating them slightly. So now you have a flat stream of tokens
> instead of just characters.
Do you need to have some sort of grammar rules though before you can
tokenise? I mean what if you have the string "-3*(4-3e-2)+5*-3", the
tokeniser (if I understand right) needs to correctly decide how to interpret
those minus signs, needing some rules about what is surrouding it etc. What
would be the best way to convert an ASCII string like that into a list of
tokens?
Post a reply to this message
|