scott wrote:

> Do you need to have some sort of grammar rules though before you can 
> tokenise?

Yes, definitely.

> I mean what if you have the string "-3*(4-3e-2)+5*-3", the 
> tokeniser (if I understand right) needs to correctly decide how to 
> interpret those minus signs, needing some rules about what is surrouding 
> it etc.  What would be the best way to convert an ASCII string like that 
> into a list of tokens?

Ah yes, the old "is it unary minus or binary minus?" question.

In this case, you'd probably let minus be a token by itself, and let the 
parser decide whether it's unary or binary based on the context when it 
builds the parse tree.

Post a reply to this message