Insidion Engine
Share
Explore

Lexing

Lexing, or lexical analysis, or tokenisation, is basically taking the character stream and producing a token stream. A token has the following attributes:
type: Typeid: StringlineNo: intcolumnNo: intchildren: List<Token>

Share
 
Want to print your doc?
This is not the way.
Try clicking the ⋯ next to your doc name or using a keyboard shortcut (
CtrlP
) instead.