Lexing

⁠

Lexing, or lexical analysis, or tokenisation, is basically taking the character stream and producing a token stream. A token has the following attributes:

type: Type

id: String

lineNo: int

columnNo: int

children: List<Token>

Gallery

Want to print your doc?
This is not the way.

Try clicking the ⋯ next to your doc name or using a keyboard shortcut (

CtrlP

) instead.