Skip to content
Insidion Engine
  • Pages
    • Insidion Engine
    • Introduction
    • Overview
    • Configuring the Engine
    • Scanning
    • Lexing
    • Parsing
    • Compiling
    • Executing
    • Debugging
Insidion Engine
/
Lexing
Share
Explore

Lexing

⁠
Lexing, or lexical analysis, or tokenisation, is basically taking the character stream and producing a token stream. A token has the following attributes:
type: Type
id: String
lineNo: int
columnNo: int
children: List<Token>

Gallery
 
Want to print your doc?
This is not the way.
Try clicking the ··· in the right corner or using a keyboard shortcut (
CtrlP
) instead.