Tokens
Three Java programmers walk into a bar. The bartender creates a BeverageOrderProcessorFactory instance, uses it to create a BeverageOrderProcessor, then notifies each of the programmers of his desire to observe BeverageDesire events, handing them the BeverageOrderProcessor instance.
compiler Directory
As we learnt from the overview just now, the tokeniser is part of the compiler, so let’s set up the directory as such:
🗀 grammar
🗎 ebnf_definition.md
🗎 grammar.ebnf
🗀 runtime
🗀 compiler
🗎 pubspec.yaml
🗀 lib
🗎 main.dart
🗎 tokeniser.dart
🗎 error_handler.dart
🗀 vscode
🗎 package.json
🗀 syntaxes
🗎 mochaccino.tmGrammar.json
🗎 LICENSE
🗎 README.md
Under the compiler folder, you can set up a package for the whole compiler, as we did for ours, in whichever language you’re using.
Now, a quick note before we start writing the tokeniser is to make use of scripts wherever possible to write repetitive code. Occasionally, throughout the building of the compiler, we will come across code structures which are repetitive with slight differences. In such cases, it would be better to use scripts to reduce the chance of human error. Furthermore, if you ever need to rewrite those lines of code, a few small tweaks to the code generation script would be enough.
The following are some scripts we’ve used to generate Dart code for the tokeniser. In the succeeding chapters, you'll understand what the following scripts are used for.
Scripts
Keywords Map
void main() {
String src = """kw_ok = "ok";
kw_notok = "notok";
kw_named = "named";
kw_package = "package";
kw_include = "include";
kw_from = "from";
kw_as = "as";
kw_guarded = "guarded";
kw_except = "except";
kw_var = "var";
kw_return = "return";
kw_async = "async";
kw_static = "static";
kw_module = "module":
kw_struct = "struct";
kw_func = "func";
kw_dock = "dock";
kw_extends = "extends";
kw_implements = "implements";
kw_if = "if";
kw_elif = "elif";
kw_else = "else";
kw_for = "for";
kw_while = "while";
kw_prop = "prop";
kw_get = "get";
kw_set = "set";""";
List<String> f = src.trim().split('\n').map((String line) {
return "'" + line.split(' ').last.replaceAll(';', '').replaceAll('"', '') + "'" + ":" + "TokenType.${line.split(' ').last.replaceAll(';', '').replaceAll('"', '').toUpperCase()}";
}).toList();
print("{" + f.join(',') + "}");
}
class A implements B
void main() {
String src = """
expression = binary_exp | unary_exp | comparative_exp | logical_exp | group | value;
group = "(" expression ")";
binary_exp = expression arithmetic_op expression;
unary_exp = (unary_prefix (unary_exp | value)) | ((unary_exp | value) unary_postfix);
comparative_exp = expression comparative_op expression;
arithmetic_op = "+" | "-" | "*" | "/" | "%";
logical_op = "||" | "&&";
comparative_op = "==" | "!=" | "<=" | ">=";
unary_prefix = "!" | "-";
unary_postfix = "++" | "--";
""";
src.trim().split('\n').forEach((String line) {
print("class ${line.split(' ')[0].split('_').map((String t) => t.capitalize()).toList().join('')} implements Expression {}");
});
}
extension StringExtension on String {
String capitalize() {
return "${this[0].toUpperCase()}${this.substring(1).toLowerCase()}";
}
}
Want to print your doc?
This is not the way.
This is not the way.
Try clicking the ⋯ next to your doc name or using a keyboard shortcut (
CtrlP
) instead.