上一节完成的词法解析器存在一些明显问题,例如在识别数字时,面对字符串“123abc",它会识别为两者情况,分别为NUMBER:123,然后是IDENTIFIER:anc,实际上这样的字符串应该被认为是语法错误。另外我们还有一些”连号“操作符没有处理,例如“==, !=, – ,++"等。
本节我们将在上一节的基础上对词法解析器做进一步完善。首先在token.go里面添加新的操作符定义:
···
const (
ILLEGAL = “ILLEGAL”
EOF = “EOF”
IDENTIFIER = “IDENTIFIER” //变量类型对应的归类
NUMBER = “NUMBER” //数值类型对应的归类
ASSIGN = “=” //赋值操作符
PLUS = “+” //加号操作符
LPAR = “(”
RPAR = “)”
LBRACE = “{”
RBRACE = “}”
COMMA = “,”
COLON = “:”
DEF = “def” //关键字
INT = “int”
RETURN = “return”
ASSERT = “assert”
AND = “and”
//第三节添加
TRUE = "True"
FALSE = "False"
IF = "if"
ELSE = "else"
EQUAL = "=="
NOTEQUAL = "!="
GREATEREQUAL = ">="
LESSEQUAL = "<="
MINUS = "-"
BANG = "!"
ASTERISK = "*"
SLASH = "/"
LT = "<"
GT = ">"
//第三节添加
)
var keywords = map[string]TokenType {
“def” : DEF,
“int” : INT,
“return” : RETURN,
“assert” : ASSERT,
“and” : AND,
//第三节添加