Token&Tokenizer

最新推荐文章于 2025-06-19 09:13:54 发布

原创最新推荐文章于 2025-06-19 09:13:54 发布 · 1.6k 阅读

2 ·

CC 4.0 BY-SA版权

读书笔记专栏收录该内容

8 篇文章

订阅专栏

本文深入探讨了Esprima作为ECMAScript词法语法解析器的角色，如何将JS源码转换为Token集合，以及Token如何唯一标识代码状态。同时，文章详细介绍了自定义接口IDoom3Token和IDoom3Tokenizer的设计，展示了接口暴露与类实现分离的编程理念。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Token与Tokenizer
Esprima(ECMAScript词法语法解析器)是Tokenizer，会将js源码读取，进行处理，结果是Token。js源码解析成Token的集合，Token能唯一的标出一种状态。

if(b===true){
	alert("true");
}

[
    {
        "type": "Keyword",
        "value": "if"
    },
    {
        "type": "Punctuator",
        "value": "("
    },
    {
        "type": "Identifier",
        "value": "b"
    },
    {
        "type": "Punctuator",
        "value": "==="
    },
    {
        "type": "Boolean",
        "value": "true"
    },
    {
        "type": "Punctuator",
        "value": ")"
    },
    {
        "type": "Punctuator",
        "value": "{"
    },
    {
        "type": "Identifier",
        "value": "alert"
    },
    {
        "type": "Punctuator",
        "value": "("
    },
    {
        "type": "String",
        "value": "\"true\""
    },
    {
        "type": "Punctuator",
        "value": ")"
    },
    {
        "type": "Punctuator",
        "value": ";"
    },
    {
        "type": "Punctuator",
        "value": "}"
    }
]

定义接口

export interface IDoom3Token {
    reset(): void;
    isString(str: string): boolean;
    readonly type: ETokenType;
    getString(): string;
    getFloat(): number;
    getInt(): number;
}
export interface IDoom3Tokenizer  {
    setSource(source: string): void;//设置要解析的字符串
    reset(): void;//重置当前索引为0
    getNextToken(token: IDoom3Token): boolean;//获取下一个token
}