词法分析器

编译原理课老师留的东西，一个词法分析器（感觉各路大学课设都会用到吧），这里贴出其python代码及算法分析过程，同时用tkinter设计出gui，再用pyinstaller打包的全过程解说，算是一个比较全面而有趣的python demo了2333

先上基础效果图：

即可以打开任意文本文件进行词法分析

接下来记录整个实现过程：

一、环境配置

我这里用的是python3.5版本。其实只是用python3.5正常开发就可以，我这里记录一下我辛苦的心路历程（可不看，感兴趣的可以在机器学习入门中看看anaconda配置，超简单方便），我先下了python3.5，又下了anaconda3(自带python3.6),其实个人理解anaconda就相当于是一个大环境，大盒子，里面可以生成好多像tensorflow这样的环境（像房间一样），anaconda里pip install pythoninstaller是不存在的555，所以我把evens里的tensorflow里的python作为运行环境（这里tensorflow里就pip install pyinstaller了）

-----------------图快的朋友，下一个python3.5pip3 install pyinstaller就行233

二、有关这个词法分析器

大体流程：先打开一个文件，然后开始读入里面的所有字符，读入字母时如果在读入字母的基础上不断读入字母和数字那么保存当前所输入字符到当前ch中，如果此时读入的字符不是字母和数字，那么退出当前状态，将已有字符串保存为标识符后初始化，此时程序会回到开始的选择状态；如果读入的是数字，并不断读入数字，此时所输入字符就会不断保存到程序的string中，如果此时读入的非数字将已有字符串保存为无符号整数后初始化退后到S状态；；如果此时读入;说明当句结束，保存为分界符后退回到S状态；如果读入=，，保存为运算符后输出，将当前字符串初始化，，退回S状态；如果当下读入并非指定的几种字符，那么归为其他类保存在字符串中输出，初始化后返回到S状态

以下为流程图：

代码：

# -*- coding: utf8 -*-
class DFA:
    file_object = ''#文件句柄
    line_number = 0 #记录行号
    state = 0 #状态
    ResWord = ['int','if','then','else','end','repeat','until','read','write']#保留字
    error_message = []#保存错误信息,存储元组,元组第一个参数是行号,第二个参数是错误字符
    annotate_message = []#注释信息,存储元组,元组第一个参数是行号,第二个参数是注释
    char_message = []#识别的字符串,存储元组,元组第一个参数是类型,第二个参数是该字符串

    def __init__(self,file_name):
        self.file_object = file_name
        self.state = 0
        self.line_number = 0
        self.error_message = []
        self.annotate_message = []
        self.char_message = []
    def Start_convert(self):
        for line in self.file_object:#一行行的处理
            line = line.strip('\n')#去除换行fu
            self.line_number += 1#没处理一行行号加一
            line_length = len(line)
            i = 0
            string = ''#存储一个字符串
            while i < line_length:
                ch = line[i]#读取该行的一个字符
                i += 1
                if self.state == 0:#初始状态
                    string = ch
                    if ch.isalpha():
                        self.state = 1
                    elif ch.isdigit():
                        self.state = 3
                    elif ch == '+':
                        self.state = 5
                    elif ch == '-':
                        self.state = 9
                    elif ch == '*':
                        self.state = 13
                    elif ch == '/':
                        self.state = 16
                    elif ch == '=':
                        self.state = 20
                        i -= 1
                    elif ch == '<':
                        self.state = 21
                        i -= 1
                    elif ch == '{':
                        self.state = 22
                        i -= 1
                    elif ch == '}':
                        self.state = 23
                        i -= 1
                    elif ch == ';':
                        i -= 1
                        self.state = 24
                    elif ch.isspace():
                        self.state = 25
                    else:
                        self.state = 26#不可识别状态
                        i -= 1
                elif self.state == 1:#判断字母数字
                    while ch.isalpha() or ch.isdigit():
                        string += ch
                        if i < line_length:
                            ch = line[i]
                            i += 1
                        else:
                            break
                    self.state = 2
                    i -= 2#回退2个字符
                elif self.state == 2:
                    if string in self.ResWord:
                        content = '(关键字,' + string + ')'
                    else:
                        content = '(标识符,' + string + ')'
                    #print content
                    self.char_message.append(content)
                    string = ''#回到初始情况
                    self.state = 0#回到状态0
                elif self.state == 3:
                    while ch.isdigit():
                        string += ch
                        if i < line_length:
                            ch = line[i]
                            i += 1
                        else:
                            break
                    self.state = 4
                    i -= 2#回退2个字符
                elif self.state == 4:
                    content = '(数字,' + string + ')'
                    self.char_message.append(content)
                    #print string
                    string = ''#回到初始情况
                    self.state = 0#回到状态0
                elif self.state == 5:
                    if ch == '+':
                        self.state = 6
                        i -= 1
                    elif ch == '=':
                        self.state = 7
                        i -= 1
                    else:
                        self.state = 8
                        i -= 2
                elif self.state == 6:#判断++
                    content = '(特殊符号,' + string + ch + ')'
                    self.char_message.append(content)
                    #print string + ch
                    string = ''#回到初始情况
                    self.state = 0#回到状态0
                elif self.state == 7:#判断+=
                    content = '(特殊符号,' + string + ch + ')'
                    self.char_message.append(content)
                    #print string + ch
                    string = ''#回到初始情况
                    self.state = 0#回到状态0
                elif self.state == 8:#判断+
                    content = '(特殊符号,' + ch + ')'
                    self.char_message.append(content)
                    #print ch
                    string = ''#回到初始情况
                    self.state = 0#回到状态0
                elif self.state == 9:
                    if ch == '-':
                        self.state = 10
                        i -= 1
                    elif ch == '=':
                        self.state = 11
                        i -= 1
                    else:
                        self.state = 12
                        i -= 2
                elif self.state == 10:
                    content = '(特殊符号,' + string + ch + ')'
                    self.char_message.append(content)
                    #print string + ch#判断--
                    string = ''#回到初始情况
                    self.state = 0#回到状态0
                elif self.state == 11:#判断-=
                    content = '(特殊符号,' + string + ch + ')'
                    self.char_message.append(content)
                    #print string + ch
                    string = ''#回到初始情况
                    self.state = 0#回到状态0
                elif self.state == 12:#判断-
                    content = '(特殊符号,' + ch + ')'
                    self.char_message.append(content)
                    #print ch
                    string = ''#回到初始情况
                    self.state = 0#回到状态0
                elif self.state == 13:
                    if ch == '=':
                        self.state = 14
                        i -= 1
                    else:
                        self.state = 15
                        i -= 2
                elif self.state == 14:#判断*=
                    content = '(特殊符号,' + string + ch + ')'
                    self.char_message.append(content)
                    #print string + ch
                    string = ''#回到初始情况
                    self.state = 0#回到状态0
                elif self.state == 15:#判断*
                    content = '(特殊符号,' + ch + ')'
                    self.char_message.append(content)
                    #print ch
                    string = ''#回到初始情况
                    self.state = 0#回到状态0
                elif self.state == 16:
                    if ch == '/':
                        self.state = 17
                        i -= 1
                    elif ch == '=':
                        self.state = 18
                        i -= 1
                    else:
                        self.state = 19
                        i -= 2
                elif self.state == 17:#判断//
                    content = '(特殊符号,' + string + ch + ')'
                    self.char_message.append(content)
                    content = '(注释,'+ line[i:] +')'
                    self.annotate_message.append(content)
                    #print content
                    string = ''#回到初始情况
                    self.state = 0#回到状态0
                elif self.state == 18:#判断/=
                    content = '(特殊符号,' + string + ch + ')'
                    self.char_message.append(content)
                    #print string + ch
                    string = ''#回到初始情况
                    self.state = 0#回到状态0
                elif self.state == 19:#判断/
                    content = '(特殊符号,' + ch + ')'
                    self.char_message.append(content)
                    #print ch
                    string = ''#回到初始情况
                    self.state = 0#回到状态0
                elif self.state == 20:
                    content = '(特殊符号,=)'
                    self.char_message.append(content)
                    #print '='
                    self.state = 0
                    string = ''
                elif self.state == 21:
                    content = '(特殊符号,<)'
                    self.char_message.append(content)
                    #print '<'
                    self.state = 0
                    string = ''
                elif self.state == 22:
                    content = '(特殊符号,{)'
                    self.char_message.append(content)
                    #print '{'
                    self.state = 0
                    string = ''
                elif self.state == 23:
                    content = '(特殊符号,})'
                    self.char_message.append(content)
                    #print '}'
                    self.state = 0
                    string = ''
                elif self.state == 24:
                    content = '(特殊符号,;)'
                    self.char_message.append(content)
                    #print ';'
                    self.state = 0
                    string = ''
                elif self.state == 25:
                    while ch.isspace():
                        if i < line_length:
                            ch = line[i]
                            i += 1
                        else:
                            break
                    self.state = 0
                    i -= 1
                elif self.state == 26:
                    content = '(行号:'+str(self.line_number)+',' + ch + ')'
                    self.error_message.append(content)
                    #print 'error:' + ch
                    self.state = 0
                    string = ''
        #print self.state
    def Get_error(self):#获取错误信息
        return self.error_message

    def Get_annotate(self):#获取注释信息
        return self.annotate_message

    def Get_char(self):#获取识别信息
        return self.char_message

try:
    file_object = open("ceshi.txt")
    dfa = DFA(file_object)
    dfa.Start_convert()
    content = dfa.Get_char()
    for item in content:
        print item
    content = dfa.Get_annotate()
    for item in content:
        print item
    content = dfa.Get_error()
    for item in content:
        print item
finally:  
file_object.close()

看到open(“ceshi.txt”)了吗，记得在同级目录下建一个ceshi.txt里面输入你想输入的文本hh

三、算法分析

以python为编程语言完成这个词法分析器，代码定义了一个类DFA，以构造函数形式读入指定路径下的一个测试txt文本，定义Start_convert函数对可识别成分如关键字或标识符等进行分析。Get_error函数获取错误信息, Get_annotate函数获取注释信息, Get_char函数获取识别信息

初始化：

self.state = 0//状态标志

self.line_number = 0//行号

self.error_message = []//#保存错误信息,存储元组,元组第一个参数是行号,第二个参数是错误字符

self.annotate_message = []#注释信息,存储元组,元组第一个参数是行号,第二个参数是注释

self.char_message = []#识别的字符串,存储元组,元组第一个参数是类型,第二个参数是该字符串

函数分析：

根据state决定做出不同操作：

State(判断状态)

1(字母)：进入状态1后不断扫描至结束，保存为标识符，转为状态2

3(数字)：进入状态3不断扫描到非数字，保存为无符号整数，转为状态4

5(+):进入状态5后扫描下一位进入判断状态6，7，8判断此操作符为+=或++或+

状态9~19同5

状态20~24：分情况扫描当下特殊符号是什么，只扫描一位就回归S状态

状态25：不断扫描空格，扫描至非空格字符，回归S状态

状态26：扫描至不识别字符（即其他字符保存下来）

State(输出状态)

2::判断是否是已有关键字，是的话输出为关键字，否则输出为标识符

6,7,8,10,11,12,14~24为特殊符号输出

26:输出其他字符及所在行号

四、gui编写

核心算法过程完成了，接下来只要写个界面读取文件传值给这个词法分析器(transss.py),分析后传到gui上显示就可以了

guic.py代码：(guic是文件名)

from tkinter import *
from tkinter.filedialog import askopenfilename
from PIL import Image, ImageTk
import transss


def selectPath():
    #  path_ = askopenfilename()
    path_ = ""
    path_ += askopenfilename()
    transss.wenben = path_
    path.set(path_)


def use_transss():
    transss.use_dfa()
    for item in transss.content1:
          output.insert(END, item)
    for item in transss.content2:
        output.insert(END, item)
    for item in transss.content3:
        output.insert(END, item)

root = Tk()
root.title("词法分析器")
path = StringVar()

Label(root, text="目标文本:").grid(row=0, column=0)
Entry(root, textvariable=path).grid(row=0, column=1)
Button(root, text="路径选择", command=selectPath).grid(row=0, column=2)
Button(root, text="开始识别", command=use_transss).grid(row=1, column=1)
output = Listbox(root, width=35, height=25, bg='white')
output.grid(row=2, column=1)
root.update_idletasks()
root.mainloop()

词法分析器完整代码：

transss.py: (文件名)

# -*- coding: utf8 -*-
wenben = ""
content1=[[]]
content2=[[]]
content3=[[]]
class DFA:
    file_object = ''  # 文件句柄
    line_number = 0  # 记录行号
    state = 0  # 状态
    ResWord = ['int', 'if', 'then', 'else', 'end', 'repeat', 'until', 'read', 'write']  # 保留字
    error_message = []  # 保存错误信息,存储元组,元组第一个参数是行号,第二个参数是错误字符
    annotate_message = []  # 注释信息,存储元组,元组第一个参数是行号,第二个参数是注释
    char_message = []  # 识别的字符串,存储元组,元组第一个参数是类型,第二个参数是该字符串
    def __init__(self, file_name):
        self.file_object = file_name
        self.state = 0
        self.line_number = 0
        self.error_message = []
        self.annotate_message = []
        self.char_message = []

    def Start_convert(self):
        for line in self.file_object:  # 一行行的处理
            line = line.strip('\n')  # 去除换行fu
            self.line_number += 1  # 没处理一行行号加一
            line_length = len(line)
            i = 0
            string = ''  # 存储一个字符串
            while i < line_length:
                ch = line[i]  # 读取该行的一个字符
                i += 1
                if self.state == 0:  # 初始状态
                    string = ch
                    if ch.isalpha():
                        self.state = 1
                    elif ch.isdigit():
                        self.state = 3
                    elif ch == '+':
                        self.state = 5
                    elif ch == '-':
                        self.state = 9
                    elif ch == '*':
                        self.state = 13
                    elif ch == '/':
                        self.state = 16
                    elif ch == '=':
                        self.state = 20
                        i -= 1
                    elif ch == '<':
                        self.state = 21
                        i -= 1
                    elif ch == '{':
                        self.state = 22
                        i -= 1
                    elif ch == '}':
                        self.state = 23
                        i -= 1
                    elif ch == ';':
                        i -= 1
                        self.state = 24
                    elif ch.isspace():
                        self.state = 25
                    else:
                        self.state = 26  # 不可识别状态
                        i -= 1
                elif self.state == 1:  # 判断字母数字
                    while ch.isalpha() or ch.isdigit():
                        string += ch
                        if i < line_length:
                            ch = line[i]
                            i += 1
                        else:
                            break
                    self.state = 2
                    i -= 2  # 回退2个字符
                elif self.state == 2:
                    if string in self.ResWord:
                        content = '(关键字,' + string + ')'
                    else:
                        content = '(标识符,' + string + ')'
                    # print content
                    self.char_message.append(content)
                    string = ''  # 回到初始情况
                    self.state = 0  # 回到状态0
                elif self.state == 3:
                    while ch.isdigit():
                        string += ch
                        if i < line_length:
                            ch = line[i]
                            i += 1
                        else:
                            break
                    self.state = 4
                    i -= 2  # 回退2个字符
                elif self.state == 4:
                    content = '(数字,' + string + ')'
                    self.char_message.append(content)
                    # print string
                    string = ''  # 回到初始情况
                    self.state = 0  # 回到状态0
                elif self.state == 5:
                    if ch == '+':
                        self.state = 6
                        i -= 1
                    elif ch == '=':
                        self.state = 7
                        i -= 1
                    else:
                        self.state = 8
                        i -= 2
                elif self.state == 6:  # 判断++
                    content = '(特殊符号,' + string + ch + ')'
                    self.char_message.append(content)
                    # print string + ch
                    string = ''  # 回到初始情况
                    self.state = 0  # 回到状态0
                elif self.state == 7:  # 判断+=
                    content = '(特殊符号,' + string + ch + ')'
                    self.char_message.append(content)
                    # print string + ch
                    string = ''  # 回到初始情况
                    self.state = 0  # 回到状态0
                elif self.state == 8:  # 判断+
                    content = '(特殊符号,' + ch + ')'
                    self.char_message.append(content)
                    # print ch
                    string = ''  # 回到初始情况
                    self.state = 0  # 回到状态0
                elif self.state == 9:
                    if ch == '-':
                        self.state = 10
                        i -= 1
                    elif ch == '=':
                        self.state = 11
                        i -= 1
                    else:
                        self.state = 12
                        i -= 2
                elif self.state == 10:
                    content = '(特殊符号,' + string + ch + ')'
                    self.char_message.append(content)
                    # print string + ch#判断--
                    string = ''  # 回到初始情况
                    self.state = 0  # 回到状态0
                elif self.state == 11:  # 判断-=
                    content = '(特殊符号,' + string + ch + ')'
                    self.char_message.append(content)
                    # print string + ch
                    string = ''  # 回到初始情况
                    self.state = 0  # 回到状态0
                elif self.state == 12:  # 判断-
                    content = '(特殊符号,' + ch + ')'
                    self.char_message.append(content)
                    # print ch
                    string = ''  # 回到初始情况
                    self.state = 0  # 回到状态0
                elif self.state == 13:
                    if ch == '=':
                        self.state = 14
                        i -= 1
                    else:
                        self.state = 15
                        i -= 2
                elif self.state == 14:  # 判断*=
                    content = '(特殊符号,' + string + ch + ')'
                    self.char_message.append(content)
                    # print string + ch
                    string = ''  # 回到初始情况
                    self.state = 0  # 回到状态0
                elif self.state == 15:  # 判断*
                    content = '(特殊符号,' + ch + ')'
                    self.char_message.append(content)
                    # print ch
                    string = ''  # 回到初始情况
                    self.state = 0  # 回到状态0
                elif self.state == 16:
                    if ch == '/':
                        self.state = 17
                        i -= 1
                    elif ch == '=':
                        self.state = 18
                        i -= 1
                    else:
                        self.state = 19
                        i -= 2
                elif self.state == 17:  # 判断//
                    content = '(特殊符号,' + string + ch + ')'
                    self.char_message.append(content)
                    content = '(注释,' + line[i:] + ')'
                    self.annotate_message.append(content)
                    # print content
                    string = ''  # 回到初始情况
                    self.state = 0  # 回到状态0
                elif self.state == 18:  # 判断/=
                    content = '(特殊符号,' + string + ch + ')'
                    self.char_message.append(content)
                    # print string + ch
                    string = ''  # 回到初始情况
                    self.state = 0  # 回到状态0
                elif self.state == 19:  # 判断/
                    content = '(特殊符号,' + ch + ')'
                    self.char_message.append(content)
                    # print ch
                    string = ''  # 回到初始情况
                    self.state = 0  # 回到状态0
                elif self.state == 20:
                    content = '(特殊符号,=)'
                    self.char_message.append(content)
                    # print '='
                    self.state = 0
                    string = ''
                elif self.state == 21:
                    content = '(特殊符号,<)'
                    self.char_message.append(content)
                    # print '<'
                    self.state = 0
                    string = ''
                elif self.state == 22:
                    content = '(特殊符号,{)'
                    self.char_message.append(content)
                    # print '{'
                    self.state = 0
                    string = ''
                elif self.state == 23:
                    content = '(特殊符号,})'
                    self.char_message.append(content)
                    # print '}'
                    self.state = 0
                    string = ''
                elif self.state == 24:
                    content = '(特殊符号,;)'
                    self.char_message.append(content)
                    # print ';'
                    self.state = 0
                    string = ''
                elif self.state == 25:
                    while ch.isspace():
                        if i < line_length:
                            ch = line[i]
                            i += 1
                        else:
                            break
                    self.state = 0
                    i -= 1
                elif self.state == 26:
                    content = '(行号:' + str(self.line_number) + ',' + ch + ')'
                    self.error_message.append(content)
                    # print 'error:' + ch
                    self.state = 0
                    string = ''
                    # print self.state

    def Get_error(self):  # 获取错误信息
        return self.error_message

    def Get_annotate(self):  # 获取注释信息
        return self.annotate_message

    def Get_char(self):  # 获取识别信息
        return self.char_message


def use_dfa():
    file_object = open(wenben)
    dfa = DFA(file_object)
    dfa.Start_convert()
    global content1,content2,content3
    content1 = dfa.Get_char()
    for item in content1:
        print(item)
    content2 = dfa.Get_annotate()
    for item in content2:
        print(item)
    content3 = dfa.Get_error()
    for item in content3:
        print(item)

    file_object.close()

五、打包成exe
为了方便些，可以在没有python的环境下运行，我打包了下，方法：
两个py文件都放到python3文件夹的scripts，运行命令行，进入到scripts文件夹下，pyinstaller -F guic.py(把F换成w是另一种打包方法，运行没有黑白框的,但冗赘内容较多)，标准的最好的打包命令：pyinstaller -F -w guic.py,成功后会在同目录下的dist文件夹里看到guic.exe,内心兴奋(ps:pyinstaller -F -w -i p.ico guic.py本来应该是同时把p.ico图标打包给guic.exe的不到为啥没成功，可能是pyinstaller版本问题。。。。。
至此一个小型可移植基于tkinter的词法分析器就完成啦