FMM算法 分词

FMM算法的最简单思想是使用贪心算法向前找n个，如果这n个组成的词在词典中出现，就ok，如果没有出现，那么找n-1个...然后继续下去。假如n个词在词典中出现，那么从n+1位置继续找下去，知道句子结束。

标签： Python

代码片段(3)

[代码][Python]代码

01

importre

02	defPreProcess(sentence,edcode="utf-8"):

03	sentence=sentence.decode(edcode)

04	sentence=re.sub(u"[。，,！……!《》<>\"':：？\?、\|“”‘’；]"," ",sentence)

05	returnsentence

06

07	defFMM(sentence,diction,result=[],maxwordLength=4,edcode="utf-8"):

08

i=0

09	sentence=PreProcess(sentence,edcode)

10	length=len(sentence)

11	whilei < length:

12	# find the ascii word

13

tempi=i

14	tok=sentence[i:i+1]

15	whilere.search("[0-9A-Za-z\-\+#@_\.]{1}",tok)<>None:

16

i=i+1

17	tok=sentence[i:i+1]

18	ifi-tempi>0:

19	result.append(sentence[tempi:i].lower().encode(edcode))

20	# find chinese word

21	left=len(sentence[i:])

22	ifleft==1:

23	"""go to 4 step over the FMM"""

24	"""should we add the last one? Yes, if not blank"""

25	ifsentence[i:] <>" ":

26	result.append(sentence[i:].encode(edcode))

27	returnresult

28	m=min(left,maxwordLength)

29

30	forjinxrange(m,0,-1):

31	leftword=sentence[i:j+i].encode(edcode)

32	# print leftword.decode(edcode)

33	ifLookUp(leftword,diction):

34	# find the left word in dictionary

35	# it's the right one

36

i=j+i

37	result.append(leftword)

38

break

39

elifj==1:

40	"""only one word, add into result, if not blank"""

41	ifleftword.decode(edcode) <>" ":

42	result.append(leftword)

43

i=i+1

44

else:

45

continue

46	returnresult

47	defLookUp(word,dictionary):

48	ifdictionary.has_key(word):

49	returnTrue

50	returnFalse

51	defConvertGBKtoUTF(sentence):

52	returnsentence.decode('gbk').encode('utf-8')

[代码][Python]代码

01	dictions={}

02	dictions["ab"]=1

03	dictions["cd"]=2

04	dictions["abc"]=1

05	dictions["ss"]=1

06	dictions[ConvertGBKtoUTF("好的")]=1

07	dictions[ConvertGBKtoUTF("真的")]=1

08	sentence="asdfa好的是这样吗vasdiw呀真的daf dasfiw asid是吗？"

09	s=FMM(ConvertGBKtoUTF(sentence),dictions)

10

foriins:

11	printi.decode("utf-8")

[代码][Python]代码

1	test=open("test.txt","r")

2	forlineintest:

3	s=FMM(CovertGBKtoUTF(line),dictions)

4

foriins:

5	printi.decode("utf-8")