计数方法应用于PTB数据集

PTB数据集

内容如下：

一行保存一个句子；将稀有单词替换成特殊字符 < unk > ；将具体的数字替换 成“N”

 we 're talking about years ago before anyone heard of asbestos having any questionable properties 
 there is no asbestos in our products now 
 neither <unk> nor the researchers who studied the workers were aware of any research on smokers of the kent cigarettes 
 we have no useful information on whether users are at risk said james a. <unk> of boston 's <unk> cancer institute 
 dr. <unk> led a team of researchers from the national cancer institute and the medical schools of harvard university and boston university 

ptb.py

使用PTB数据集：

由下面这句话，可知用PTB数据集时候，是把所有句子首尾连接了。