Lucene修改停用分词器添加新的需要停用的单词

最新推荐文章于 2022-09-27 10:21:44 发布

庸医2048

最新推荐文章于 2022-09-27 10:21:44 发布

阅读量1.2k

点赞数 1

CC 4.0 BY-SA版权

分类专栏： Lucence 爬虫

本文链接：https://blog.youkuaiyun.com/qq_20120669/article/details/53861654

爬虫同时被 2 个专栏收录

12 篇文章

订阅专栏

Lucence

3 篇文章

订阅专栏

本文介绍如何在Lucene中创建自定义分词器(MyStopAnalyzer)，并演示了如何通过该分词器过滤停用词及显示TokenStream中的词汇信息。

代码：

第一步：创建自己的分词器

public final class MyStopAnalyzer extends Analyzer {   //继承Analyzer这个类
    private Set stops;
    public  MyStopAnalyzer(String[] sws) {   //构造方法
        stops=StopFilter.makeStopSet(Version.LUCENE_35,sws,true);
    }

    public  final  TokenStream tokenStream(String s, Reader reader) {
        return new StopFilter(Version.LUCENE_35,       //过滤停用词
                new LowerCaseFilter(Version.LUCENE_35,        //过滤成小写  
                        new LetterTokenizer(Version.LUCENE_35,reader)),stops);  //转换成词汇
    }
}

第二步：把读出的TokenStream显示出来。

package com.lk;
import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
import org.apache.lucene.analysis.tokenattributes.OffsetAttribute;
import org.apache.lucene.analysis.tokenattributes.PositionIncrementAttribute;
import org.apache.lucene.analysis.tokenattributes.TypeAttribute;

import java.io.IOException;
import java.io.StringReader;

/**
 * Created by LK on 2016/12/24.
 */
public  class AnalyzerUtils {

    public static  void  displayAllTokenInfo(String str, Analyzer a) {
        TokenStream stream = a.tokenStream("content",new StringReader(str)) ;
        PositionIncrementAttribute pia= stream.addAttribute(PositionIncrementAttribute.class);   // 保存词与词之间的位置增量
        OffsetAttribute oa = stream.addAttribute(OffsetAttribute.class);   // 保存各个词汇之间的偏移量 
        CharTermAttribute cta =stream.addAttribute(CharTermAttribute.class);  // 词汇属性 保存相应词汇
        TypeAttribute  ta = stream.addAttribute(TypeAttribute.class);  //类型属性
        try {
            for (;stream.incrementToken();){
                System.out.println(pia.getPositionIncrement()+":");
                System.out.println(cta+"["+oa.startOffset()+"-"+oa.endOffset()+"]-->"+ta.type()+"\n");
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

第三步：进行测试

public class TestMyAnalyzer {
    @Test
    public void test001() {
        Analyzer a1 = new MyStopAnalyzer(new String[]{"I", "are", "you" ,"me","is"}); //定义自己要停用的词
        String str = "how are you  my name is lk  thanks you  i hope you like me";
        AnalyzerUtils.displayAllTokenInfo(str,a1);
    }
}

第四步：测试结果

1:
how[0-3]-->word

3:
my[13-15]-->word

1:
name[16-20]-->word

2:
lk[24-26]-->word

1:
thanks[28-34]-->word

3:
hope[42-46]-->word

2:
like[51-55]-->word

把StopAnalyzer的停用词加进去：

package com.lk;
import org.apache.lucene.analysis.*;
import org.apache.lucene.util.Version;

import java.io.Reader;
import java.util.Set;

/**
 * Created by LK on 2016/12/24.
 */
public final class MyStopAnalyzer extends Analyzer {
    private Set stops;
    public  MyStopAnalyzer(String[] sws) {
        stops=StopFilter.makeStopSet(Version.LUCENE_35,sws,true);
        stops.addAll(StopAnalyzer.ENGLISH_STOP_WORDS_SET);//加入进去
    }
    public  MyStopAnalyzer() {
        stops=StopAnalyzer.ENGLISH_STOP_WORDS_SET;
    }

    public  final  TokenStream tokenStream(String s, Reader reader) {
        return new StopFilter(Version.LUCENE_35,
                new LowerCaseFilter(Version.LUCENE_35,
                        new LetterTokenizer(Version.LUCENE_35,reader)),stops);
    }
}