lucene4.2 java.lang.NullPointerException at org.apache.lucene.analysis.standard.StandardTokenizerIm

本文介绍了一个关于Lucene分词器的问题及其解决方案。问题出现在尝试使用StandardAnalyzer对字符串进行分词时,未调用TokenStream的reset方法导致空指针异常。通过在incrementToken方法前增加reset操作,成功解决了此问题。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

最近在学习Lucene时遇到了问题,先上代码:

public class AnalyzerUtil {

    public void displayToken(String str, Analyzer a) {
        try {
            TokenStream stream = a.tokenStream("content", new StringReader(str));
//            PositionIncrementAttribute pia = stream.addAttribute(PositionIncrementAttribute.class);
            CharTermAttribute cta = stream.addAttribute(CharTermAttribute.class);
//            TypeAttribute ta = stream.addAttribute(TypeAttribute.class);
            while(stream.incrementToken()) {
                System.out.println(cta);
            }
            System.out.println("---------------------------------------------");
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

public class AnalyzerTest {
    private AnalyzerUtil au = null;

    @Before
    public void before() {
        au = new AnalyzerUtil();
    }


    @Test
    public void AnalyzerTest() {
        String str = "this is my book,thank you";
        StandardAnalyzer a1 = new StandardAnalyzer(Version.LUCENE_42);
        au.displayToken(str, a1);
    }

}


运行结果:

java.lang.NullPointerException
	at org.apache.lucene.analysis.standard.StandardTokenizerImpl.zzRefill(StandardTokenizerImpl.java:923)
	at org.apache.lucene.analysis.standard.StandardTokenizerImpl.getNextToken(StandardTokenizerImpl.java:1133)
	at org.apache.lucene.analysis.standard.StandardTokenizer.incrementToken(StandardTokenizer.java:180)
	at org.apache.lucene.analysis.standard.StandardFilter.incrementToken(StandardFilter.java:49)
	at org.apache.lucene.analysis.core.LowerCaseFilter.incrementToken(LowerCaseFilter.java:54)
	at org.apache.lucene.analysis.util.FilteringTokenFilter.incrementToken(FilteringTokenFilter.java:50)
	at com.my.test.analyzer.AnalyzerUtil.displayToken(AnalyzerUtil.java:23)
	at com.my.test.analyzer.AnalyzerTest.AnalyzerTest(AnalyzerTest.java:24)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at org.junit.internal.runners.TestMethodRunner.executeMethodBody(TestMethodRunner.java:99)
	at org.junit.internal.runners.TestMethodRunner.runUnprotected(TestMethodRunner.java:81)
	at org.junit.internal.runners.BeforeAndAfterRunner.runProtected(BeforeAndAfterRunner.java:34)
	at org.junit.internal.runners.TestMethodRunner.runMethod(TestMethodRunner.java:75)
	at org.junit.internal.runners.TestMethodRunner.run(TestMethodRunner.java:45)
	at org.junit.internal.runners.TestClassMethodsRunner.invokeTestMethod(TestClassMethodsRunner.java:75)
	at org.junit.internal.runners.TestClassMethodsRunner.run(TestClassMethodsRunner.java:36)
	at org.junit.internal.runners.TestClassRunner$1.runUnprotected(TestClassRunner.java:42)
	at org.junit.internal.runners.BeforeAndAfterRunner.runProtected(BeforeAndAfterRunner.java:34)
	at org.junit.internal.runners.TestClassRunner.run(TestClassRunner.java:52)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:130)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:78)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:212)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:68)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)


解决办法:


在进行incrementToken之前进行reset。问题随之解决。


运行结果:





评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值