语种检测

最新推荐文章于 2024-09-11 09:07:21 发布

转载最新推荐文章于 2024-09-11 09:07:21 发布 · 613 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/xmaples/p/6654153.html

语种检测language detector工具https://github.com/optimaize/language-detector

官网示例小改动后的代码：

//load all languages:
List<LanguageProfile> languageProfiles = new LanguageProfileReader().readAllBuiltIn();

//build language detector:
LanguageDetector languageDetector = LanguageDetectorBuilder.create(NgramExtractors.standard())
        .withProfiles(languageProfiles)
        .build();

//create a text object factory
TextObjectFactory textObjectFactory = CommonTextObjectFactories.forDetectingOnLargeText();

//query:
//TextObject textObject = textObjectFactory.forText("my text");
//上面的my text这个字符串不会检测到英语，因为示例中CommonTextObjectFactories创建的默认TextObjectFactory附带文本URL过滤器和0.3阈值的RemoveMinorityScriptsTextFilter，后者会把那么短的文本全过滤掉，导致不能检检测出语种。
TextObject textObject = textObjectFactory.forText("Song of the Sea (Irish: Amhrán na Mara) is a 2014 animated fantasy drama film directed and co-produced by Tomm Moore");   // long enough text containing many words....
Optional<LdLocale> lang = languageDetector.detect(textObject);

if(lang.isPresent())
    System.out.println(lang.get());
else
    System.out.println("nothing");

官网的“my text”不能检测出语种，通过查看源代码，并分析逻辑，得知原因是文本过短。涉及的具体类看代码中注释。

转载于:https://www.cnblogs.com/xmaples/p/6654153.html