FuzzyQuery中计算edit distance的算法函数

FuzzyQuery使用中有一个计算edit distance的函数,在类FuzzyTermEnum中,如下:

 

 /******************************
   * Compute Levenshtein distance
   ******************************/
  
  /**
   * <p>Similarity returns a number that is 1.0f or less (including negative numbers)
   * based on how similar the Term is compared to a target term.  It returns
   * exactly 0.0f when
   * <pre>
   *    editDistance &gt; maximumEditDistance</pre>
   * Otherwise it returns:
   * <pre>
   *    1 - (editDistance / length)</pre>
   * where length is the length of the shortest term (text or target) including a
   * prefix that are identical and editDistance is the Levenshtein distance for
   * the two words.</p>
   *
   * <p>Embedded within this algorithm is a fail-fast Levenshtein distance
   * algorithm.  The fail-fast algorithm differs from the standard Levenshtein
   * distance algorithm in that it is aborted if it is discovered that the
   * minimum distance between the words is greater than some threshold.
   *
   * <p>To calculate the maximum distance threshold we use the following formula:
   * <pre>
   *     (1 - minimumSimilarity) * length</pre>
   * where length is the shortest term including any prefix that is not part of the
   * similarity comparison.  This formula was derived by solving for what maximum value
   * of distance returns false for the following statements:
   * <pre>
   *   similarity = 1 - ((float)distance / (float) (prefixLength + Math.min(textlen, targetlen)));
   *   return (similarity > minimumSimilarity);</pre>
   * where distance is the Levenshtein distance for the two words.
   * </p>
   * <p>Levenshtein distance (also known as edit distance) is a measure of similarity
   * between two strings where the distance is measured as the number of character
   * deletions, insertions or substitutions required to transform one string to
   * the other string.
   * @param target the target word or phrase
   * @return the similarity,  0.0 or less indicates that it matches less than the required
   * threshold and 1.0 indicates that the text and target are identical
   */
  private float similarity(final String target) {
    final int m = target.length();
    final int n = text.length;
    if (n == 0)  {
      //we don't have anything to compare.  That means if we just add
      //the letters for m we get the new word
      return prefix.length() == 0 ? 0.0f : 1.0f - ((float) m / prefix.length());
    }
    if (m == 0) {
      return prefix.length() == 0 ? 0.0f : 1.0f - ((float) n / prefix.length());
    }

    final int maxDistance = calculateMaxDistance(m);

    if (maxDistance < Math.abs(m-n)) {
      //just adding the characters of m to n or vice-versa results in
      //too many edits
      //for example "pre" length is 3 and "prefixes" length is 8.  We can see that
      //given this optimal circumstance, the edit distance cannot be less than 5.
      //which is 8-3 or more precisely Math.abs(3-8).
      //if our maximum edit distance is 4, then we can discard this word
      //without looking at it.
      return 0.0f;
    }

    // init matrix d
    for (int i = 0; i<=n; ++i) {
      p[i] = i;
    }

    // start computing edit distance
    for (int j = 1; j<=m; ++j) { // iterates through target
      int bestPossibleEditDistance = m;
      final char t_j = target.charAt(j-1); // jth character of t
      d[0] = j;

      for (int i=1; i<=n; ++i) { // iterates through text
        // minimum of cell to the left+1, to the top+1, diagonally left and up +(0|1)
        if (t_j != text[i-1]) {
          d[i] = Math.min(Math.min(d[i-1], p[i]),  p[i-1]) + 1;
		} else {
          d[i] = Math.min(Math.min(d[i-1]+1, p[i]+1),  p[i-1]);
		}
        bestPossibleEditDistance = Math.min(bestPossibleEditDistance, d[i]);
      }

      //After calculating row i, the best possible edit distance
      //can be found by found by finding the smallest value in a given column.
      //If the bestPossibleEditDistance is greater than the max distance, abort.

      if (j > maxDistance && bestPossibleEditDistance > maxDistance) {  //equal is okay, but not greater
        //the closest the target can be to the text is just too far away.
        //this target is leaving the party early.
        return 0.0f;
      }

      // copy current distance counts to 'previous row' distance counts: swap p and d
      int _d[] = p;
      p = d;
      d = _d;
    }
fuzzysearch查找字符串库中精确和模糊匹配 fuzzy-search 查找字符串库中精确和模糊匹配 立即下载

相关推荐

揭秘Outlook/QQ邮箱伪造:如何利用Unicode编码绕过SPF检测

本文深入剖析了利用Unicode编码进行邮件显示名伪造的攻击手法,揭示了攻击者如何通过插入零宽字符等特殊编码绕过SPF检测,使邮件在Outlook、QQ邮箱等客户端中看似来自可信发件人。文章通过Python代码示例演示了伪造过程,并提供了从用户自查到管理员部署DMARC策略的多层次防御建议。

weixin_29063035的博客 332

Edit Distance

题目:Edit Distance  (链接地址) Definition of  Minimum Edit Distance  Edit Distance 用于衡量两个strings之间的相似性。 两个strings之间的 Minimum edit distance 是指把其中一个string通过编辑(包括插入,删除,替换操作)转换为另一个string的最小操作数。 如上图所示

Boyhou 的专栏 1317

ARM架构实战:在鲲鹏920服务器上用Debian12部署PVE虚拟化平台

本文详细介绍了在基于ARM架构的华为鲲鹏920服务器上,以Debian 12为底层系统,部署Proxmox VE(PVE)虚拟化平台的完整实战流程。文章重点解析了ARM64架构下PVE软件源的配置、内核安装的注意事项、部署后网络与存储的调优,并提供了针对ARM虚拟化环境的性能优化建议,为在国产化硬件上构建稳定高效的虚拟化环境提供了宝贵经验。

efc12345678的博客 780

elasticsearch笔记

-------------------------------基础---------------------------------- #基本概念: 索引库(indices):indices是index的复数,代表许多的索引 类型(type):类型是模拟mysql中的table概念,一个索引库下可以有不同类型的索引,比如商品索引,订单索引,其数据格式不同。不过这会导致索引库混乱,因此未来版本中会移...

鱼获飞的博客 359

告别里程焦虑:手把手教你用MATLAB/Simulink仿真电动车AMT换挡规律(附代码)

本文详细介绍了如何使用MATLAB/Simulink仿真电动车AMT换挡规律,帮助解决里程焦虑问题。通过搭建完整的仿真系统,包括电机MAP图处理、Stateflow逻辑实现和动力性与经济性换挡曲线生成,优化电动车换挡策略以提升续航与性能。附带的代码示例和实用技巧使开发者能够快速上手并验证仿真结果。

weixin_26757925的博客 304

封神台靶场,为了小芳,冲之——cookie注入

第二个靶场 打开网站 找一下注入点 一般在新闻这种里面点开看看 先判断是否存在注入 有waf 尝试了很多方法绕过都没成功,/and/,aandnd,AnD等一系列都没绕过去,乌鸡鲅鱼 删掉id=169 发现 然后判断cookie注入 在搜索框输入 Javascript:alert(document.cookie=”id=”+escape(‘169’)); 然后弹窗了(其中 “id=” 取决URL中的id= escape中的数值也是URL中id=的值)返回刚刚的页面,把ID删除后一样能正常显示。说明服务器能

Wubuqing的博客 649

Lucene5学习之FuzzyQuery使用

     接着学习MultiTermQuery下的另一个Query子类FuzzyQuery,它用于模糊相似度查询,那这里说的相似度是如何判定的?用到的是Damerau-Levenshtein算法,具体这个算法的原理我也不是很清楚,只知道个大概,Levenshtein中文一般翻译为编辑距,何为编辑距?即两个字符串有一个转变成另一个所需要的最小的操作步骤,这里说的操作步骤指的是插入一个字符或修改一...

嘿↗你的益达 5191

Springboot信息泄露以及heapdump的利用

努力是为了不平庸~安全有些时候是枯燥的,这一次,让我们先人一步,趣学渗透!

LiBai'S BLOG 3万+

FuzzyQuery查询

FuzzyQuery查询(可以简单地识别两个相近的词语)   1.在FuzzyQuery类定义中定义了两个成员变量: private float minimumSimilarity; private int prefixLength; minimumSimilarity是最小相似度,取值范围为0.0~1.0,包含0.0但不包含1.0,默认值为0.5。prefixLength是前缀长度,默

iamaboyy的专栏 4769

ArcGIS利用模型构建器批量导出nc文件中的不同波段数据

ArcGIS利用模型构建器批量导出nc文件中的不同波段数据

Yuanbaokk的博客 7252

Arduino读取串口接收到的数据并发送数据

Arduino读取串口接收到的数据并发送数据 源程序: void setup() { Serial.begin(115200); } void loop() { String comdata ; while (Serial.available() > 0) { comdata += char(Serial.read()); delay(2); } if (comdata.length() > 0) { Serial.print("串口接收到的数据:"); Serial.println(comd

perseverance51博客 1万+

Elasticsearch fuzzy query -- 模糊查询

Elasticsearch fuzzy query -- 模糊查询摘要参数field 下级参数请求示例 摘要 返回包含与搜索字词相似的字词文档;为了找到相似的术语,fuzzy查询将在指定的编辑距离内创建一组搜索词的所有可能的变体或扩展。查询然后返回每个扩展的完全匹配。 如果search.allow_expensive_queries 设置为false,则不会执行模糊查询。 编辑距离,是将一个术语转换成另一个术语所需更改一个字符的次数(Levenshtein编辑距离衡量)。这些更改可以包括: 更改字符,bo

wei_bo_cai的博客 2820

SAP PS 收入确认方法配置清单(成本推导收入 / 项目进度推导收入)

摘要:本文详细介绍了SAPPS模块中两种收入确认方法的配置流程:成本推导收入(成本比例法)和项目进度推导收入(里程碑法)。通过事务码OKG1-OKG5等关键配置步骤,分别实现基于累计成本比例和里程碑完成权重的收入确认逻辑。文章对比了两种方法的适用场景和管控要点:成本法适用于成本与进度强相关的项目,需严控预计总成本;进度法则依赖明确的里程碑验收。同时提供了落地建议,包括项目类型绑定、数据校验和变更管控等最佳实践,为SAPPS系统收入确认功能实施提供完整指导。

金牌架构师 952

Lucene-2.2.0 源代码阅读学习(36)

关于MultiTermQuery查询。 这里研究FuzzyQuery查询。 MultiTermQuery是一个抽象类,继承自它的一种有3个,分别为:FuzzyQuery、WildcardQuery、RegexQuery,其中RegexQuery使用了第三方提供的服务,可以使用正则表达式,如果你对正则表达式很熟悉,可以尝试着使用RegexQuery查询。 FuzzyQuery查询,即模糊查询。 在F...

pavel0的专栏 365

深度学习工具包 DeepLearningToolbox

深度学习工具包Deprecation notice.-----This toolbox is outdated and no longer maintained.There are much better tools available for deep learning than this toolbox, e.g. [Theano](http://deeplearning.net/software/theano/), [torch](http://torch.ch/) or [tensorflow](http://www.tensorflow.org/)I would suggest you use one of the tools mentioned above rather than use this toolbox.Best, Rasmus.DeepLearnToolbox================A Matlab toolbox for Deep Learning.Deep Learning is a new subfield of machine learning that focuses on learning deep hierarchical models of data.It is inspired by the human brain's apparent deep (layered, hierarchical) architecture.A good overview of the theory of Deep Learning theory is[Learning Deep Architectures for AI](http://www.iro.umontreal.ca/~bengioy/papers/ftml_book.pdf)For a more informal introduction, see the following videos by Geoffrey Hinton and Andrew Ng.* [The Next Generation of Neural Networks](http://www.youtube.com/watch?v=AyzOUbkUf3M) (Hinton, 2007)* [Recent Developments in Deep Learning](http://www.youtube.com/watch?v=VdIURAu1-aU) (Hinton, 2010)* [Unsupervised Feature Learning and Deep Learning](http://www.youtube.com/watch?v=ZmNOAtZIgIk) (Ng, 2011)If you use this toolbox in your research please cite [Prediction as a candidate for learning deep hierarchical models of data](http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6284)```@MASTERSTHESIS\{IMM2012-06284, author = "R. B. Palm", title = "Prediction as a candidate for learning deep hierarchical models of data", year = "2012",}```Contact: rasmusbergpalm at gmail dot comDirectories included in the toolbox-----------------------------------`NN/` - A library for Feedforward Backpropagation Neural Networks`CNN/` - A library for Convolutional Neural Networks`DBN/` - A library for Deep Belief Networks`SAE/` - A library for Stacked Auto-Encoders`CAE/` - A library for Convolutional Auto-Encoders`util/` - Utility functions used by the libraries`data/` - Data used by the examples`tests/` - unit tests to verify toolbox is workingFor references on each library check REFS.mdSetup-----1. Download.2. addpath(genpath('DeepLearnToolbox'));Example: Deep Belief Network---------------------```matlabfunction test_example_DBNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit RBM and visualize its weightsrand('state',0)dbn.sizes = [100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);figure; visualize(dbn.rbm{1}.W'); % Visualize the RBM weights%% ex2 train a 100-100 hidden unit DBN and use its weights to initialize a NNrand('state',0)%train dbndbn.sizes = [100 100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);%unfold dbn to nnnn = dbnunfoldtonn(dbn, 10);nn.activation_function = 'sigm';%train nnopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.10, 'Too big error');```Example: Stacked Auto-Encoders---------------------```matlabfunction test_example_SAEload mnist_uint8;train_x = double(train_x)/255;test_x = double(test_x)/255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit SDAE and use it to initialize a FFNN% Setup and train a stacked denoising autoencoder (SDAE)rand('state',0)sae = saesetup([784 100]);sae.ae{1}.activation_function = 'sigm';sae.ae{1}.learningRate = 1;sae.ae{1}.inputZeroMaskedFraction = 0.5;opts.numepochs = 1;opts.batchsize = 100;sae = saetrain(sae, train_x, opts);visualize(sae.ae{1}.W{1}(:,2:end)')% Use the SDAE to initialize a FFNNnn = nnsetup([784 100 10]);nn.activation_function = 'sigm';nn.learningRate = 1;nn.W{1} = sae.ae{1}.W{1};% Train the FFNNopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.16, 'Too big error');```Example: Convolutional Neural Nets---------------------```matlabfunction test_example_CNNload mnist_uint8;train_x = double(reshape(train_x',28,28,60000))/255;test_x = double(reshape(test_x',28,28,10000))/255;train_y = double(train_y');test_y = double(test_y');%% ex1 Train a 6c-2s-12c-2s Convolutional neural network %will run 1 epoch in about 200 second and get around 11% error. %With 100 epochs you'll get around 1.2% errorrand('state',0)cnn.layers = { struct('type', 'i') %input layer struct('type', 'c', 'outputmaps', 6, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %sub sampling layer struct('type', 'c', 'outputmaps', 12, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %subsampling layer};cnn = cnnsetup(cnn, train_x, train_y);opts.alpha = 1;opts.batchsize = 50;opts.numepochs = 1;cnn = cnntrain(cnn, train_x, train_y, opts);[er, bad] = cnntest(cnn, test_x, test_y);%plot mean squared errorfigure; plot(cnn.rL);assert(er<0.12, 'Too big error');```Example: Neural Networks---------------------```matlabfunction test_example_NNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);% normalize[train_x, mu, sigma] = zscore(train_x);test_x = normalize(test_x, mu, sigma);%% ex1 vanilla neural netrand('state',0)nn = nnsetup([784 100 10]);opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samples[nn, L] = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.08, 'Too big error');%% ex2 neural net with L2 weight decayrand('state',0)nn = nnsetup([784 100 10]);nn.weightPenaltyL2 = 1e-4; % L2 weight decayopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex3 neural net with dropoutrand('state',0)nn = nnsetup([784 100 10]);nn.dropoutFraction = 0.5; % Dropout fraction opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex4 neural net with sigmoid activation functionrand('state',0)nn = nnsetup([784 100 10]);nn.activation_function = 'sigm'; % Sigmoid activation functionnn.learningRate = 1; % Sigm require a lower learning rateopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex5 plotting functionalityrand('state',0)nn = nnsetup([784 20 10]);opts.numepochs = 5; % Number of full sweeps through datann.output = 'softmax'; % use softmax outputopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex6 neural net with sigmoid activation and plotting of validation and training error% split training data into training and validation datavx = train_x(1:10000,:);tx = train_x(10001:end,:);vy = train_y(1:10000,:);ty = train_y(10001:end,:);rand('state',0)nn = nnsetup([784 20 10]); nn.output = 'softmax'; % use softmax outputopts.numepochs = 5; % Number of full sweeps through dataopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, tx, ty, opts, vx, vy); % nntrain takes validation set as last two arguments (optionally)[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');```[![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/rasmusbergpalm/deeplearntoolbox/trend.png)](https://bitdeli.com/free "Bitdeli Badge")

华为的C语言编程规范.pdf

华为技术有限公司内部技术规范。本规范制定了编写C语言程序的基本原则、规则和建议。从代码的清晰、简洁、可测试、安全、程序效率、可移植各个方面对C语言编程作出了具体指导。

Spring休眠教程

1.简介 在本文中,我们将演示如何利用最流行的ORM(对象关系映射)工具之一的Hibernate的功能 ,该工具可将面向对象的域模型转换为传统的关系数据库。 Hibernate是目前最流行的Java框架之一。 由于这个原因,我们在Java Code Geeks上提供了很多教程,其中大多数可以在此处找到。 在本课中,我们将创建一个基于Spring Boot的简单应用程序,它将利用Hiber...

最佳 Java 编程 467

efm32 入门.pdf

本PDF共计22页,介绍的是操作驱动API,下载与否自己斟酌。

Levenshtein distance

In information theory and computer science, the Levenshtein distance is a string metric for measuring the difference between two sequences. Informally, the Levenshtein distance between two words is the minimum number of single-character edits (i.e. inserti

庖丁解牛 2261

精雕软件(JDpaint) 5.5 绿色版

精雕软件(JDpaint) 5.5 绿色版精雕软件(JDpaint) 5.5 绿色版精雕软件(JDpaint) 5.5 绿色版

idea的String字符串超出长度编译不通过

idea的string字符串超出长度编译不通过

weixin_38216634的博客 890

FuzzyScore与模糊搜索

导包(maven),其他的上下面那个网站查: &lt;!-- https://mvnrepository.com/artifact/org.apache.commons/commons-text --&gt; &lt;dependency&gt; &lt;groupId&gt;org.apache.commons&lt;/groupId&gt; &lt;artifactId&gt...

与望 1585

SimilarityMeasure&Distance Measure(相似性与距离度量)

SimilarityMeasure&Distance Measure(相似性与距离度量):   EuclideanDistance(欧式距离),Manhattan Distance(曼哈顿距离),Chebyshev Distance(切比雪夫距离),Minkowski Distance(闵可夫斯基距离),Standardized EuclideanDistance(标准化欧氏距离),Maha

If you can dream it,you can do it. 2414

FuzzyQuery模糊查找

模糊查找

杨鑫newlife的专栏 1万+

各种距离算法汇总

1. 欧氏距离,最常见的两点之间或多点之间的距离表示法,又称之为欧几里得度量,它定义于欧几里得空间中,如点 x = (x1,...,xn) 和 y = (y1,...,yn) 之间的距离为: (1)二维平面上两点a(x1,y1)与b(x2,y2)间的欧氏距离: (2)三维空间两点a(x1,y1,z1)与b(x2,y2,z2)间的欧氏距离: (3...

从未完美过的博客 3779

Edit Distance in biology

There are a number of advanced variants of minimum edit distance that play a special role in computational biology. String x : AGA String y : AG Needleman-Wunsch In computational biology, what we ...

weixin_34293246的博客 97

Hibernate Search的使用

Hibernate Search 构建查询流程 从FullTextEntityManager获取QueryBuilder FullTextEntityManager fullTextEntityManager = Search.getFullTextEntityManager(entityManager); QueryBuilder queryBuilder = fullTextEntityManager.getSearchFactory() .buildQueryBuilder() .

weixin_43968026的博客 497

[LeetCode] Maximum Subarray、Edit Distance

Maximum Subarray: Find the contiguous subarray within an array (containing at least one number) which has the largest sum. For example, given the array [−2,1,−3,4,−1,2,1,−5,4], the contiguous

a83610312的专栏 981

Lucene的FuzzyQuery中用到的Levenshtein Distance(LD)算法

主题:Levenshtein Distance(LD); 相关介绍:Levenshtein distance是由俄国科学家Vladimir Levenshtein在1965年设计并以他的名字命名的。如果不能拼写或发Levenshtein音,通常可以称它edit distance(编辑距离);...

chunli5093的博客 268
上一篇: 重新Query对象树及里边用到的常用算法
下一篇: c++实现反射机制(两篇)
eric509
博客等级 码龄19年 8粉丝 170原创
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付:10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值