源码地址:
1.确定要解决的问题及意义
在基于代码片段的分类过程中,由于程序员对数据变量名的选取可能具有一定的规范性,在某一特定业务处理逻辑代码中,可能多个变量名之间具有关联性或相似性(如“trade”(商品交易)类中,可能存在“business”,“transaction”,“deal”等同义词),在某些情况下,它们以不同的词语表达了相同的含义。因此,为了能够对代码片段做出更加科学的类别判断,更好地识别这些同义词,我们有必要寻找一种能够解决避免由于同义词的存在而导致误分类的方法。说白了,就是要去判断词语之间的相似度(即确定是否为近义词),并找出代码段中出现次数最多的一组语义。
2.要达到的效果
即在给定的代码段中,能够发现哪些词是属于同义词,并且能够实现分类。
Eg.public static void function(){
String trade=”money”;
Int deal=5;
Long long business=0xfffffff;
Boolen transaction=TRUE;
……
}
Output:同义词有:trade,deal,business,transaction
这段代码很可能与trade有关
3.初识WordNet
问题确定了之后,通过网上的搜索,发现了WordNet和word2vec这两个相关的词汇。(后知后觉,这本身就是一个找近义词的过程)
3.1 WordNet是什么
首先,来看WordNet。搜了一下相关介绍:
WordNet是一个由认识科学实验室在心理学教授的指导下建立和维护的。开发工作从1985年开始,从此以后该项目接受了超过300万美元的资助(主要来源于对有兴趣的)。
由于它包含了语义信息,所以有别于通常意义上的字典。WordNet根据词条的意义将它们分组,每一个具有相同意义的字条组称为一个synset(同义词集合)。WordNet为每一个synset提供了简短,概要的定义,并记录不同synset之间的语义关系。
WordNet的开发有两个目的:
它既是一个,又是一个,它比单纯的辞典或词典都更加易于使用。
支持自动的以及人工智能应用。
WordNet内部结构
在WordNet中,,,和各自被组织成一个同义词的网络,每个同义词集合都代表一个基本的语义概念,并且这些集合之间也由各种关系连接。(一个多义词将出现在它的每个意思的同义词集合中)。在WordNet的第一版中(标记为1.x),四种不同词性的网络之间并无连接。WordNet的名词网络是第一个发展起来的。
名词网络的主干是蕴涵关系的层次(上位/下位关系),它占据了关系中的将近80%。层次中的最顶层是11个抽象概念,称为基本类别始点(unique beginners),例如实体(entity,“有生命的或无生命的具体存在”),心理特征(psychological feature,“生命有机体的精神上的特征)。名词层次中最深的层次是16个节点。
(wikipedia)
通俗地来说,WordNet是一个结构化很好的知识库,它不但包括一般的词典功能,另外还有词的分类信息。目前,基于WordNet的方法相对来说比较成熟,比如路径方法 (lch)、基于信息论方法(res)等。(详见参考文献)
3.2 WordNet的安装与配置
有了WordNet ,也就等于是有了我们所要的单词库。所以,暂时先不考虑相似度的计算,把WordNet下载下来再说。
参考。顺利地下载,安装以及跑demo。
之后,一起来看一下WordNet的文件结构:
bin目录下,有可执行文件WordNet 2.1.exe:
可以看到,WordNet对所有的英文单词都进行的分类,并且形成了一棵语义树。在本例中,entity——>abstract entity——>abstraction——>attribute——>state——>feeling——> emotion——>love;
从叶子节点到根节点
WordNet名次分类中的25个基本类:
dict目录里面存放的就是资源库了,可以看到,它以形容词,副词,名词,动词来分类:
doc为WordNet的用户手册文件文件夹
lib为WordNet软件使用Windows资源的函数库
src为源码文件夹
4.解决问题的大致思路
我们首先以 WordNet 的词汇语义分类作为基础,抽取出其中的同义词,然后采用基于向量空间的方法计算出相似度。工作流程如下:
5.基于WordNet的相似度计算
以下摘自:《基于WordNet的英语词语相似度计算》
5.1 特征提取
5.2 意义相似度和词语相似度的计算
6.实现效果
与“trade”的相似度比较:
分析:
先看第一组:trade vs trade
自己和自己当然是相似度100%
再看第二组:trade#n#5 vs deal#n#1
相似度竟然和第一组是一样的!根据结果,trade作为名词时,它的第5种含义和deal作为名词时的第1种含义是完全相似的。让我们去库里看个究竟:
trade#n#5:
deal#n#1:
再来看一组不是很好理解的:
trade#n#7 vs deal#n#2
他们的相似度达到了0.14+,算是比较高的了,这是为什么呢?
trade#n#7:
sunshine#n#2:
相信聪明的你一定明白了为什么。
与“cat”的相似度比较:
7.代码分析
工程结构图:
test.java
1 package JWordNetSim.test; 2 3 import java.io.FileInputStream; 4 import java.util.HashMap; 5 import java.util.Map; 6 7 import net.didion.jwnl.JWNL; 8 import net.didion.jwnl.data.IndexWord; 9 import net.didion.jwnl.data.POS;10 import net.didion.jwnl.dictionary.Dictionary;11 import shef.nlp.wordnet.similarity.SimilarityMeasure;12 13 /**14 * A simple test of this WordNet similarity library.15 * @author Mark A. Greenwood16 */17 public class Test18 {19 public static void main(String[] args) throws Exception20 { 21 //在运行代码前,必须在本机上安装wordnet2.0,只能装2.0,装了2.1会出错22 JWNL.initialize(new FileInputStream("D:\\JAVAProjectWorkSpace\\jwnl\\JWordNetSim\\test\\wordnet.xml"));23 24 //建议一个映射去配置相关参数25 Mapparams = new HashMap ();26 27 //the simType parameter is the class name of the measure to use28 params.put("simType","shef.nlp.wordnet.similarity.JCn");29 30 //this param should be the URL to an infocontent file (if required31 //by the similarity measure being loaded)32 params.put("infocontent","file:D:\\JAVAProjectWorkSpace\\jwnl\\JWordNetSim\\test\\ic-bnc-resnik-add1.dat");33 34 //this param should be the URL to a mapping file if the35 //user needs to make synset mappings36 params.put("mapping","file:D:\\JAVAProjectWorkSpace\\jwnl\\JWordNetSim\\test\\domain_independent.txt");37 38 //create the similarity measure39 SimilarityMeasure sim = SimilarityMeasure.newInstance(params);40 41 //取词42 // Dictionary dict = Dictionary.getInstance(); 43 // IndexWord word1 = dict.getIndexWord(POS.NOUN, "trade"); //这里把trade和dog完全定义为名词来进行处理44 // IndexWord word2 = dict.getIndexWord(POS.NOUN,"dog"); //45 // 46 // //and get the similarity between the first senses of each word47 // System.out.println(word1.getLemma()+"#"+word1.getPOS().getKey()+"#1 " + word2.getLemma()+"#"+word2.getPOS().getKey()+"#1 " + sim.getSimilarity(word1.getSense(1), word2.getSense(1))); 48 49 // //get similarity using the string methods (note this also makes use50 // //of the fake root node)51 // System.out.println(sim.getSimilarity("trade#n","deal#n"));52 53 //get a similarity that involves a mapping54 System.out.println(sim.getSimilarity("trade", "trade"));55 System.out.println(sim.getSimilarity("trade", "deal"));56 System.out.println(sim.getSimilarity("trade", "commerce"));57 System.out.println(sim.getSimilarity("trade", "transaction")); 58 System.out.println(sim.getSimilarity("trade", "finance"));59 System.out.println(sim.getSimilarity("trade", "financial"));60 System.out.println(sim.getSimilarity("trade", "business"));61 System.out.println(sim.getSimilarity("trade", "economy")); 62 System.out.println(sim.getSimilarity("trade", "school"));63 System.out.println(sim.getSimilarity("trade", "dog"));64 System.out.println(sim.getSimilarity("trade", "cat"));65 System.out.println(sim.getSimilarity("trade", "book"));66 System.out.println(sim.getSimilarity("trade", "sunshine"));67 System.out.println(sim.getSimilarity("trade", "smile"));68 System.out.println(sim.getSimilarity("trade", "nice"));69 System.out.println(sim.getSimilarity("trade", "hardly"));70 System.out.println(sim.getSimilarity("trade", "beautiful"));71 }72 }
SimilarityMeasure.java
1 package shef.nlp.wordnet.similarity; 2 3 import java.io.BufferedReader; 4 import java.io.InputStreamReader; 5 import java.net.URL; 6 import java.util.Arrays; 7 import java.util.HashMap; 8 import java.util.HashSet; 9 import java.util.LinkedHashMap; 10 import java.util.Map; 11 import java.util.Set; 12 13 import net.didion.jwnl.JWNLException; 14 import net.didion.jwnl.data.IndexWord; 15 import net.didion.jwnl.data.POS; 16 import net.didion.jwnl.data.Synset; 17 import net.didion.jwnl.dictionary.Dictionary; 18 19 /** 20 * An abstract notion of a similarity measure that all provided 21 * implementations extend. 22 * @author Mark A. Greenwood 23 */ 24 public abstract class SimilarityMeasure 25 { 26 /** 27 * A mapping of terms to specific synsets. Usually used to map domain 28 * terms to a restricted set of synsets but can also be used to map 29 * named entity tags to appropriate synsets. 30 */ 31 private Map> domainMappings = new HashMap >(); 32 33 /** 34 * The maximum size the cache can grow to 35 */ 36 private int cacheSize = 5000; 37 38 /** 39 * To speed up computation of the similarity between two synsets 40 * we cache each similarity that is computed so we only have to 41 * do each one once. 42 */ 43 private Map cache = new LinkedHashMap (16,0.75f,true) 44 { 45 public boolean removeEldestEntry(Map.Entry eldest) 46 { 47 //if the size is less than zero then the user is asking us 48 //not to limit the size of the cache so return false 49 if (cacheSize < 0) return false; 50 51 //if the cache has crown bigger than it's max size return true 52 return size() > cacheSize; 53 } 54 }; 55 56 /** 57 * Get a previously computed similarity between two synsets from the cache. 58 * @param s1 the first synset between which we are looking for the similarity. 59 * @param s2 the other synset between which we are looking for the similarity. 60 * @return The similarity between the two sets or null 61 * if it is not in the cache. 62 */ 63 protected final Double getFromCache(Synset s1, Synset s2) 64 { 65 return cache.get(s1.getKey()+"-"+s2.getKey()); 66 } 67 68 /** 69 * Add a computed similarity between two synsets to the cache so that 70 * we don't have to compute it if it is needed in the future. 71 * @param s1 one of the synsets between which we are storring a similarity. 72 * @param s2 the other synset between which we are storring a similarity. 73 * @param sim the similarity between the two supplied synsets. 74 * @return the similarity score just added to the cache. 75 */ 76 protected final double addToCache(Synset s1, Synset s2, double sim) 77 { 78 cache.put(s1.getKey()+"-"+s2.getKey(),sim); 79 80 return sim; 81 } 82 83 /** 84 * Configures the similarity measure using the supplied parameters. 85 * @param params a set of key-value pairs that are used to configure 86 * the similarity measure. See concrete implementations for details 87 * of expected/possible parameters. 88 * @throws Exception if an error occurs while configuring the similarity measure. 89 */ 90 protected abstract void config(Map params) throws Exception; 91 92 /** 93 * Create a new instance of a similarity measure. 94 * @param confURL the URL of a configuration file. Parameters are specified 95 * one per line as key:value pairs. 96 * @return a new instance of a similairy measure as defined by the 97 * supplied configuration URL. 98 * @throws Exception if an error occurs while creating the similarity measure. 99 */100 public static SimilarityMeasure newInstance(URL confURL) throws Exception101 {102 //create map to hold the key-value pairs we are going to read from103 //the configuration file104 Map params = new HashMap ();105 106 //create a reader for the config file107 BufferedReader in = null;108 109 try110 {111 //open the config file112 in = new BufferedReader(new InputStreamReader(confURL.openStream()));113 114 String line = in.readLine();115 while (line != null)116 {117 line = line.trim();118 119 if (!line.equals(""))120 {121 //if the line contains something then122 123 //split the data so we get the key and value124 String[] data = line.split("\\s*:\\s*",2);125 126 if (data.length == 2)127 {128 //if the line is valid add the two parts to the map129 params.put(data[0], data[1]);130 }131 else132 {133 //if the line isn't valid tell the user but continue on134 //with the rest of the file135 System.out.println("Config Line is Malformed: " + line);136 }137 }138 139 //get the next line ready to process140 line = in.readLine();141 }142 }143 finally144 {145 //close the config file if it got opened146 if (in != null) in.close();147 }148 149 //create and return a new instance of the similarity measure specified150 //by the config file151 return newInstance(params);152 }153 154 /**155 * Creates a new instance of a similarity measure using the supplied parameters.156 * @param params a set of key-value pairs which define the similarity measure.157 * @return the newly created similarity measure.158 * @throws Exception if an error occurs while creating the similarity measure.159 */160 public static SimilarityMeasure newInstance(Map params) throws Exception161 {162 //get the class name of the implementation we need to load163 String name = params.remove("simType");164 165 //if the name hasn't been specified then throw an exception166 if (name == null) throw new Exception("Must specifiy the similarity measure to use");167 168 //Get hold of the class we need to load169 @SuppressWarnings("unchecked") Class c = (Class )Class.forName(name);170 171 //create a new instance of the similarity measure172 SimilarityMeasure sim = c.newInstance();173 174 //get the cache parameter from the config params175 String cSize = params.remove("cache");176 177 //if a cache size was specified then set it178 if (cSize != null) sim.cacheSize = Integer.parseInt(cSize);179 180 //get the url of the domain mapping file181 String mapURL = params.remove("mapping");182 183 if (mapURL != null)184 {185 //if a mapping file has been provided then 186 187 //open a reader over the file188 BufferedReader in = new BufferedReader(new InputStreamReader((new URL(mapURL)).openStream()));189 190 //get the first line ready for processing191 String line = in.readLine();192 193 while (line != null)194 {195 if (!line.startsWith("#"))196 {197 //if the line isn't a comment (i.e. it doesn't start with #) then...198 199 //split the line at the white space200 String[] data = line.trim().split("\\s+");201 202 //create a new set to hold the mapped synsets203 Set mappedTo = new HashSet ();204 205 for (int i = 1 ; i < data.length ; ++i)206 {207 //for each synset mapped to get the actual Synsets208 //and store them in the set209 mappedTo.addAll(sim.getSynsets(data[i]));210 }211 212 //if we have found some actual synsets then213 //store them in the domain mappings214 if (mappedTo.size() > 0) sim.domainMappings.put(data[0], mappedTo);215 }216 217 //get the next line from the file218 line = in.readLine();219 }220 221 //we have finished with the mappings file so close it222 in.close();223 } 224 225 //make sure it is configured properly226 sim.config(params);227 228 //then return it229 return sim;230 }231 232 /**233 * This is the method responsible for computing the similarity between two234 * specific synsets. The method is implemented differently for each235 * similarity measure so see the subclasses for detailed information.236 * @param s1 one of the synsets between which we want to know the similarity.237 * @param s2 the other synset between which we want to know the similarity.238 * @return the similarity between the two synsets.239 * @throws JWNLException if an error occurs accessing WordNet.240 */241 public abstract double getSimilarity(Synset s1, Synset s2) throws JWNLException;242 243 /**244 * Get the similarity between two words. The words can be specified either245 * as just the word or in an encoded form including the POS tag and possibly246 * the sense number, i.e. cat#n#1 would specifiy the 1st sense of the noun cat.247 * @param w1 one of the words to compute similarity between.248 * @param w2 the other word to compute similarity between.249 * @return a SimilarityInfo instance detailing the similarity between the250 * two words specified.251 * @throws JWNLException if an error occurs accessing WordNet.252 */253 public final SimilarityInfo getSimilarity(String w1, String w2) throws JWNLException254 {255 //Get the (possibly) multiple synsets associated with each word256 Set ss1 = getSynsets(w1);257 Set ss2 = getSynsets(w2);258 259 //assume the words are not at all similar260 SimilarityInfo sim = null;261 262 for (Synset s1 : ss1)263 {264 for (Synset s2 : ss2)265 {266 //for each pair of synsets get the similarity267 double score = getSimilarity(s1, s2);268 269 if (sim == null || score > sim.getSimilarity())270 {271 //if the similarity is better than we have seen before272 //then create and store an info object describing the273 //similarity between the two synsets274 sim = new SimilarityInfo(w1, s1, w2, s2, score);275 }276 }277 }278 279 //return the maximum similarity we have found280 return sim; 281 }282 283 /**284 * Finds all the synsets associated with a specific word.285 * @param word the word we are interested. Note that this may be encoded286 * to include information on POS tag and sense index.287 * @return a set of synsets that are associated with the supplied word288 * @throws JWNLException if an error occurs accessing WordNet289 */290 private final Set getSynsets(String word) throws JWNLException291 { 292 //get a handle on the WordNet dictionary293 Dictionary dict = Dictionary.getInstance();294 295 //create an emptuy set to hold any synsets we find296 Set synsets = new HashSet ();297 298 //split the word on the # characters so we can get at the299 //upto three componets that could be present: word, POS tag, sense index300 String[] data = word.split("#");301 302 //if the word is in the domainMappings then simply return the mappings303 if (domainMappings.containsKey(data[0])) return domainMappings.get(data[0]);304 305 if (data.length == 1)306 {307 //if there is just the word308 309 for (IndexWord iw : dict.lookupAllIndexWords(data[0]).getIndexWordArray())310 {311 //for each matching word in WordNet add all it's senses to312 //the set we are building up313 synsets.addAll(Arrays.asList(iw.getSenses()));314 }315 316 //we have finihsed so return the synsets we found317 return synsets;318 }319 320 //the calling method specified a POS tag as well so get that321 POS pos = POS.getPOSForKey(data[1]);322 323 //if the POS tag isn't valid throw an exception324 if (pos == null) throw new JWNLException("Invalid POS Tag: " + data[1]);325 326 //get the word with the specified POS tag from WordNet327 IndexWord iw = dict.getIndexWord(pos, data[0]);328 329 if (data.length > 2)330 {331 //if the calling method specified a sense index then332 //add just that sysnet to the set we are creating333 synsets.add(iw.getSense(Integer.parseInt(data[2])));334 }335 else336 {337 //no sense index was specified so add all the senses of338 //the word to the set we are creating339 synsets.addAll(Arrays.asList(iw.getSenses()));340 }341 342 //return the set of synsets we found for the specified word343 return synsets;344 }345 }
每个函数都有详细注解,大家应该都看的明白。
262~277的循环过程如下:
JCN.java
1 /************************************************************************ 2 * Copyright (C) 2006-2007 The University of Sheffield * 3 * Developed by Mark A. Greenwood* 4 * * 5 * This program is free software; you can redistribute it and/or modify * 6 * it under the terms of the GNU General Public License as published by * 7 * the Free Software Foundation; either version 2 of the License, or * 8 * (at your option) any later version. * 9 * * 10 * This program is distributed in the hope that it will be useful, * 11 * but WITHOUT ANY WARRANTY; without even the implied warranty of * 12 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * 13 * GNU General Public License for more details. * 14 * * 15 * You should have received a copy of the GNU General Public License * 16 * along with this program; if not, write to the Free Software * 17 * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. * 18 ************************************************************************/ 19 20 package shef.nlp.wordnet.similarity; 21 22 import net.didion.jwnl.JWNLException; 23 import net.didion.jwnl.data.Synset; 24 25 /** 26 * An implementation of the WordNet similarity measure developed by Jiang and 27 * Conrath. For full details of the measure see: 28 * Jiang J. and Conrath D. 1997. Semantic similarity based on corpus 29 * statistics and lexical taxonomy. In Proceedings of International 30 * Conference on Research in Computational Linguistics, Taiwan.31 * @author Mark A. Greenwood 32 */ 33 public class JCn extends ICMeasure 34 { 35 /** 36 * Instances of this similarity measure should be generated using the 37 * factory methods of { @link SimilarityMeasure}. 38 */ 39 protected JCn() 40 { 41 //A protected constructor to force the use of the newInstance method 42 } 43 44 @Override public double getSimilarity(Synset s1, Synset s2) throws JWNLException 45 { 46 //if the POS tags are not the same then return 0 as this measure 47 //only works with 2 nouns or 2 verbs. 48 if (!s1.getPOS().equals(s2.getPOS())) return 0; 49 50 //see if the similarity is already cached and... 51 Double cached = getFromCache(s1, s2); 52 53 //if it is then simply return it 54 if (cached != null) return cached.doubleValue(); 55 56 //Get the Information Content (IC) values for the two supplied synsets 57 double ic1 = getIC(s1); 58 double ic2 = getIC(s2); 59 60 //if either IC value is zero then cache and return a sim of 0 61 if (ic1 == 0 || ic2 == 0) return addToCache(s1,s2,0); 62 63 //Get the Lowest Common Subsumer (LCS) of the two synsets 64 Synset lcs = getLCSbyIC(s1,s2); 65 66 //if there isn't an LCS then cache and return a sim of 0 67 if (lcs == null) return addToCache(s1,s2,0); 68 69 //get the IC valueof the LCS 70 double icLCS = getIC(lcs); 71 72 //compute the distance between the two synsets 73 //NOTE: This is the original JCN measure 74 double distance = ic1 + ic2 - (2 * icLCS); 75 76 //assume the similarity between the synsets is 0 77 double sim = 0; 78 79 if (distance == 0) 80 { 81 //if the distance is 0 (i.e. ic1 + ic2 = 2 * icLCS) then... 82 83 //get the root frequency for this POS tag 84 double rootFreq = getFrequency(s1.getPOS()); 85 86 if (rootFreq > 0.01) 87 { 88 //if the root frequency has a value then use it to generate a 89 //very large sim value 90 sim = 1/-Math.log((rootFreq - 0.01) / rootFreq); 91 } 92 } 93 else 94 { 95 //this is the normal case so just convert the distance 96 //to a similarity by taking the multiplicative inverse 97 sim = 1/distance; 98 } 99 100 //cache and return the calculated similarity101 return addToCache(s1,s2,sim);102 }103 }
LIN.java
1 package shef.nlp.wordnet.similarity; 2 3 import net.didion.jwnl.JWNLException; 4 import net.didion.jwnl.data.Synset; 5 6 /** 7 * An implementation of the WordNet similarity measure developed by Lin. For 8 * full details of the measure see: 9 *Lin D. 1998. An information-theoretic definition of similarity. In10 * Proceedings of the 15th International Conference on Machine11 * Learning, Madison, WI.12 * @author Mark A. Greenwood13 */14 public class Lin extends ICMeasure15 {16 /**17 * Instances of this similarity measure should be generated using the18 * factory methods of { @link SimilarityMeasure}.19 */20 protected Lin()21 {22 //A protected constructor to force the use of the newInstance method23 }24 25 @Override public double getSimilarity(Synset s1, Synset s2) throws JWNLException26 {27 //if the POS tags are not the same then return 0 as this measure28 //only works with 2 nouns or 2 verbs.29 if (!s1.getPOS().equals(s2.getPOS())) return 0;30 31 //see if the similarity is already cached and...32 Double cached = getFromCache(s1, s2);33 34 //if it is then simply return it35 if (cached != null) return cached.doubleValue();36 37 //Get the Information Content (IC) values for the two supplied synsets38 double ic1 = getIC(s1);39 double ic2 = getIC(s2);40 41 //if either IC value is zero then cache and return a sim of 042 if (ic1 == 0 || ic2 == 0) return addToCache(s1,s2,0);43 44 //Get the Lowest Common Subsumer (LCS) of the two synsets45 Synset lcs = getLCSbyIC(s1,s2);46 47 //if there isn't an LCS then cache and return a sim of 048 if (lcs == null) return addToCache(s1,s2,0);49 50 //get the IC valueof the LCS51 double icLCS = getIC(lcs);52 53 //caluclaue the similarity score54 double sim = (2*icLCS)/(ic1+ic2);55 56 //cache and return the calculated similarity57 return addToCache(s1,s2,sim);58 }59 }
参考文献:
《基于维基百科的语义相似度计算》盛志超,陶晓鹏(复旦大学计算机科学技术学院);
《基于WordNet的英语词语相似度计算》颜伟,荀恩东(北京语言大学 语言信息处理研究所)
WordNet中的名词:
MIT的JWI(Java WordNet Interface)和JWNL(Java WordNet Library)比较
http://jxr19830617.blog.163.com/blog/static/163573067201301985219857/
http://jxr19830617.blog.163.com/blog/static/1635730672013019105255295/