relief

Relief [generation=heuristic, evaluation=distance].

•Basic algorithm construct :

- each feature is assigned cumulative weightage

   computed over a predefined number of sample data set

   selected from the training data set.

- feature with weightage over a certain threshold

   is the selected feature subset.

•Assignment of weightage :

-  instances belongs to similar class should stay closer together than     

   those in a different class.

- near-hit instance    = similar class.

- near-miss instance = different class.

- W = W - diff(X,nearhit)2 + diff(X,nearmiss)2

1. selected_subset = {}

2. init. all feature weightage = 0  (eg. for 2 features : w1=0, w2=0)

3. for i = 1 to no_of_sample

   get one instance X from the training data set D.

   get nearhit    H  = instance in D where dist(X,H) is closest & X.class=H.class

   get nearmiss M = instance in D where dist(X,M) is closest & X.class<>M.class

   update weightage for all features :

-    weightage   = weightage  -diff(x,h)2    +diff(x,m)2

  eg. weightage1 = weightage1 -diff(x1,h1)2 +diff(x1,m1)2

  eg. weightage2 = weightage2 -diff(x2,h2)2 +diff(x2,m2)2

4. for j = 1 to no_of_feature (eg. 2)

   if weightagej >= Threshold, add featurej to selected_subset

•W = W - diff(X,nearhit)2 - diff(X,nearmiss)2

- try to decrease weightage for instances belong to the same class

   (*note: their dist. diff. should be small).

- try to increase weightage for instances belong to diff class

   (*note: their dist. diff. should be large).

- If (W<=0), then sign of irrelevancy or redundancy.

- If (W>0),   then instances in diff. class is further apart as expected.

•Disadvantages:

- applicable only to binary class problem.

- insufficient training instances fool relief.

- if most features are relevant, relief select all (even if not necessary).

•Advantages:

- noise-tolerant.

- unaffected by feature interaction

   (weightage is cumulative & det. collectively).