ML Wiki

Term Contribution

Idea: result of clustering highly depends on how similar are documents

so contribution of a term $t$ is how much it contributes to similarity of two documents

Text clustering is highly dependent on the documents similarity.

• Suppose use a Dot Product based similarity:
• $\text{similarity}(d_i, d_j) = \sum_{t \in V} f(t, d_i) \times f(t, d_j)$
  • where $f(t, d)$ represents the weight of term $t$ in document $d$

The contribution of each term is the overall contribution to documents’ similarities and shown by the following equation:

• $\text{TC}(t) = \sum_{i,j} f(t, d_i) \times f(t, d_j)$

It's slow - $O(n^2)$

• sample to speed it up

Sources

• Liu, Tao, et al. "An evaluation on feature selection for text clustering." ICML. Vol. 3. 2003. [1]
• Aggarwal, Charu C., and ChengXiang Zhai. "A survey of text clustering algorithms." Mining Text Data. Springer US, 2012. [2]
• http://cs.gmu.edu/~carlotta/teaching/INFS-795-s05/readings/INFS795_MCayci.ppt