The different existing similarity measures

No account yet ?

A similarity measure is a metric that measures the distance between two character strings.

It intervenes in the fuzzy search or the comparison of strings.

The choice of distance measures is a critical step in clustering that defines how the similarity of two elements x and y is calculated and how they influence the shape of the clusters.

The classic methods for distance measurements are Euclidean and Manhattan distances.

Euclidean distance

Manhattan distance

Where x and y are two vectors of length n.

The Euclidean distance between two points in Euclidean space is the length of a line segment between the two points.

It can be calculated from the Cartesian coordinates of the points using the Pythagorean theorem, it is sometimes called the Pythagorean distance.

The Manhattan distance is a metric of distance between two points in an N-dimensional vector space.

The Manhattan distance is the sum of the lengths of the projections of the line segment between the points on the coordinate axes.

Simply put, it is the sum of the absolute difference between the measurements in all dimensions of two points.

Pearson correlation distance

Pearson correlation measures the degree of a linear relationship between two profiles.

Eisen cosine correlation distance

It’s a special case of Pearson’s correlation with x¯ and y¯ both replaced by zero.

Spearman correlation distance

Spearman's correlation method calculates the correlation between the rank of x and the rank y of the variables.

Where x′i = rank(xi) and y′i = rank(y).

Kendall's Correlation Distance

Kendall's correlation method measures the correspondence between the ranking of the x and y variables.

The total number of possible matches of x with y observations is n(n−1) / 2, where n is the size of x and y.

Start by ordering the pairs by the x values.

If x and y are correlated, then they would have the same relative rank orders.

Now, for each yi, count the number of yj > yi concordant pairs (c) and the number of yj < yi discordant pairs (d).

Where,

nc : total number of concordant pairs

nd : total number of discordant pairs

n : size of x and y

Pearson's correlation analysis is the most commonly used method.

It is also known as parametric correlation which depends on the distribution of the data.

Kendall's and Spearman's correlations are non-parametric and are used to perform rank-based correlation analysis.

Minkowski distance

The Minkowski distance is a metric in a normed vector space that can be viewed as a generalization of both the Euclidean distance and the Manhattan distance.

Distance from Chebyshev

This is the extreme case of the Minkowski distance.

When we use infinity as the value of the parameter p, we end up with a metric that defines distance as the maximum absolute difference between coordinates.

Tested in Anaconda and Python 3.7

import math
 
def Cosine(vec1, vec2) :
    result = InnerProduct(vec1,vec2) / (VectorSize(vec1) * VectorSize(vec2))
    return result
 
def VectorSize(vec) :
    return math.sqrt(sum(math.pow(v,2) for v in vec))
 
def InnerProduct(vec1, vec2) :
    return sum(v1*v2 for v1,v2 in zip(vec1,vec2))
 
def Euclidean(vec1, vec2) :
    return math.sqrt(sum(math.pow((v1-v2),2) for v1,v2 in zip(vec1, vec2)))
 
def Theta(vec1, vec2) :
    return math.acos(Cosine(vec1,vec2)) + math.radians(10)
 
def Triangle(vec1, vec2) :
    theta = math.radians(Theta(vec1,vec2))
    return (VectorSize(vec1) * VectorSize(vec2) * math.sin(theta)) / 2
 
def Magnitude_Difference(vec1, vec2) :
    return abs(VectorSize(vec1) - VectorSize(vec2))
 
def Sector(vec1, vec2) :
    ED = Euclidean(vec1, vec2)
    MD = Magnitude_Difference(vec1, vec2)
    theta = Theta(vec1, vec2)
    return math.pi * math.pow((ED+MD),2) * theta/360
 
def TS_SS(vec1, vec2) :
    return Triangle(vec1, vec2) * Sector(vec1, vec2)
 
vec1 = [2,5]
vec2 = [2,10]
 
print('Vect 1 = ', vec1)
print('Vect 2 = ', vec2, '\n')
 
print('Cosine = ', Cosine(vec1,vec2), '\n')
print('InnerProduct = ', InnerProduct(vec1, vec2), '\n')
print('Euclidean = ', Euclidean(vec1,vec2), '\n')
print('Theta = ', Theta(vec1, vec2), '\n')
print('Triangle = ', Triangle(vec1, vec2), '\n')
print('Magnitude_Difference = ', Magnitude_Difference(vec1, vec2), '\n')
print('Sector = ', Sector(vec1, vec2), '\n')
print('TS_SS = ', TS_SS(vec1,vec2), '\n')
 

Vector_Similarity

  Copyright (c) 2017 KimJunho

GitHub

Vect 1 = [2, 5]
Vect 2 = [2, 10]

Cosine = 0.9832820049844603

InnerProduct = 54

Euclidean = 5.0

Theta = 0.35764374246191644

Triangle = 0.17140001320519763

Magnitude_Difference = 4.812874220051065

Sector = 0.300531823876711

TS_SS = 0.0515111585810504

Cosine drawbacks

Euclidean drawbacks

Triangle's Area Similarity (TS)

Sector's Area Similarity (SS)

TS-SS

Results

In the largest dataset, TS-SS outperforms the cosine with a significant difference, while in other datasets, the TS-SS slightly outperforms the cosine.

Therefore, the best significant result of TS-SS in the larger dataset justifies the robustness and reliability of the model for big data and real-world data where document/text variety is high.