Cosine similarity doc

[rchennnn]

Context / Scenario

Read the document Cosine Similarity.

What happened?

The document Cosine Similarity contains the following text:

Cosine similarity is particularly useful when working with high-dimensional data such as word embeddings because it takes into account both the magnitude and direction of each vector. This makes it more robust than other measures like Euclidean distance, which only considers the direction.

This is simply false.

Example

Cosine similarity checks the similarity by first scaling vectors to become a unit vector, wiping all magnitude effects. And then it gives a distance measure via the how close the two vectors are aligned via angles (cosine of angle). A vector [1,1] and vector [100, 100] have a cosine similarity of 1, i.e. they are the same.

The Euclidean distance does not involve a normed the magnitude and the above two vectors have a L2 distance of about 140.

Now comparing [1,1] to [10, 10], the cosine similarity is still 1, but the L2 distance is now about 12.72.

Thus the Euclidean distance clearly takes the magnitude into account while the cosine similarity does not.

This was an issue raised by @aguzev in #651 but it appears there was a misunderstanding and the error was not corrected.

Further Discussion Cautioning the Use of Cosine

Additionally, calling cosine a more robust method of similarity is not true, as there is ongoing discussion on cosine similarity's usefulness

• https://arxiv.org/pdf/2205.05092 (Problems with Cosine as a Measure of Embedding Similarity for High Frequency Words)
• https://arxiv.org/pdf/2403.05440 (Is Cosine-Similarity of Embeddings Really About Similarity?)

Proposed edits

My suggestion is to simply remove the paragraph, and the article would be solid piece. As it stands, the description can easily be mistaken as a false mathematical statement.

Importance

a fix would make my life easier

Platform, Language, Versions

KM Version 0.62

Relevant log output

[dluc]

Is this correct?

[dluc]

Cosine similarity checks the similarity by first scaling vectors to become a unit vector, wiping all magnitude effects.

That is what "it takes into account the magnitude" in the docs means. Cosine similarity takes into account the noise introduced by magnitude and removes it.

It depends how one reads the text - so, to be clearer, the text could be rewritten as:

Cosine similarity is particularly useful when working with high-dimensional data such as word embeddings because it normalizes vectors, focusing on their orientation to capture the relevant information, removing their magnitude to avoid the noise from scaling, frequency, and length.

[rchennnn]

I agree, the rewritten version reads more accurately and clearly. It is good to reduce the claims of robustness to just indicating that it is indeed a particularly useful measurement that removes affects coming from magnitude. But because not all problems benefit from such normalization, it is not always useful.

Cosine similarity is particularly useful when working with high-dimensional data such as word embeddings because it normalizes vectors, focusing on their orientation to capture the relevant information, removing their magnitude to avoid the noise from scaling, frequency, and length.

[dluc]

Doc updated

[aguzev]

Much better, but this part still is false:

Cosine similarity is particularly useful when working with high-dimensional data such as word embeddings because it takes into account both the magnitude and direction of each vector. This makes it more robust than other measures like Euclidean distance, which only considers the direction.

[rchennnn]

@dluc I agree with the above.

The current wording is still false. Please use the below:

Cosine similarity is particularly useful when working with high-dimensional data such as word embeddings because it normalizes vectors, focusing on their orientation to capture the relevant information, removing their magnitude to avoid the noise from scaling, frequency, and length.

[aguzev]

It seems that now we have both wordings next to each other. I believe removing the older one will resolve the issue.