[AGE-287] Add semantic similarity evaluator #1742
Labels
enhancement
New feature or request
feature
Medium priority
Created by Linear-GitHub Sync
python
Pull requests that update Python code
Milestone
Issue from user:
I would like to add semantic similarity as an evaluator. Here is a candidate code
| from sentence_transformers import SentenceTransformer
from sentence_transformers.util import pytorch_cos_sim
def semantic_similarity(row: pandas.core.series.Series, expected: str, response_column_name: str = "response") -> float:
if len(expected) == 1:
logging.warn("Expected should be a list of strings." + "You may have passed in a single string")
doc1 = expected
doc2 = row[response_column_name]
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
embedding_1 = model.encode(doc1, convert_to_tensor=True)
embedding_2 = model.encode(doc2, convert_to_tensor=True)
Notes: We can have this implemented to use the huggingface API if the API key is not provided and run it locally in case it is not.
From SyncLinear.com | AGE-287
The text was updated successfully, but these errors were encountered: