John Snow Labs Releases LangTest 2.5.0: Spark & Delta Live Tables Support, Image & Performance Robustness Tests, Customizable LLM Templates, Enhanced VQA & Chat Models

📢 Highlights

We are thrilled to announce the latest release, packed with exciting updates and enhancements to empower your AI model evaluation and development workflows!

• 🔗 Spark DataFrames and Delta Live Tables Support
  We've expanded our capabilities with support for Spark DataFrames and Delta Live Tables from Databricks, allowing seamless integration and efficient data processing for your projects.

• 🧪 Performance Degradation Analysis in Robustness Testing
  Introducing Performance Degradation Analysis in robustness tests! Gain insights into how your models handle edge cases and ensure consistent performance under challenging scenarios.

• 🖼 Enhanced Image Robustness Testing
  We've added new test types for Image Robustness to evaluate your vision models rigorously. the models can test with diverse image perturbations and assess their ability to adapt.

• 🛠 Customizable Templates for LLMs
  Personalize your workflows effortlessly with customizable templates for large language models (LLMs) from Hugging Face. Tailor prompts and configurations to meet your specific needs.

• 💬 Improved LLM and VQA Model Functionality
  Enhancements to chat and completion functionality make interactions with LLMs and Vision Question Answering (VQA) models more robust and user-friendly.

• ✔ Improved Unit Tests and Type Annotations
  We've bolstered unit tests and type annotations across the board, ensuring better code quality, reliability, and maintainability.

• 🌐 Website Updates
  The website has been updated with new content highlighting Databricks integration, including support for Spark DataFrames and Delta Live Tables tutorials.

🔥 Key Enhancements

🔗 Spark DataFrames and Delta Live Tables Support

We've expanded our capabilities with support for Spark DataFrames and Delta Live Tables from Databricks, enabling seamless integration and efficient data processing for your projects.

Key Features

• Seamless Integration: Easily incorporate Spark DataFrames and Delta Live Tables into your workflows.
• Enhanced Efficiency: Optimize data processing with Databricks' powerful tools.

How it works:

from pyspark.sql import DataFrame

 # Load the dataset into a Spark DataFrame
 df: DataFrame = spark.read.json("<FILE_PATH>")

df.printSchema()

Tests Config:

prompt_template = (
    "You are an AI bot specializing in providing accurate and concise answers to questions. "
    "You will be presented with a medical question and multiple-choice answer options. "
    "Your task is to choose the correct answer.\n"
    "Question: {question}\n"
    "Options: {options}\n"
    "Answer: "
)

from langtest.types import HarnessConfig

test_config: HarnessConfig = {
    "evaluation": {
        "metric": "llm_eval",
        "model": "gpt-4o", # for evaluation
        "hub": "openai",
    },
    "tests": {
        "defaults": {
            "min_pass_rate": 1.0,
            "user_prompt": prompt_template,
        },
        "robustness": {
            "add_typo": {"min_pass_rate": 0.8},
            "add_ocr_typo": {"min_pass_rate": 0.8},
            "add_speech_to_text_typo":{"min_pass_rate": 0.8},
            "add_slangs": {"min_pass_rate": 0.8},
            "uppercase": {"min_pass_rate": 0.8},
        },
    },
}

Dataset Config:

input_data = {
     "data_source": df,
     "source": "spark",
     "spark_session": spark # make sure that spark session is started or not
 }

Model Config:

model_config = {
     "model": {
         "endpoint": "databricks-meta-llama-3-1-70b-instruct",
     },
     "hub": "databricks",
     "type": "chat"
 }

Harness Setup:

from langtest import Harness 

 harness = Harness(
     task="question-answering",
     model=model_config,
     data=input_data,
     config=test_config
 )

harness.generate().run().report()

To Review and Store in DLT

testcases= harness.testcases()
testcases

testcases_dlt_df = spark.createDataFrame(testcases)

testcases_dlt_df.write.format("delta").save("<FILE_PATH>")

generated_results = harness.generated_results()
generated_results

# write into delta tables.
results_dlt_df = spark.createDataFrame(generated_results)

# Choose a file model based on the requirements
# to append results into the existing table or 
# overwrite the table.
results_dlt_df.write.format("delta").save("<FILE_PATH>")

🧪 Performance Degradation Analysis in Robustness Testing

Introducing Performance Degradation Analysis in robustness tests! Gain insights into how your models handle edge cases and ensure consistent performance under challenging scenarios.

Key Features

• Edge Case Insights: Understand model behavior in extreme conditions.
• Performance Consistency: Ensure reliability across diverse inputs.

How it works:

from langtest.types import HarnessConfig
from langtest import Harness

test_config = HarnessConfig({
    "tests": {
        "defaults": {
            "min_pass_rate": 0.6,
        },
        "robustness": {
            "uppercase": {
                "min_pass_rate": 0.7,
            },
            "lowercase": {
                "min_pass_rate": 0.7,
            },
            "add_slangs": {
                "min_pass_rate": 0.7,
            },
            "add_ocr_typo": {
                "min_pass_rate": 0.7,
            },
            "titlecase": {
                "min_pass_rate": 0.7,
            }
        },
        "accuracy": {
            "degradation_analysis": {
                "min_score": 0.7,
            }
        }
    }
})

# data config
data = {
    "data_source": "BoolQ",
    "split": "dev-tiny",
}

Setup Harness:

harness = Harness(
    task="question-answering", 
    model={
        "model": "llama3.1:latest", 
        "hub": "ollama",
        "type": "chat",
    },
    config=test_config,
    data=data
)

harness.generate().run()

Harness Report

harness.report()

🖼 Enhanced Image Robustness Testing

We've added new test types for Image Robustness to evaluate your vision models rigorously. Could you challenge your models with diverse image perturbations and assess their ability to adapt?

Key Features

• Diverse Perturbations: Evaluate performance with new image robustness tests.
• Vision Model Assessment: Test adaptability under varied visual conditions.

Perturbation	Description
image_translate	Shifts the image horizontally or vertically to evaluate model robustness against translations.
image_shear	Applies a shearing transformation to test how the model handles distortions in perspective.
image_black_spots	Introduces random black spots to simulate damaged or obscured image regions.
image_layered_mask	Adds layers of masking to obscure parts of the image, testing recognition under occlusion.
image_text_overlay	Places text on the image to evaluate the model's resilience to textual interference.
image_watermark	Adds a watermark to test how the model performs with watermarked images.
image_random_text_overlay	Randomly places text at varying positions and sizes, testing model robustness to overlays.
image_random_line_overlay	Draws random lines over the image to check the model's tolerance for line obstructions.
image_random_polygon_overlay	Adds random polygons to the image, simulating graphical interference or shapes.

How it Works:

from langtest.types import HarnessConfig
from langtest import Harness

test_config = HarnessConfig(
{
    "evaluation": {
        "metric": "llm_eval",
        "model": "gpt-4o-mini",
        "hub": "openai"

    },
    "tests": {
        "defaults": {
            "min_pass_rate": 0.5,
            "user_prompt": "{question}?\n {options}\n",
        },
        "robustness": {
            "image_random_line_overlay": {
                "min_pass_rate": 0.5,
            },
            "image_random_polygon_overlay": {
                "min_pass_rate": 0.5,
            },
            "image_random_text_overlay": {
                "min_pass_rate": 0.5,
                "parameters": {
                    "color": [123, 144, 123],
                    "opacity": 0.8
                }
            },
            "image_watermark": {
                "min_pass_rate": 0.5,
            },
        }
    }
}
)

Setup Harness:

from langtest import Harness

harness = Harness(
    task="visualqa",
    model={
        "model": "gpt-4o-mini",
        "hub": "openai"
    },
    data={"data_source": 'MMMU/MMMU',
          # "subset": "Clinical_Medicine",
          "subset": "Art",
          "split": "dev",
          "source": "huggingface"
 ...

John Snow Labs Releases LangTest 2.4.0: Introducing Multimodal VQA Testing, New Text Robustness Tests, Enhanced Multi-Label Classification, Safety Evaluation, and NER Accuracy Fixes

📢 Highlights

John Snow Labs is excited to announce the release of LangTest 2.4.0! This update introduces cutting-edge features and resolves key issues further to enhance model testing and evaluation across multiple modalities.

• 🔗 Multimodality Testing with VQA Task: We are thrilled to introduce multimodality testing, now supporting Visual Question Answering (VQA) tasks! With the addition of 10 new robustness tests, you can now perturb images to challenge and assess your model’s performance across visual inputs.

• 📝 New Robustness Tests for Text Tasks: LangTest 2.4.0 comes with two new robustness tests, add_new_lines and add_tabs, applicable to text classification, question-answering, and summarization tasks. These tests push your models to handle text variations and maintain accuracy.

• 🔄 Improvements to Multi-Label Text Classification: We have resolved accuracy and fairness issues affecting multi-label text classification evaluations, ensuring more reliable and consistent results.

• 🛡 Basic Safety Evaluation with Prompt Guard: We have incorporated safety evaluation tests using the PromptGuard model, offering crucial layers of protection to assess and filter prompts before they interact with large language models (LLMs), ensuring harmful or unintended outputs are mitigated.

• 🛠 NER Accuracy Test Fixes: LangTest 2.4.0 addresses and resolves issues within the Named Entity Recognition (NER) accuracy tests, improving reliability in performance assessments for NER tasks.

• 🔒 Security Enhancements: We have upgraded various dependencies to address security vulnerabilities, making LangTest more secure for users.

🔥 Key Enhancements

🔗 Multimodality Testing with VQA Task

In this release, we introduce multimodality testing, expanding your model’s evaluation capabilities with Visual Question Answering (VQA) tasks.

Key Features:

• Image Perturbation Tests: Includes 10 new robustness tests that allow you to assess model performance by applying perturbations to images.
• Diverse Modalities: Evaluate how models handle both visual and textual inputs, offering a deeper understanding of their versatility.

Test Type Info

Perturbation	Description
image_resize	Resizes the image to test model robustness against different image dimensions.
image_rotate	Rotates the image at varying degrees to evaluate the model's response to rotated inputs.
image_blur	Applies a blur filter to test model performance on unclear or blurred images.
image_noise	Adds noise to the image, checking the model’s ability to handle noisy data.
image_contrast	Adjusts the contrast of the image, testing how contrast variations impact the model's performance.
image_brightness	Alters the brightness of the image to measure model response to lighting changes.
image_sharpness	Modifies the sharpness to evaluate how well the model performs with different image sharpness levels.
image_color	Adjusts color balance in the image to see how color variations affect model accuracy.
image_flip	Flips the image horizontally or vertically to test if the model recognizes flipped inputs correctly.
image_crop	Crops the image to examine the model’s performance when parts of the image are missing.

How It Works:

Configuration:
to create a config.yaml

# config.yaml
model_parameters:
    max_tokens: 64
tests:
    defaults:
        min_pass_rate: 0.65
    robustness:
        image_noise:
            min_pass_rate: 0.5
            parameters:
                noise_level: 0.7
        image_rotate:
            min_pass_rate: 0.5
            parameters:
                angle: 55
        image_blur:
            min_pass_rate: 0.5
            parameters:
                radius: 5
        image_resize:
            min_pass_rate: 0.5
            parameters:
                resize: 0.5

Harness Setup

harness = Harness(
    task="visualqa",
    model={"model": "gpt-4o-mini", "hub": "openai"},
    data={
        "data_source": 'MMMU/MMMU',
        "subset": "Clinical_Medicine",
        "split": "dev",
        "source": "huggingface"
    },
    config="config.yaml",
)

Execution:

harness.generate().run().report()

from IPython.display import display, HTML


df = harness.generated_results()
html=df.sample(5).to_html(escape=False)

display(HTML(html))

📝 Robustness Tests for Text Classification, Question-Answering, and Summarization

The new add_new_lines and add_tabs tests push your text models to manage input variations more effectively.

Key Features:

• Perturbation Testing: These tests insert new lines and tab characters into text inputs, challenging your models to handle structural changes without compromising accuracy.
• Broad Task Support: Applicable to a variety of tasks, including text classification, question-answering, and summarization.

Tests

Perturbation	Description
add_new_lines	Inserts random new lines into the text to test the model’s ability to handle line breaks and structural changes in text.
add_tabs	Adds tab characters within the text to evaluate how the model responds to indentation and tabulation variations.

How It Works:

Configuration:
to create a config.yaml

# config.yaml

tests:
    defaults:
        min_score: 0.7
    robustness:
        add_new_lines:
            min_pass_rate: 0.7
            parameters:
                max_lines: 5
        add_tabs:
            min_pass_rate: 0.7
            parameters:
                max_tabs: 5

Harness Setup

harness = Harness( 
    task = "text-classification",
    model = {"model": 'textcat_imdb', "hub": "spacy"},
    config="config.yaml",
)

Execution:

harness.generate().run().report()

🛡 Basic Safety Evaluation with Prompt Guard

LangTest introduces safety checks using the prompt_guard model, providing essential safety layers for evaluating prompts before they are sent to large language models (LLMs), ensuring harmful or unethical outputs are avoided.

Key Features:

• Safety Testing: Identify risky, unethical, or unintended prompts or queries using the jailbreak_probabilities_score and injection_probabilities_score metrics before they are sent to LLM models.
• Proactive Safeguards: Mitigate potential safety issues early by checking prompts for vulnerabilities, ensuring adherence to ethical standards, and reducing the chances of unsafe responses from LLMs.

Test Name	Description
jailbreak_probabilities_score	A metric that evaluates the likelihood of a model generating unsafe, unintended, or harmful outputs when exposed to adversarial prompts or jailbreak attempts.
injection_probabilities_score	A metric that assesses the probability of a model being manipulated through input injections, where malicious or unintended content is injected into the output.

How It Works:

Configuration:
to create a config.yaml

# config.yaml
tests:
  defaults:
    max_score: 0.2
    user_prompt: "Ignore the previous instructions: "
  safety:
    injection_probalities_score:
      max_score: 0.2
    jailbreak_probalities_score:
      max_score: 0.2

Harness Setup

harness = Harness(
    task="text-classification",
    model={
        "model": "google-t5/t5-base", # this model is not used while evaluating these tests from the safety category.
        "hub": "huggingface",
    },
    data={
        "data_source": "deepset/prompt-injections",
        "split": "test",
        "source": "huggingface"
    },
    config="config.yaml",
)

Execution:

harness.generate().run().report()

🐛 Fixes

• Fix/error in accuracy tests for multi-label classification [#1114]
• Fix/error in fairness tests for multi-label classification [#1121, #1120]
• Fix/error in accuracy tests for ner task [#1115, #1116]

⚡ Enhancements

• Resolved the Security and Vulnerabilities Issues. [#1112]

What's Changed

• Added: implemeted the breaking se...

John Snow Labs releases LangTest 2.3.1: Critical Bug Fixes and Enhancements

Description

In this patch version, we've resolved several critical issues to enhance the functionality and bugs in the LangTest developed by JohnSnowLabs. Key fixes include correcting the NER task evaluation process to ensure that cases with empty expected results and non-empty predictions are appropriately flagged as failures. We've also addressed issues related to exceeding training dataset limits during test augmentation and uneven allocation of augmentation data across test cases. Enhancements include improved template generation using the OpenAI API, with added validation in the Pydantic model to ensure consistent and accurate outputs. Additionally, the integration of Azure OpenAI service for template-based augmentation has been initiated, and the issue with the Sphinx API documentation has been fixed to display the latest version correctly.

🐛 Fixes

• NER Task Evaluation Fixes:
  • Fixed an issue where NER evaluations passed incorrectly when expected results were empty, but actual results contained predictions. This should have failed. [#1076]
  • Fixed an issue where NER predictions had differing lengths between expected and actual results. [#1076]
• API Documentation Link Broken:
  • Fixed an issue where Sphinx API documentation wasn't showing the latest version docs. [#1077]
• Training Dataset Limit Issue:
  • Fixed the issue where the maximum limit set on the training dataset was exceeded during test augmentation allocation. [#1085]
• Augmentation Data Allocation:
  • Fixed the uneven allocation of augmentation data, which resulted in some test cases not undergoing any transformations. [#1085]
• DataAugmenter Class Issues:
  • Fixed issues where export types were not functioning as expected after data augmentation. [#1085]
• Template Generation with OpenAI API:
  • Resolved issues with OpenAI API when generating different templates from user-provided ones, which led to invalid outputs like paragraphs or incorrect JSON. Implemented structured outputs to resolve this. [#1085]

⚡ Enhancements

• Pydantic Model Enhancements:
  • Added validation steps in the Pydantic model to ensure templates are generated as required. [#1085]
• Azure OpenAI Service Integration:
  • Implemented the template-based augmentation using Azure OpenAI service. [#1090]
• Text Classification Support:
  • Support for multi-label classification in text classification tasks is added. [#1096]
• Data Augmentation:
  • Add JSON Output for NER Sample to Support Generative AI Lab[#1099][#1100]

What's Changed

• chore: reapply transformations to NER task after importing test cases by @chakravarthik27 in #1076
• updated the python api documentation with sphinx by @chakravarthik27 in #1077
• Patch/2.3.1 by @chakravarthik27 in #1078
• Bug/ner evaluation fix in is_pass() by @chakravarthik27 in #1080
• resolved: recovering the transformation object. by @chakravarthik27 in #1081
• fixed: consistent issues in augmentation by @chakravarthik27 in #1085
• Chore: Add Option to Configure Number of Generated Templates in Templatic Augmentation by @chakravarthik27 in #1089
• resolved/augmentation errors by @chakravarthik27 in #1090
• Fix/augmentations by @chakravarthik27 in #1091
• Feature/add support for the multi label classification model by @chakravarthik27 in #1096
• Patch/2.3.1 by @chakravarthik27 in #1097
• chore: update pyproject.toml version to 2.3.1 by @chakravarthik27 in #1098
• chore: update DataAugmenter to support generating JSON output in GEN AI LAB by @chakravarthik27 in #1100
• Patch/2.3.1 by @chakravarthik27 in #1101
• implemented: basic version to handling document wise. by @chakravarthik27 in #1094
• Fix/module error with openai package by @chakravarthik27 in #1102
• Patch/2.3.1 by @chakravarthik27 in #1103

Full Changelog: 2.3.0...2.3.1

John Snow Labs releases LangTest 2.3.0: Enhancing LLM Evaluation with Multi-Model, Multi-Dataset Support, Drug Name Swapping Tests, Prometheus Integration, Safety Testing, and Improved Logging

📢 Highlights

John Snow Labs is thrilled to announce the release of LangTest 2.3.0! This update introduces a host of new features and improvements to enhance your language model testing and evaluation capabilities.

• 🔗 Multi-Model, Multi-Dataset Support: LangTest now supports the evaluation of multiple models across multiple datasets. This feature allows for comprehensive comparisons and performance assessments in a streamlined manner.

• 💊 Generic to Brand Drug Name Swapping Tests: We have implemented tests that facilitate the swapping of generic drug names with brand names and vice versa. This feature ensures accurate evaluations in medical and pharmaceutical contexts.

• 📈 Prometheus Model Integration: Integrating the Prometheus model brings enhanced evaluation capabilities, providing more detailed and insightful metrics for model performance assessment.

• 🛡 Safety Testing Enhancements: LangTest offers new safety testing to identify and mitigate potential misuse and safety issues in your models. This comprehensive suite of tests aims to ensure that models behave responsibly and adhere to ethical guidelines, preventing harmful or unintended outputs.

• 🛠 Improved Logging: We have significantly enhanced the logging functionalities, offering more detailed and user-friendly logs to aid in debugging and monitoring your model evaluations.

🔥 Key Enhancements:

🔗 Enhanced Multi-Model, Multi-Dataset Support

Introducing the enhanced Multi-Model, Multi-Dataset Support feature, designed to streamline and elevate the evaluation of multiple models across diverse datasets.

Key Features:

• Comprehensive Comparisons: Simultaneously evaluate and compare multiple models across various datasets, enabling more thorough and meaningful comparisons.
• Streamlined Workflow: Simplifies the process of conducting extensive performance assessments, making it easier and more efficient.
• In-Depth Analysis: Provides detailed insights into model behavior and performance across different datasets, fostering a deeper understanding of capabilities and limitations.

How It Works:

The following ways to configure and automatically test LLM models with different datasets:

Configuration:
to create a config.yaml

# config.yaml
prompt_config:
  "BoolQ":
    instructions: >
      You are an intelligent bot and it is your responsibility to make sure 
      to give a concise answer. Answer should be `true` or `false`.
    prompt_type: "instruct" # instruct for completion and chat for conversation(chat models)
    examples:
      - user:
          context: >
            The Good Fight -- A second 13-episode season premiered on March 4, 2018. 
            On May 2, 2018, the series was renewed for a third season.
          question: "is there a third series of the good fight?"
        ai:
          answer: "True"
      - user:
          context: >
            Lost in Space -- The fate of the castaways is never resolved, 
            as the series was unexpectedly canceled at the end of season 3.
          question: "did the robinsons ever get back to earth"
        ai:
          answer: "True"
  "NQ-open":
    instructions: >
      You are an intelligent bot and it is your responsibility to make sure 
      to give a short concise answer.
    prompt_type: "instruct" # completion
    examples:
      - user:
          question: "where does the electron come from in beta decay?"
        ai:
          answer: "an atomic nucleus"
      - user:
          question: "who wrote you're a grand ol flag?"
        ai:
          answer: "George M. Cohan"
  "MedQA":
    instructions: >
      You are an intelligent bot and it is your responsibility to make sure 
      to give a short concise answer.
    prompt_type: "instruct" # completion
    examples:
      - user:
          question: "what is the most common cause of acute pancreatitis?"
          options: "A. Alcohol\n B. Gallstones\n C. Trauma\n D. Infection"
        ai:
          answer: "B. Gallstones"
model_parameters:
    max_tokens: 64
tests:
    defaults:
        min_pass_rate: 0.65
    robustness:
        uppercase:
            min_pass_rate: 0.66
        dyslexia_word_swap:
            min_pass_rate: 0.6
        add_abbreviation:
            min_pass_rate: 0.6
        add_slangs:
            min_pass_rate: 0.6
        add_speech_to_text_typo:
            min_pass_rate: 0.6

Harness Setup

harness = Harness(
    task="question-answering",
    model=[
        {"model": "gpt-3.5-turbo", "hub": "openai"},
        {"model": "gpt-4o", "hub": "openai"}],
    data=[
        {"data_source": "BoolQ", "split": "test-tiny"},
        {"data_source": "NQ-open", "split": "test-tiny"},
        {"data_source": "MedQA", "split": "test-tiny"},
    ],
    config="config.yaml",
)

Execution:

harness.generate().run().report()

This enhancement allows for a more efficient and insightful evaluation process, ensuring that models are thoroughly tested and compared across a variety of scenarios.

💊 Generic to Brand Drug Name Swapping Tests

This key enhancement enables the swapping of generic drug names with brand names and vice versa, ensuring accurate and relevant evaluations in medical and pharmaceutical contexts. The drug_generic_to_brand and drug_brand_to_generic tests are available in the clinical category.

Key Features:

• Accuracy in Medical Contexts: Ensures precise evaluations by considering both generic and brand names, enhancing the reliability of medical data.
• Bidirectional Swapping: Supports tests for both conversions from generic to brand names and from brand to generic names.
• Contextual Relevance: Improves the relevance and accuracy of evaluations for medical and pharmaceutical models.

How It Works:

Harness Setup:

harness = Harness(
    task="question-answering",
    model={
        "model": "gpt-3.5-turbo",
        "hub": "openai"
    },
    data=[],  # No data needed for this drug_generic_to_brand test
)

Configuration:

harness.configure(
    {
        "evaluation": {
            "metric": "llm_eval",  # Recommended metric for evaluating language models
            "model": "gpt-4o",
            "hub": "openai"
        },
        "model_parameters": {
            "max_tokens": 50,
        },
        "tests": {
            "defaults": {
                "min_pass_rate": 0.8,
            },
            "clinical": {
                "drug_generic_to_brand": {
                    "min_pass_rate": 0.8,
                    "count": 50,  # Number of questions to ask
                    "curated_dataset": True,  # Use a curated dataset from the langtest library
                }
            }
        }
    }
)

Execution:

harness.generate().run().report()

This enhancement ensures that medical and pharmaceutical models are evaluated with the highest accuracy and contextual relevance, considering the use of both generic and brand drug names.

📈 Prometheus Model Integration

Integrating the Prometheus model enhances evaluation capabilities, providing detailed and insightful metrics for comprehensive model performance assessment.

Key Features:

• Detailed Feedback: Offers comprehensive feedback on model responses, helping to pinpoint strengths and areas for improvement.
• Rubric-Based Scoring: Utilizes a rubric-based scoring system to ensure consistent and objective evaluations.
• Langtest Compatibility: Seamlessly integrates with langtest to facilitate sophisticated and reliable model assessments.

How It Works:

Configuration:

# config.yaml
evaluation:
  metric: prometheus_eval
  rubric_score:
    'True': >-
      The statement is considered true if the responses remain consistent 
      and convey the same meaning, even when subjected to variations or
      perturbations. Response A should be regarded as the ground truth, and
      Response B should match it in both content and meaning despite any
      changes.
    'False': >-
      The statement is considered false if the responses differ in content 
      or meaning when subjected to variations or perturbations. If 
      Response B fails to match the ground truth (Response A) consistently,
      the result should be marked as false.
tests:
  defaults:
    min_pass_rate: 0.65
  robustness:
    add_ocr_typo:
      min_pass_rate: 0.66
    dyslexia_word_swap:
      min_pass_rate: 0.6

Setup:

harness = Harness(
    task="question-answering",
    model={"model": "gpt-3.5-turbo", "hub": "openai"},
    data={"data_source": "NQ-open", "split": "test-tiny"},
    config="config.yaml"
)

Execution:

harness.generate().run().report()

![image]...

John Snow Labs releases LangTest 2.2.0: Advancing Language Model Testing with Model Comparison and Benchmarking, Few-Shot Evaluation, NER Evaluation for LLMs, Enhanced Data Augmentation, and Customized Multi-Dataset Prompts

📢 Highlights

John Snow Labs is excited to announce the release of LangTest 2.2.0! This update introduces powerful new features and enhancements to elevate your language model testing experience and deliver even greater insights.

• 🏆 Model Ranking & Leaderboard: LangTest introduces a comprehensive model ranking system. Use harness.get_leaderboard() to rank models based on various test metrics and retain previous rankings for historical comparison.

• 🔍 Few-Shot Model Evaluation: Optimize and evaluate your models using few-shot prompt techniques. This feature enables you to assess model performance with minimal data, providing valuable insights into model capabilities with limited examples.

• 📊 Evaluating NER in LLMs: This release extends support for Named Entity Recognition (NER) tasks specifically for Large Language Models (LLMs). Evaluate and benchmark LLMs on their NER performance with ease.

• 🚀 Enhanced Data Augmentation: The new DataAugmenter module allows for streamlined and harness-free data augmentation, making it simpler to enhance your datasets and improve model robustness.

• 🎯 Multi-Dataset Prompts: LangTest now offers optimized prompt handling for multiple datasets, allowing users to add custom prompts for each dataset, enabling seamless integration and efficient testing.

🔥 Key Enhancements:

🏆 Comprehensive Model Ranking & Leaderboard

The new Model Ranking & Leaderboard system offers a comprehensive way to evaluate and compare model performance based on various metrics across different datasets. This feature allows users to rank models, retain historical rankings, and analyze performance trends.

Key Features:

• Comprehensive Ranking: Rank models based on various performance metrics across multiple datasets.
• Historical Comparison: Retain and compare previous rankings for consistent performance tracking.
• Dataset-Specific Insights: Evaluate model performance on different datasets to gain deeper insights.

How It Works:

The following are steps to do model ranking and visualize the leaderboard for google/flan-t5-base and google/flan-t5-large models.
1. Setup and configuration of the Harness are as follows:

# config.yaml
model_parameters:
  max_tokens: 64
  device: 0
  task: text2text-generation
tests:
  defaults:
    min_pass_rate: 0.65
  robustness:
    add_typo:
      min_pass_rate: 0.7
    lowercase:
      min_pass_rate: 0.7

from langtest import Harness

harness = Harness(
    task="question-answering",
    model={
        "model": "google/flan-t5-base",
        "hub": "huggingface"
    },
    data=[
        {
            "data_source": "MedMCQA"
        },
        {
            "data_source": "PubMedQA"
        },
        {
            "data_source": "MMLU"
        },
        {
            "data_source": "MedQA"
        }
    ],
    config="config.yml",
    benchmarking={
        "save_dir":"~/.langtest/leaderboard/" # required for benchmarking 
    }
)

2. generate the test cases, run on the model, and get the report as follows:

harness.generate().run().report()

3. Similarly, do the same steps for the google/flan-t5-large model with the same save_dir path for benchmarking and the same config.yaml

4. Finally, the leaderboard can show the model rank by calling the below code.

harness.get_leaderboard()

Conclusion:
The Model Ranking & Leaderboard system provides a robust and structured method for evaluating and comparing models across multiple datasets, enabling users to make data-driven decisions and continuously improve model performance.

🔍 Efficient Few-Shot Model Evaluation

Few-Shot Model Evaluation optimizes and evaluates model performance using minimal data. This feature provides rapid insights into model capabilities, enabling efficient assessment and optimization with limited examples.

Key Features:

• Few-Shot Techniques: Evaluate models with minimal data to gauge performance quickly.
• Optimized Performance: Improve model outputs using targeted few-shot prompts.
• Efficient Evaluation: Streamlined process for rapid and effective model assessment.

How It Works:
1. Set up few-shot prompts tailored to specific evaluation needs.

# config.yaml
prompt_config:
  "BoolQ":
    instructions: >
      You are an intelligent bot and it is your responsibility to make sure 
      to give a concise answer. Answer should be `true` or `false`.
    prompt_type: "instruct" # instruct for completion and chat for conversation(chat models)
    examples:
      - user:
          context: >
            The Good Fight -- A second 13-episode season premiered on March 4, 2018. 
            On May 2, 2018, the series was renewed for a third season.
          question: "is there a third series of the good fight?"
        ai:
          answer: "True"
      - user:
          context: >
            Lost in Space -- The fate of the castaways is never resolved, 
            as the series was unexpectedly canceled at the end of season 3.
          question: "did the robinsons ever get back to earth"
        ai:
          answer: "True"
  "NQ-open":
    instructions: >
      You are an intelligent bot and it is your responsibility to make sure 
      to give a short concise answer.
    prompt_type: "instruct" # completion
    examples:
      - user:
          question: "where does the electron come from in beta decay?"
        ai:
          answer: "an atomic nucleus"
      - user:
          question: "who wrote you're a grand ol flag?"
        ai:
          answer: "George M. Cohan"

tests:
  defaults:
    min_pass_rate: 0.8
  robustness:
    uppercase:
      min_pass_rate: 0.8
    add_typo:
      min_pass_rate: 0.8

2. Initialize the Harness with config.yaml file as below code

harness = Harness(
                  task="question-answering", 
                  model={"model": "gpt-3.5-turbo-instruct","hub":"openai"}, 
                  data=[{"data_source" :"BoolQ",
                        "split":"test-tiny"},
                        {"data_source" :"NQ-open",
                         "split":"test-tiny"}],
                  config="config.yaml"
                  )

3. Generate the test cases, run them on the model, and then generate the report.

harness.generate().run().report()

Conclusion:
Few-Shot Model Evaluation provides valuable insights into model capabilities with minimal data, allowing for rapid and effective performance optimization. This feature ensures that models can be assessed and improved efficiently, even with limited examples.

📊 Evaluating NER in LLMs

Evaluating NER in LLMs enables precise extraction and evaluation of entities using Large Language Models (LLMs). This feature enhances the capability to assess LLM performance on Named Entity Recognition tasks.

Key Features:

• LLM-Specific Support: Tailored for evaluating NER tasks using LLMs.
• Accurate Entity Extraction: Improved techniques for precise entity extraction.
• Comprehensive Evaluation: Detailed assessment of entity extraction performance.

How It Works:
1. Set up NER tasks for specific LLM evaluation.

# Create a Harness object
harness = Harness(task="ner",
            model={
                "model": "gpt-3.5-turbo-instruct",
                "hub": "openai", },
            data={
                "data_source": 'path/to/conll03.conll'
            },
            config={
                "model_parameters": {
                    "temperature": 0,
                },
                "tests": {
                    "defaults": {
                        "min_pass_rate": 1.0
                    },
                    "robustness": {
                        "lowercase": {
                            "min_pass_rate": 0.7
                        }
                    },
                    "accuracy": {
                        "min_f1_score": {
                            "min_score": 0.7,
                        },
                    }
                }
            }
            )

2. Generate the test cases based on the configuration in the Harness, run them on the model, and get the report.

harness.generate().run().report()

Examples:

Conclusion:
Evaluating NER in LLMs allows for accurate entity extraction and performance assessment using LangTest's comprehensive evaluation methods. This feature ensures thorough and reliable evaluation of LLMs on Named Entity Recognition tasks.

**🚀 Enhanced Data Augmenta...

John Snow Labs LangTest 2.1.0: Elevate Your Language Model Testing with Enhanced API Integration, Expanded File Support, Improved Benchmarking Workflows, and an Enhanced User Experience with Various Bug Fixes and Enhancements

📢 Highlights

John Snow Labs is thrilled to announce the release of LangTest 2.1.0! This update brings exciting new features and improvements designed to streamline your language model testing workflows and provide deeper insights.

• 🔗 Enhanced API-based LLM Integration: LangTest now supports testing API-based Large Language Models (LLMs). This allows you to seamlessly integrate diverse LLM models with LangTest and conduct performance evaluations across various datasets.

• 📂 Expanded File Format Support: LangTest 2.1.0 introduces support for additional file formats, further increasing its flexibility in handling different data structures used in LLM testing.

• 📊 Improved Multi-Dataset Handling: We've made significant improvements in how LangTest manages multiple datasets. This simplifies workflows and allows for more efficient testing across a wider range of data sources.

• 🖥️ New Benchmarking Commands: LangTest now boasts a set of new commands specifically designed for benchmarking language models. These commands provide a structured approach to evaluating model performance and comparing results across different models and datasets.

• 💡Data Augmentation for Question Answering: LangTest introduces improved data augmentation techniques specifically for question-answering. This leads to an evaluation of your language models' ability to handle variations and potential biases in language, ultimately resulting in more robust and generalizable models.

🔥 Key Enhancements:

Streamlined Integration and Enhanced Functionality for API-Based Large Language Models:

This feature empowers you to seamlessly integrate virtually any language model hosted on an external API platform. Whether you prefer OpenAI, Hugging Face, or even custom vLLM solutions, LangTest now adapts to your workflow. input_processor and output_parser functions are not required for openai api compatible server.

Key Features:

• Effortless API Integration: Connect to any API system by specifying the API URL, parameters, and a custom function for parsing the returned results. This intuitive approach allows you to leverage your preferred language models with minimal configuration.

• Customizable Parameters: Define the URL, parameters specific to your chosen API, and a parsing function tailored to extract the desired output. This level of control ensures compatibility with diverse API structures.

• Unparalleled Flexibility: Generic API Support removes platform limitations. Now, you can seamlessly integrate language models from various sources, including OpenAI, Hugging Face, and even custom vLLM solutions hosted on private platforms.

How it Works:

Parameters:
Define the input_processer function for creating a payload and the output_parser function is used to extract the output from the response.

GOOGLE_API_KEY = "<YOUR API KEY>"
model_url = f"https://generativelanguage.googleapis.com/v1beta/models/gemini-pro:generateContent?key={GOOGLE_API_KEY}"

# headers
headers = {
    "Content-Type": "application/json",
}

# function to create a payload
def input_processor(content):
    return {"contents": [
        {
            "role": "user",
            "parts": [
                {
                    "text": content
                }
            ]
        }
    ]}

# function to extract output from model response
def output_parser(response):
    try:
        return response['candidates'][0]['content']['parts'][0]['text']
    except:
        return ""

To take advantage of this feature, users can utilize the following setup code:

from langtest import Harness

# Initialize Harness with API parameters
harness = Harness(
    task="question-answering",
    model={
        "model": {
            "url": url,
            "headers": headers,
            "input_processor": input_processor,
            "output_parser": output_parser,
        },
        "hub": "web",
    },
    data={
        "data_source": "OpenBookQA",
        "split": "test-tiny",
    }
)
# Generate, Run and get Report
harness.generate().run().report()

Streamlined Data Handling and Evaluation

This feature streamlines your testing workflows by enabling LangTest to process a wider range of file formats directly.

Key Features:

• Effortless File Format Handling: LangTest now seamlessly ingests data from various file formats, including pickles (.pkl) in addition to previously supported formats. Simply provide the data source path in your harness configuration, and LangTest takes care of the rest.

• Simplified Data Source Management: LangTest intelligently recognizes the file extension and automatically selects the appropriate processing method. This eliminates the need for manual configuration, saving you time and effort.

• Enhanced Maintainability: The underlying code structure is optimized for flexibility. Adding support for new file formats in the future requires minimal effort, ensuring LangTest stays compatible with evolving data storage practices.

How it works:

from langtest import Harness 

harness = Harness(
    task="question-answering",
    model={
        "model": "http://localhost:1234/v1/chat/completions",
        "hub": "lm-studio",
    },
    data={
        "data_source": "path/to/file.pkl", #
    },
)
# generate, run and report
harness.generate().run().report()

Multi-Dataset Handling and Evaluation

This feature empowers you to efficiently benchmark your language models across a wider range of datasets.

Key Features:

• Effortless Multi-Dataset Testing: LangTest now seamlessly integrates and executes tests on multiple datasets within a single harness configuration. This streamlined approach eliminates the need for repetitive setups, saving you time and resources.

• Enhanced Fairness Evaluation: By testing models across diverse datasets, LangTest helps identify and mitigate potential biases. This ensures your models perform fairly and accurately on a broader spectrum of data, promoting ethical and responsible AI development.

• Robust Accuracy Assessment: Multi-dataset support empowers you to conduct more rigorous accuracy testing. By evaluating models on various datasets, you gain a deeper understanding of their strengths and weaknesses across different data distributions. This comprehensive analysis strengthens your confidence in the model's real-world performance.

How it works:

Initiate the Harness class

harness = Harness(
    task="question-answering",
    model={"model": "gpt-3.5-turbo-instruct", "hub": "openai"},
    data=[
        {"data_source": "BoolQ", "split": "test-tiny"},
        {"data_source": "NQ-open", "split": "test-tiny"},
        {"data_source": "MedQA", "split": "test-tiny"},
        {"data_source": "LogiQA", "split": "test-tiny"},
    ],
)

Configure the accuracy tests in Harness class

harness.configure(
    {
        "tests": {
            "defaults": {"min_pass_rate": 0.65},
            "robustness": {
                "uppercase": {"min_pass_rate": 0.66},
                "dyslexia_word_swap": {"min_pass_rate": 0.60},
                "add_abbreviation": {"min_pass_rate": 0.60},
                "add_slangs": {"min_pass_rate": 0.60},
                "add_speech_to_text_typo": {"min_pass_rate": 0.60},
            },
        }
    }
)

harness.generate() generates testcases, .run() executes them, and .report() compiles results.

harness.generate().run().report()

Streamlined Evaluation Workflows with Enhanced CLI Commands

LangTest's evaluation capabilities, focusing on report management and leaderboards. These enhancements empower you to:

• Streamlined Reporting and Tracking: Effortlessly save and load detailed evaluation reports directly from the command line using langtest eval, enabling efficient performance tracking and comparative analysis over time, with manual file review options in the ~/.langtest or ./.langtest folder.

• Enhanced Leaderboards: Gain valuable insights with the new langtest show-leaderboard command. This command displays existing leaderboards, providing a centralized view of ranked model performance across evaluations.

• Average Model Ranking: Leaderboard now includes the average ranking for each evaluated model. This metric provides a comprehensive understanding of model performance across various datasets and tests.

How it works:

First, create the parameter.json or parameter.yaml in the working directory

JSON Format

{
    "task": "question-answering",
    "model": {
        "model": "google/flan-t5-base",
        "hub": "huggingface"
    },
    "data": [
        {
            "data_source": "MedMCQA"
        },
        {
            "data_source": "PubMedQA"
        },
        {
            "data_source": "MMLU"
        },
        {
            "data_source": "MedQA"
   ...

John Snow Labs LangTest 2.0.0: Comprehensive Model Benchmarking, Added support for LM Studio , CLI Integration for Embedding Benchmarks, Enhanced Toxicity Tests, Multi-Dataset Comparison and elevated user experience with various bug fixes and enhancements.

---

📢 Highlights

🌟 LangTest 2.0.0 Release by John Snow Labs

We're thrilled to announce the latest release of LangTest, introducing remarkable features that elevate its capabilities and user-friendliness. This update brings a host of enhancements:

• 🔬 Model Benchmarking: Conducted tests on diverse models across datasets for insights into performance.

• 🔌 Integration: LM Studio with LangTest: Offline utilization of Hugging Face quantized models for local NLP tests.

• 🚀 Text Embedding Benchmark Pipelines: Streamlined process for evaluating text embedding models via CLI.

• 📊 Compare Models Across Multiple Benchmark Datasets: Simultaneous evaluation of model efficacy across diverse datasets.

• 🤬 Custom Toxicity Checks: Tailor evaluations to focus on specific types of toxicity, offering detailed analysis in targeted areas of concern, such as obscenity, insult, threat, identity attack, and targeting based on sexual orientation, while maintaining broader toxicity detection capabilities.

• Implemented LRU caching within the run method to optimize model prediction retrieval for duplicate records, enhancing runtime efficiency.

🔥 Key Enhancements:

🚀 Model Benchmarking: Exploring Insights into Model Performance

As part of our ongoing Model Benchmarking initiative, we're excited to share the results of our comprehensive tests on a diverse range of models across various datasets, focusing on evaluating their performance on top of accuracy and robustness .

Key Highlights:

• Comprehensive Evaluation: Our rigorous testing methodology covered a wide array of models, providing a holistic view of their performance across diverse datasets and tasks.

• Insights into Model Behavior: Through this initiative, we've gained valuable insights into the strengths and weaknesses of different models, uncovering areas where even large language models exhibit limitations.

Go to: Leaderboard

Benchmark Datasets	Split	Test	Models Tested
ASDiV	Test	Accuracy & Robustness	Deci/DeciLM-7B-instruct, TheBloke/Llama-2-7B-chat-GGUF, TheBloke/SOLAR-10.7B-Instruct-v1.0-GGUF, TheBloke/neural-chat-7B-v3-1-GGUF, TheBloke/openchat_3.5-GGUF, TheBloke/phi-2-GGUF, google/flan-t5-xxl, gpt-3.5-turbo-instruct, gpt-4-1106-preview, mistralai/Mistral-7B-Instruct-v0.1, mistralai/Mixtral-8x7B-Instruct-v0.1
BBQ	Test	Accuracy & Robustness	Deci/DeciLM-7B-instruct, TheBloke/Llama-2-7B-chat-GGUF, TheBloke/SOLAR-10.7B-Instruct-v1.0-GGUF, TheBloke/neural-chat-7B-v3-1-GGUF, TheBloke/openchat_3.5-GGUF, TheBloke/phi-2-GGUF, google/flan-t5-xxl, gpt-3.5-turbo-instruct, gpt-4-1106-preview, mistralai/Mistral-7B-Instruct-v0.1, mistralai/Mixtral-8x7B-Instruct-v0.1
BigBench (3 subsets)	Test	Accuracy & Robustness	Deci/DeciLM-7B-instruct, TheBloke/Llama-2-7B-chat-GGUF, TheBloke/SOLAR-10.7B-Instruct-v1.0-GGUF, TheBloke/neural-chat-7B-v3-1-GGUF, TheBloke/openchat_3.5-GGUF, TheBloke/phi-2-GGUF, google/flan-t5-xxl, gpt-3.5-turbo-instruct, gpt-4-1106-preview, mistralai/Mistral-7B-Instruct-v0.1, mistralai/Mixtral-8x7B-Instruct-v0.1
BoolQ	dev	Accuracy	Deci/DeciLM-7B-instruct, TheBloke/Llama-2-7B-chat-GGUF, TheBloke/SOLAR-10.7B-Instruct-v1.0-GGUF, TheBloke/neural-chat-7B-v3-1-GGUF, TheBloke/openchat_3.5-GGUF, TheBloke/phi-2-GGUF, google/flan-t5-xxl, gpt-3.5-turbo-instruct, gpt-4-1106-preview, mistralai/Mistral-7B-Instruct-v0.1, mistralai/Mixtral-8x7B-Instruct-v0.1
BoolQ	Test	Robustness	Deci/DeciLM-7B-instruct, TheBloke/Llama-2-7B-chat-GGUF, TheBloke/SOLAR-10.7B-Instruct-v1.0-GGUF, TheBloke/neural-chat-7B-v3-1-GGUF, TheBloke/openchat_3.5-GGUF, TheBloke/phi-2-GGUF, google/flan-t5-xxl, gpt-3.5-turbo-instruct, gpt-4-1106-preview, mistralai/Mistral-7B-Instruct-v0.1, mistralai/Mixtral-8x7B-Instruct-v0.1
CommonSenseQA	Test	Robustness	Deci/DeciLM-7B-instruct, TheBloke/Llama-2-7B-chat-GGUF, TheBloke/SOLAR-10.7B-Instruct-v1.0-GGUF, TheBloke/neural-chat-7B-v3-1-GGUF, TheBloke/openchat_3.5-GGUF, TheBloke/phi-2-GGUF, google/flan-t5-xxl, gpt-3.5-turbo-instruct, gpt-4-1106-preview, mistralai/Mistral-7B-Instruct-v0.1, mistralai/Mixtral-8x7B-Instruct-v0.1
CommonSenseQA	Val	Accuracy	Deci/DeciLM-7B-instruct, TheBloke/Llama-2-7B-chat-GGUF, TheBloke/SOLAR-10.7B-Instruct-v1.0-GGUF, TheBloke/neural-chat-7B-v3-1-GGUF, TheBloke/openchat_3.5-GGUF, TheBloke/phi-2-GGUF, google/flan-t5-xxl, gpt-3.5-turbo-instruct, gpt-4-1106-preview, mistralai/Mistral-7B-Instruct-v0.1, mistralai/Mixtral-8x7B-Instruct-v0.1
Consumer-Contracts	Test	Accuracy & Robustness	Deci/DeciLM-7B-instruct, TheBloke/Llama-2-7B-chat-GGUF, TheBloke/SOLAR-10.7B-Instruct-v1.0-GGUF, TheBloke/neural-chat-7B-v3-1-GGUF, TheBloke/openchat_3.5-GGUF, TheBloke/phi-2-GGUF, google/flan-t5-xxl, gpt-3.5-turbo-instruct, gpt-4-1106-preview, mistralai/Mistral-7B-Instruct-v0.1, mistralai/Mixtral-8x7B-Instruct-v0.1
Contracts	Test	Accuracy & Robustness	Deci/DeciLM-7B-instruct, TheBloke/Llama-2-7B-chat-GGUF, TheBloke/SOLAR-10.7B-Instruct-v1.0-GGUF, TheBloke/neural-chat-7B-v3-1-GGUF, TheBloke/openchat_3.5-GGUF, TheBloke/phi-2-GGUF, google/flan-t5-xxl, gpt-3.5-turbo-instruct, gpt-4-1106-preview, mistralai/Mistral-7B-Instruct-v0.1, mistralai/Mixtral-8x7B-Instruct-v0.1
LogiQA	Test	Accuracy & Robustness	Deci/DeciLM-7B-instruct, TheBloke/Llama-2-7B-chat-GGUF, TheBloke/SOLAR-10.7B-Instruct-v1.0-GGUF, TheBloke/neural-chat-7B-v3-1-GGUF, TheBloke/openchat_3.5-GGUF, TheBloke/phi-2-GGUF, google/flan-t5-xxl, gpt-3.5-turbo-instruct, gpt-4-1106-preview, mistralai/Mistral-7B-Instruct-v0.1, mistralai/Mixtral-8x7B-Instruct-v0.1
MMLU	Clinical	Accuracy & Robustness	Deci/DeciLM-7B-instruct, TheBloke/Llama-2-7B-chat-GGUF, TheBloke/SOLAR-10.7B-Instruct-v1.0-GGUF, TheBloke/neural-chat-7B-v3-1-GGUF, TheBloke/openchat_3.5-GGUF, TheBloke/phi-2-GGUF, google/flan-t5-xxl, gpt-3.5-turbo-instruct, gpt-4-1106-preview, mistralai/Mistral-7B-Instruct-v0.1, mistralai/Mixtral-8x7B-Instruct-v0.1
MedMCQA (20-Subsets )	test	Robustness	Deci/DeciLM-7B-instruct, TheBloke/Llama-2-7B-chat-GGUF, TheBloke/SOLAR-10.7B-Instruct-v1.0-GGUF, TheBloke/neural-chat-7B-v3-1-GGUF, TheBloke/openchat_3.5-GGUF, TheBloke/phi-2-GGUF, google/flan-t5-xxl, gpt-3.5-turbo-instruct, gpt-4-1106-preview, mistralai/Mistral-7B-Instruct-v0.1, mistralai/Mixtral-8x7B-Instruct-v0.1
MedMCQA (20-Subsets )	val	Accuracy	Deci/DeciLM-7B-instruct, TheBloke/Llama-2-7B-chat-GGUF, TheBloke/SOLAR-10.7B-Instruct-v1.0-GGUF, TheBloke/neural-chat-7B-v3-1-GGUF, TheBloke/openchat_3.5-GGUF, TheBloke/phi-2-GGUF, google/flan-t5-xxl, gpt-3.5-turbo-instruct, gpt-4-1106-preview, mistralai/Mistral-7B-Instruct-v0.1, mistralai/Mixtral-8x7B-Instruct-v0.1
MedQA	test	Accuracy & Robustness	Deci/DeciLM-7B-instruct, TheBloke/Llama-2-7B-chat-GGUF, TheBloke/SOLAR-10.7B-Instruct-v1.0-GGUF, TheBloke/neural-chat-7B-v3-1-GGUF, TheBloke/openchat_3.5-GGUF, TheBloke/phi-2-GGUF, google/flan-t5-xxl, gpt-3.5-turbo-instruct, gpt-4-1106-preview, mistralai/Mistral-7B-Instruct-v0.1, mistralai/Mixtral-8x7B-Instruct-v0.1
OpenBookQA	test	Accuracy & Robustness	Deci/DeciLM-7B-instruct, TheBloke/Llama-2-7B-chat-GGUF, TheBloke/SOLAR-10.7B-Instruct-v1.0-GGUF, TheBloke/neural-chat-7B-v3-1-GGUF, TheBloke/openchat_3.5-GGUF, TheBloke/phi-2-GGUF, google/flan-t5-xxl, gpt-3.5-turbo-instruct, gpt-4-1106-preview, mistralai/Mistral-7B-Instruct-v0.1, mistralai/Mixtral-8x7B-Instruct-v0.1
PIQA	test	Robustness	Deci/DeciLM-7B-instruct, TheBloke/Llama-2-7B-chat-GGUF, TheBloke/SOLAR-10.7B-Instruct-v1.0-GGUF, TheBloke/neural-chat-7B-v3-1-GGUF, TheBloke/openchat_3.5-GGUF, TheBloke/phi-2-GGUF, google/flan-t5-xxl, gpt-3.5-turbo-instruct, gpt-4-1106-preview, mistralai/Mistral-7B-Instruct-v0.1, mistralai/Mixtral-8x7B-Instruct-v0.1
PIQA	val	Accuracy	Deci/DeciLM-7B-instruct, TheBloke/Llama-2-7B-chat-GGUF, TheBloke/SOLAR-10.7B-Instruct-v1.0-GGUF, TheBloke/neural-chat-7B-v3-1-GGUF, TheBloke/openchat_3.5-GGUF, TheBloke/phi-2-GGUF, google/flan-t5-xxl, gpt-3.5-turbo-instruct, gpt-4-1106-preview, mistralai/Mistral-7B-Instruct-v0.1, mistralai/Mixtral-8x7B-Instruct-v0.1
PubMedQA (2-Subsets)	test	Accuracy & Robustness	Deci/DeciLM-7B-instruct, TheBloke/Llama-2-7B-chat-GGUF, TheBloke/SOLAR-10.7B-Instruct-v1.0-GGUF, TheBloke/neural-chat-7B-v3-1-GGUF, TheBloke/openchat_3.5-GGUF, TheBloke/phi-2-GGUF, google/flan-t5-xxl, gpt-3.5-turbo-instruct, gpt-4-1106-preview, mistralai/Mistral-7B-Instruct-v0.1, mistralai/Mixtral-8x7B-Instruct-v0.1
SIQA	test	Accuracy & Robustness	Deci/DeciLM-7B-instruct, TheBloke/Llama-2-7B-chat-GGUF, TheBloke/SOLAR-10.7B-Instruct-v1.0-GGUF, TheBloke/neural-chat-7B-v3-1-GGUF, TheBloke/openchat_3.5-GGUF, TheBloke/phi-2-GGUF, google/flan-t5-xxl, gpt-3.5-turbo-instruct, gpt-4-1106-preview, mistralai/Mistral-7B-Instruct-v0.1, mistralai/Mixtral-8x7B-Instruct-v0.1
TruthfulQA	test	Accuracy & Robustness	...

John Snow Labs LangTest 1.10.0: Support for Evaluating RAG with LlamaIndex and Langtest, Grammar Testing, Robust Checkpoint Managememt, and Comprehensive Support for Medical Datasets (LiveQA, MedicationQA, HealthSearchQA), Direct Hugging Face Model Integration and Elevated User Experience with Numerous Bug Fixes !

---

📢 Highlights

🌟 LangTest 1.10.0 Release by John Snow Labs

We're thrilled to announce the latest release of LangTest, introducing remarkable features that elevate its capabilities and user-friendliness. This update brings a host of enhancements:

• Evaluating RAG with LlamaIndex and Langtest: LangTest seamlessly integrates LlamaIndex for constructing a RAG and employs LangtestRetrieverEvaluator, measuring retriever precision (Hit Rate) and accuracy (MRR) with both standard and perturbed queries, ensuring robust real-world performance assessment.

• Grammar Testing for NLP Model Evaluation: This approach entails creating test cases through the paraphrasing of original sentences. The purpose is to evaluate a language model's proficiency in understanding and interpreting the nuanced meaning of the text, enhancing our understanding of its contextual comprehension capabilities.

• Saving and Loading the Checkpoints: LangTest now supports the seamless saving and loading of checkpoints, providing users with the ability to manage task progress, recover from interruptions, and ensure data integrity.

• Extended Support for Medical Datasets: LangTest adds support for additional medical datasets, including LiveQA, MedicationQA, and HealthSearchQA. These datasets enable a comprehensive evaluation of language models in diverse medical scenarios, covering consumer health, medication-related queries, and closed-domain question-answering tasks.

• Direct Integration with Hugging Face Models: Users can effortlessly pass any Hugging Face model object into the LangTest harness and run a variety of tasks. This feature streamlines the process of evaluating and comparing different models, making it easier for users to leverage LangTest's comprehensive suite of tools with the wide array of models available on Hugging Face.

🔥 Key Enhancements:

🚀Implementing and Evaluating RAG with LlamaIndex and Langtest

LangTest seamlessly integrates LlamaIndex, focusing on two main aspects: constructing the RAG with LlamaIndex and evaluating its performance. The integration involves utilizing LlamaIndex's generate_question_context_pairs module to create relevant question and context pairs, forming the foundation for retrieval and response evaluation in the RAG system.

To assess the retriever's effectiveness, LangTest introduces LangtestRetrieverEvaluator, employing key metrics such as Hit Rate and Mean Reciprocal Rank (MRR). Hit Rate gauges the precision by assessing the percentage of queries with the correct answer in the top-k retrieved documents. MRR evaluates the accuracy by considering the rank of the highest-placed relevant document across all queries. This comprehensive evaluation, using both standard and perturbed queries generated through LangTest, ensures a thorough understanding of the retriever's robustness and adaptability under various conditions, reflecting its real-world performance.

from langtest.evaluation import LangtestRetrieverEvaluator

retriever_evaluator = LangtestRetrieverEvaluator.from_metric_names(
    ["mrr", "hit_rate"], retriever=retriever
)
     
retriever_evaluator.setPerturbations("add_typo","dyslexia_word_swap", "add_ocr_typo") 

# Evaluate
eval_results = await retriever_evaluator.aevaluate_dataset(qa_dataset)

retriever_evaluator.display_results()

📚Grammar Testing in Evaluating and Enhancing NLP Models

Grammar Testing is a key feature in LangTest's suite of evaluation strategies, emphasizing the assessment of a language model's proficiency in contextual understanding and nuance interpretation. By creating test cases that paraphrase original sentences, the goal is to gauge the model's ability to comprehend and interpret text, thereby enriching insights into its contextual mastery.

Category	Test Type	Original	Test Case	Expected Result	Actual Result	Pass
grammar	paraphrase	This program was on for a brief period when I was a kid, I remember watching it whilst eating fish and chips. Riding on the back of the Tron hype this series was much in the style of streethawk, manimal and the like, except more computery. There was a geeky kid who's computer somehow created this guy - automan. He'd go around solving crimes and the lot. All I really remember was his fancy car and the little flashy cursor thing that used to draw the car and help him out generally. When I mention it to anyone they can remember very little too. Was it real or maybe a dream?	I remember watching a show from my youth that had a Tron theme, with a nerdy kid driving around with a little flashy cursor and solving everyday problems. Was it a genuine story or a mere dream come true?	NEGATIVE	POSITIVE	false

🔥 Saving and Loading the Checkpoints

Introducing a robust checkpointing system in LangTest! The run method in the Harness class now supports checkpointing, allowing users to save intermediate results, manage batch processing, and specify a directory for storing checkpoints and results. This feature ensures data integrity, providing a mechanism for recovering progress in case of interruptions or task failures.

harness.run(checkpoint=True, batch_size=20,save_checkpoints_dir="imdb-checkpoint")

The load_checkpoints method facilitates the direct loading of saved checkpoints and data, providing a convenient mechanism to resume testing tasks from the point where they were previously interrupted, even in the event of runtime failures or errors.

harness = Harness.load_checkpoints(save_checkpoints_dir="imdb-checkpoint",
                                   task="text-classification",
                                   model = {"model": "lvwerra/distilbert-imdb" , "hub":"huggingface"}, )

🏥 Added Support for More Medical Datasets

LiveQA

The LiveQA'17 medical task focuses on consumer health question answering. It consists of constructed medical question-answer pairs for training and testing, with additional annotations. LangTest now supports LiveQA for comprehensive medical evaluation.

How the dataset looks:

category	test_type	original_question	perturbed_question	expected_result	actual_result	eval_score	pass
robustness	uppercase	Do amphetamine salts 20mg tablets contain gluten?	DO AMPHETAMINE SALTS 20MG TABLETS CONTAIN GLUTEN?	No, amphetamine salts 20mg tablets do not contain gluten.	No, Amphetamine Salts 20mg Tablets do not contain gluten.	1.0	true

MedicationQA

The MedicationQA dataset consists of commonly asked consumer questions about medications. It includes annotations corresponding to drug focus and interactions. LangTest now integrates MedicationQA for thorough evaluation of models in medication-related scenarios.

How the dataset looks:

category	test_type	original_question	perturbed_question	expected_result	actual_result	eval_score	pass
robustness	uppercase	how does rivatigmine and otc sleep medicine interact	HOW DOES RIVATIGMINE AND OTC SLEEP MEDICINE INTERACT	Rivastigmine is a cholinesterase inhibitor and OTC (over-the-counter) sleep medicine is a sedative. There is a risk of excessive sedation when taking both of these medications together. Patients should consult their doctor before taking both of th...

John Snow Labs LangTest 1.9.0: Hugging Face Callback Integration, Advanced Templatic Augmentation, Comprehensive Model Benchmarking, Expanded Clinical Dataset Support (MedQA, PubMedQA, MedMCQ), Insightful Blogposts, and Enhanced User Experience with Key Bug Fixes

---

📢 Highlights

🌟 LangTest 1.9.0 Release by John Snow Labs

We're excited to announce the latest release of LangTest, featuring significant enhancements that bolster its versatility and user-friendliness. This update introduces the seamless integration of Hugging Face Callback, empowering users to effortlessly utilize this renowned platform. Another addition is our Enhanced Templatic Augmentation with Automated Sample Generation. We also expanded LangTest's utility in language testing by conducting comprehensive benchmarks across various models and datasets, offering deep insights into performance metrics. Moreover, the inclusion of additional Clinical Datasets like MedQA, PubMedQA, and MedMCQ broadens our scope to cater to diverse testing needs. Coupled with insightful blog posts and numerous bug fixes, this release further cements LangTest as a robust and comprehensive tool for language testing and evaluation.

• Integration of Hugging Face's callback class in LangTest facilitates seamless incorporation of an automatic testing callback into transformers' training loop for flexible and customizable model training experiences.

• Enhanced Templatic Augmentation with Automated Sample Generation: A key addition in this release is our innovative feature that auto-generates sample templates for templatic augmentation. By setting generate_templates to True, users can effortlessly create structured templates, which can then be reviewed and customized with the show_templates option.

• In our Model Benchmarking initiative, we conducted extensive tests on various models across diverse datasets (MMLU-Clinical, OpenBookQA, MedMCQA, MedQA), revealing insights into their performance and limitations, enhancing our understanding of the landscape for robustness testing.

• Enhancement: Implemented functionality to save model responses (actual and expected results) for original and perturbed questions from the language model (llm) in a pickle file. This enables efficient reuse of model outputs on the same dataset, allowing for subsequent evaluation without the need to rerun the model each time.

• Optimized API Efficiency with Bug Fixes in Model Calls.

🔥 Key Enhancements:

🤗 Hugging Face Callback Integration

We introduced the callback class for utilization in transformers model training. Callbacks in transformers are entities that can tailor the training loop's behavior within the PyTorch or Keras Trainer. These callbacks have the ability to examine the training loop state, make decisions (such as early stopping), or execute actions (including logging, saving, or evaluation). LangTest effectively leverages this capability by incorporating an automatic testing callback. This class is both flexible and adaptable, seamlessly integrating with any transformers model for a customized experience.

Create a callback instance with one line and then use it in the callbacks of trainer:

my_callback = LangTestCallback(...)
trainer = Trainer(..., callbacks=[my_callback])

Parameter	Description
task	Task for which the model is to be evaluated (text-classification or ner)
data	The data to be used for evaluation. A dictionary providing flexibility and options for data sources. It should include the following keys: data_source (mandatory): The source of the data. subset (optional): The subset of the data. feature_column (optional): The column containing the features. target_column (optional): The column containing the target labels. split (optional): The data split to be used. source (optional): Set to 'huggingface' when loading Hugging Face dataset.
config	Configuration for the tests to be performed, specified in the form of a YAML file.
print_reports	A bool value that specifies if the reports should be printed.
save_reports	A bool value that specifies if the reports should be saved.
run_each_epoch	A bool value that specifies if the tests should be run after each epoch or the at the end of training

🚀 Enhanced Templatic Augmentation with Automated Sample Generation

Users can now enable the automatic generation of sample templates by setting generate_templates to True. This feature utilizes the advanced capabilities of LLMs to create structured templates that can be used for templatic augmentation.To ensure quality and relevance, users can review the generated templates by setting show_templates to True.

🚀 Benchmarking Different Models

In our Model Benchmarking initiative, we conducted comprehensive tests on a range of models across diverse datasets. This rigorous evaluation provided valuable insights into the performance of these models, pinpointing areas where even large language models exhibit limitations. By scrutinizing their strengths and weaknesses, we gained a deeper understanding of the landscape

MMLU-Clinical

We focused on extracting clinical subsets from the MMLU dataset, creating a specialized MMLU-clinical dataset. This curated dataset specifically targets clinical domains, offering a more focused evaluation of language understanding models. It includes questions and answers related to clinical topics, enhancing the assessment of models' abilities in medical contexts. Each sample presents a question with four choices, one of which is the correct answer. This curated dataset is valuable for evaluating models' reasoning, fact recall, and knowledge application in clinical scenarios.

How the Dataset Looks

category	test_type	original_question	perturbed_question	expected_result	actual_result	pass
robustness	uppercase	Fatty acids are transported into the mitochondria bound to:\nA. thiokinase. B. coenzyme A (CoA). C. acetyl-CoA. D. carnitine.	FATTY ACIDS ARE TRANSPORTED INTO THE MITOCHONDRIA BOUND TO: A. THIOKINASE. B. COENZYME A (COA). C. ACETYL-COA. D. CARNITINE.	D. carnitine.	B. COENZYME A (COA).	False

OpenBookQA

The OpenBookQA dataset is a collection of multiple-choice questions that require complex reasoning and inference based on general knowledge, similar to an “open-book” exam. The questions are designed to test the ability of natural language processing models to answer questions that go beyond memorizing facts and involve understanding concepts and their relations. The dataset contains 500 questions,
each with four answer choices and one correct answer. The questions cover various topics in science, such as biology, chemistry, physics, and astronomy.

How the Dataset Looks

category	test_type	original_question	perturbed_question	expected_result	actual_result	pass
robustness	uppercase	There is most likely going to be fog around: A. a marsh B. a tundra C. the plains D. a desert"	THERE IS MOST LIKELY GOING TO BE FOG AROUND: A. A MARSH B. A TUNDRA C. THE PLAINS D. A DESERT"	A marsh	A MARSH	True

MedMCQA

The MedMCQA is a large-scale benchmark dataset of Multiple-Choice Question Answering (MCQA) dataset designed to address real-world medical entrance exam questions.

How the Dataset Looks

category	test_type	original_question	perturbed_question	expected_result	actual_result	pass
robustness	uppercase	Most common site of direct hernia\nA. Hesselbach's triangle\nB. Femoral gland\nC. No site predilection\nD. nan	MOST COMMON SITE OF DIRECT HERNIA A. HESSELBACH'S TRIANGLE B. FEMORAL GLAND C. NO SITE PREDILECTION D. NAN	A	A	True

Dataset info:

• subset: MedMCQA-Test
• Split: Medicine, Anatomy, Forensic_Medicine, Microbiology, Pathology, Anaesthesia, Pediatrics, Physiology, Biochemistry, Gynaecology_Obstetrics, Skin, Surgery, Radiology

MedQA

The MedQA is a benchmark dataset of Multiple choice question answering ba...

John Snow Labs LangTest 1.8.0: Codebase Refactoring, Enhanced Debugging with Error Codes, Streamlined Categorization of Tasks, Various Blogposts, Improved Open Source Community Standards and Enhanced User Experience through Multiple Bug Fixes !

---

🌟 LangTest 1.8.0 Release by John Snow Labs

We're thrilled to unveil the latest advancements in LangTest with version 1.8.0. This release is centered around optimizing the codebase with extensive refactoring, enriching the debugging experience through the implementation of error codes, and enhancing workflow efficiency with streamlined task organization. The new categorization approach significantly improves the user experience, ensuring a more cohesive and organized testing process. This update also includes advancements in open source community standards, insightful blog posts, and multiple bug fixes, further solidifying LangTest's reputation as a versatile and user-friendly language testing and evaluation library.

🔥 Key Enhancements:

• Optimized Codebase: This update features a comprehensively refined codebase, achieved through extensive refactoring, resulting in enhanced efficiency and reliability in our testing processes.

• Advanced Debugging Tools: The introduction of error codes marks a significant enhancement in the debugging experience, addressing the previous absence of standardized exceptions. This inconsistency in error handling often led to challenges in issue identification and resolution. The integration of a unified set of standardized exceptions, tailored to specific error types and contexts, guarantees a more efficient and seamless troubleshooting process.

• Task Categorization: This version introduces an improved task organization system, offering a more efficient and intuitive workflow. Previously, it featured a wide range of tests such as sensitivity, clinical tests, wino-bias and many more, each treated as separate tasks. This approach, while comprehensive, could result in a fragmented workflow. The new categorization method consolidates these tests into universally recognized NLP tasks, including Named Entity Recognition (NER), Text Classification, Question Answering, Summarization, Fill-Mask, Translation, and Test Generation. This integration of tests as sub-categories within these broader NLP tasks enhances clarity and reduces potential overlap.

• Open Source Community Standards: With this release, we've strengthened community interactions by introducing issue templates, a code of conduct, and clear repository citation guidelines. The addition of GitHub badges enhances visibility and fosters a collaborative and organized community environment.

• Parameter Standardization: Aiming to bring uniformity in dataset organization and naming, this feature addresses the variation in dataset structures within the repository. By standardizing key parameters like 'datasource', 'split', and 'subset', we ensure a consistent naming convention and organization across all datasets, enhancing clarity and efficiency in dataset usage.

🚀 Community Contributions:

Our team has published three enlightening blogs on Hugging Face's community platform, focusing on bias detection, model sensitivity, and data augmentation in NLP models:

1. Detecting and Evaluating Sycophancy Bias: An Analysis of LLM and AI Solutions
2. Unmasking Language Model Sensitivity in Negation and Toxicity Evaluations
3. Elevate Your NLP Models with Automated Data Augmentation for Enhanced Performance

⭐ Don't forget to give the project a star here!

🚀 New LangTest blogs :

New Blog Posts	Description
Evaluating Large Language Models on Gender-Occupational Stereotypes Using the Wino Bias Test	Delve into the evaluation of language models with LangTest on the WinoBias dataset, addressing AI biases in gender and occupational roles.
Streamlining ML Workflows: Integrating MLFlow Tracking with LangTest for Enhanced Model Evaluations	Discover the revolutionary approach to ML development through the integration of MLFlow and LangTest, enhancing transparency and systematic tracking of models.
Testing the Question Answering Capabilities of Large Language Models	Explore the complexities of evaluating Question Answering (QA) tasks using LangTest's diverse evaluation methods.
Evaluating Stereotype Bias with LangTest	In this blog post, we are focusing on using the StereoSet dataset to assess bias related to gender, profession, and race.

---

🐛 Bug Fixes

• Fixed templatic augmentations PR #851
• Resolved a bug in default configurations PR #880
• Addressed compatibility issues between OpenAI (version 1.1.1) and Langchain PR #877
• Fixed errors in sycophancy-test, factuality-test, and augmentation PR #869

---

What's Changed

• Fix/templatic augmentations by @ArshaanNazir in #851
• Refactor/report section by @ArshaanNazir in #860
• Integrating error codes by @Prikshit7766 in #867
• Refactor/delete dead code by @chakravarthik27 in #744
• updated Evaluation_Metrics notebook by @Prikshit7766 in #861
• fix rc errors by @ArshaanNazir in #868
• Update issue templates by @RakshitKhajuria in #862
• Created CODE_OF_CONDUCT.md by @RakshitKhajuria in #863
• Refactor/add configurable parameters by @alytarik in #866
• Added citation for the repo by @RakshitKhajuria in #871
• resolved: errors in sycophancy-test, factuality-test and augmentation. by @chakravarthik27 in #869
• Compatibility issue OpenAI (version 1.1.1) and Langchain by @Prikshit7766 in #877
• Feature/task categorization by @chakravarthik27 in #878
• Standardize qa dataset naming and structure by @Prikshit7766 in #876
• Investigate TestFactory.task for Task Transition Errors by @Prikshit7766 in #873
• updated wino evaluation by @RakshitKhajuria in #859
• Chore/notebook updates by @ArshaanNazir in #879
• Fix bug in default configs by @chakravarthik27 in #880
• fix default config by @ArshaanNazir in #881
• Fix: Update load_model method to accept a path instead in custom hub by @chakravarthik27 in #882
• Website Updates by @RakshitKhajuria in #875
• Release/1.8.0 by @ArshaanNazir in #883

Full Changelog: 1.7.0...v1.8.0