Merge branch 'main' of https://github.com/kyegomez/VisionDatasets

.github/workflows/code_quality_control.yml

@@ -11,7 +11,7 @@ jobs:
 
     steps:
       - name: Checkout code
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - name: Set up Python
         uses: actions/setup-python@v5

.github/workflows/cos_integration.yml

@@ -10,7 +10,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout code
-        uses: actions/checkout@v2
+        uses: actions/checkout@v4
 
       - name: Set up Python
         uses: actions/setup-python@v5

.github/workflows/docs.yml

@@ -10,7 +10,7 @@ jobs:
   deploy:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - uses: actions/setup-python@v5
         with:
           python-version: 3.x

.github/workflows/docs_test.yml

@@ -11,7 +11,7 @@ jobs:
 
     steps:
       - name: Checkout code
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - name: Set up Python
         uses: actions/setup-python@v5

.github/workflows/lints.yml

@@ -11,7 +11,7 @@ jobs:
 
     steps:
       - name: Checkout code
-        uses: actions/checkout@v2
+        uses: actions/checkout@v4
 
       - name: Set up Python
         uses: actions/setup-python@v5

.github/workflows/pr_request_checks.yml

@@ -11,7 +11,7 @@ jobs:
 
     steps:
       - name: Checkout code
-        uses: actions/checkout@v2
+        uses: actions/checkout@v4
 
       - name: Set up Python
         uses: actions/setup-python@v5

.github/workflows/testing.yml

@@ -11,7 +11,7 @@ jobs:
 
     steps:
       - name: Checkout code
-        uses: actions/checkout@v2
+        uses: actions/checkout@v4
 
       - name: Set up Python
         uses: actions/setup-python@v5

example.py

@@ -0,0 +1,13 @@
+from vision_datasets.iter_over_dataset import ImageDatasetIterator
+from swarms import QwenVLMultiModal
+
+model = QwenVLMultiModal(system_prompt="You, as the model, are presented with a visual problem. This could be an image containing various elements that you need to analyze, a graph that requires interpretation, or a visual puzzle. Your task is to examine the visual information carefully and describe your process of understanding and solving the problem.",)
+
+iterator = ImageDatasetIterator(
+    "coco",
+    model
+)
+
+
+# Run the iterator
+iterator.run()

vision_datasets/__init__.py

@@ -0,0 +1,3 @@
+from vision_datasets.iter_over_dataset import ImageDatasetIterator
+
+__all__ = ["ImageDatasetIterator"]

vision_datasets/iter_over_dataset.py

@@ -0,0 +1,123 @@
+from typing import Any, Callable, Iterator
+from datasets import load_dataset, DatasetDict
+from PIL import Image
+import json
+
+
+class ImageDatasetIterator:
+    """
+    An iterator class for processing images in a dataset.
+
+    Args:
+        dataset_name (str): The name of the dataset.
+        model (Callable): The model function to process the images.
+        json_file (str): The path to the JSON file.
+
+    Attributes:
+        dataset: The dataset to iterate over.
+        model: The model function to process the images.
+        index (int): The current index of the iterator.
+
+    Methods:
+        __iter__(): Returns the iterator object.
+        __next__(): Returns the next image in the dataset.
+        process_images(): Processes the images in the dataset using the model function.
+        process_images_to_json(json_file: str): Processes the images and writes the output to a JSON file.
+        run(json_file: str): Runs the image processing and writes the output to a JSON file.
+        create_and_run(dataset_name: str, model: Callable[[Image.Image], Any], json_file: str): Creates an instance of ImageDatasetIterator and runs the image processing.
+
+    """
+
+    def __init__(
+        self,
+        dataset_name: str,
+        model: Callable,
+        json_file: str = "vision_datasets.json",
+    ):
+        self.dataset_name = dataset_name
+        self.model = model
+        self.json_file = json_file
+
+        self.dataset = load_dataset(dataset_name)
+        self.index = 0
+
+    def __iter__(self) -> Iterator[Image.Image]:
+        return self
+
+    def __next__(self) -> Image.Image:
+        if self.index >= len(self.dataset):
+            raise StopIteration
+        image_data = self.dataset[self.index]["image"]
+        image = Image.fromarray(image_data)
+        self.index += 1
+        return image
+
+    def process_images(self, task: str) -> Iterator[Any]:
+        """
+        Processes the images in the dataset using the model function.
+
+        Yields:
+            Any: The output of the model function for each image.
+
+        """
+        self.index = 0  # Reset index to start from the beginning
+        for image in self:
+            output = self.model(task, image)  # Pass image into the model
+            yield output
+
+    def process_images_to_json(self) -> None:
+        """
+        Processes the images and writes the output to a JSON file.
+
+        Args:
+            json_file (str): The path to the JSON file.
+
+        """
+        with open(self.json_file, "w") as f:
+            for output in self.process_images():
+                json.dump(output, f)
+                f.write("\n")  # Write each output on a new line
+
+    def run(self) -> None:
+        """
+        Runs the image processing and writes the output to a JSON file.
+
+        Args:
+            json_file (str): The path to the JSON file.
+
+        """
+        self.process_images_to_json(self.json_file)
+
+    def upload_to_huggingface(self, dataset_path: str) -> None:
+        """
+        Uploads the dataset to Hugging Face.
+
+        Args:
+            dataset_path (str): The path to save the dataset to.
+
+        """
+        dataset_dict = DatasetDict({"train": self.dataset})
+        dataset_dict.save_to_disk(dataset_path)
+
+    @classmethod
+    def create_and_run(
+        self,
+        cls,
+        dataset_name: str,
+        model: Callable[[Image.Image], Any],
+    ) -> "ImageDatasetIterator":
+        """
+        Creates an instance of ImageDatasetIterator and runs the image processing.
+
+        Args:
+            dataset_name (str): The name of the dataset.
+            model (Callable): The model function to process the images.
+            json_file (str): The path to the JSON file.
+
+        Returns:
+            ImageDatasetIterator: The created instance of ImageDatasetIterator.
+
+        """
+        processor = cls(dataset_name, model)
+        processor.run(self.json_file)
+        return processor

vision_datasets/sop.py

@@ -0,0 +1,36 @@
+VISUAL_CHAIN_OF_THOUGHT = """
+    
+You, as the model, are presented with a visual problem. This could be an image containing various elements that you need to analyze, a graph that requires interpretation, or a visual puzzle. Your task is to examine the visual information carefully and describe your process of understanding and solving the problem.
+
+Instructions:
+
+Observation: Begin by describing what you see in the image. Break down the visual elements into understandable segments. For instance, if it's a picture of a street, identify the key components like cars, buildings, people, street signs, etc. If it's a graph, start by outlining its type, the axes, and the data it presents.
+
+Initial Analysis: Based on your observation, start analyzing the image. If it's a scene, narrate the possible context or the story the image might be telling. If it's a graph or data, begin to interpret what the data might indicate. This step is about forming hypotheses or interpretations based on visual cues.
+
+Detailed Reasoning: Delve deeper into your analysis. This is where the chain of thought becomes critical. If you're looking at a scene, consider the relationships between elements. Why might that person be running? What does the traffic signal indicate? For graphs or data-driven images, analyze trends, outliers, and correlations. Explain your thought process in a step-by-step manner.
+
+Visual References: As you explain, make visual references. Draw arrows, circles, or use highlights in the image to pinpoint exactly what you're discussing. These annotations should accompany your verbal reasoning, adding clarity to your explanations.
+
+Conclusion or Solution: Based on your detailed reasoning, draw a conclusion or propose a solution. If it's a visual puzzle or problem, present your answer clearly, backed by the reasoning you've just outlined. If it’s an open-ended image, summarize your understanding of the scene or the data.
+
+Reflection: Finally, reflect on your thought process. Was there anything particularly challenging or ambiguous? How confident are you in your interpretation or solution, and why? This step is about self-assessment and providing insight into your reasoning confidence.
+
+Example:
+
+Let’s say the image is a complex graph showing climate change data over the last century.
+
+Observation: "The graph is a line graph with time on the x-axis and average global temperature on the y-axis. There are peaks and troughs, but a general upward trend is visible."
+
+Initial Analysis: "The immediate observation is that average temperatures have risen over the last century. There are fluctuations, but the overall direction is upward."
+
+Detailed Reasoning: "Looking closer, the steepest increase appears post-1950. This aligns with industrial advancements globally, suggesting a link between human activity and rising temperatures. The short-term fluctuations could be due to natural climate cycles, but the long-term trend indicates a more worrying, human-induced climate change pattern."
+
+Visual References: "Here [draws arrow], the graph shows a sharp rise. The annotations indicate major industrial events, aligning with these spikes."
+
+Conclusion or Solution: "The data strongly suggests a correlation between industrialization and global warming. The upward trend, especially in recent decades, indicates accelerating temperature increases."
+
+Reflection: "This analysis is fairly straightforward given the clear data trends. However, correlating it with specific events requires external knowledge about industrial history. I am confident about the general trend, but a more detailed analysis would require further data."    
+    
+    
+"""