chore: remove extra todos

contrib/CONTRIBUTING.md

@@ -3,6 +3,7 @@
 The following is a set of guidelines for contributing to the Bittensor ecosystem. These are **HIGHLY RECOMMENDED** guidelines, but not hard-and-fast rules. Use your best judgment, and feel free to propose changes to this document in a pull request.
 
 ## Table Of Contents
+
 1. [How Can I Contribute?](#how-can-i-contribute)
    1. [Communication Channels](#communication-channels)
    1. [Code Contribution General Guideline](#code-contribution-general-guidelines)
@@ -12,18 +13,20 @@ The following is a set of guidelines for contributing to the Bittensor ecosystem
    1. [Squashing Commits](#squashing-commits)
    1. [Refactoring](#refactoring)
    1. [Peer Review](#peer-review)
- 1. [Suggesting Features](#suggesting-enhancements-and-features)
-
+1. [Suggesting Features](#suggesting-enhancements-and-features)
 
 ## How Can I Contribute?
+
 TODO(developer): Define your desired contribution procedure.
 
 ## Communication Channels
+
 TODO(developer): Place your communication channels here
 
-> Please follow the Bittensor Subnet [style guide](./STYLE.md) regardless of your contribution type. 
+> Please follow the Bittensor Subnet [style guide](./STYLE.md) regardless of your contribution type.
 
 Here is a high-level summary:
+
 - Code consistency is crucial; adhere to established programming language conventions.
 - Use `black` to format your Python code; it ensures readability and consistency.
 - Write concise Git commit messages; summarize changes in ~50 characters.
@@ -40,16 +43,16 @@ Here is a high-level summary:
 
 ### Code Contribution General Guidelines
 
-> Review the Bittensor Subnet [style guide](./STYLE.md) and [development workflow](./DEVELOPMENT_WORKFLOW.md) before contributing. 
-
+> Review the Bittensor Subnet [style guide](./STYLE.md) and [development workflow](./DEVELOPMENT_WORKFLOW.md) before contributing.
 
 #### Pull Request Philosophy
 
-Patchsets and enhancements should always be focused. A pull request could add a feature, fix a bug, or refactor code, but it should not contain a mixture of these. Please also avoid 'super' pull requests which attempt to do too much, are overly large, or overly complex as this makes review difficult. 
+Patchsets and enhancements should always be focused. A pull request could add a feature, fix a bug, or refactor code, but it should not contain a mixture of these. Please also avoid 'super' pull requests which attempt to do too much, are overly large, or overly complex as this makes review difficult.
 
 Specifically, pull requests must adhere to the following criteria:
+
 - Contain fewer than 50 files. PRs with more than 50 files will be closed.
-- If a PR introduces a new feature, it *must* include corresponding tests.
+- If a PR introduces a new feature, it _must_ include corresponding tests.
 - Other PRs (bug fixes, refactoring, etc.) should ideally also have tests, as they provide proof of concept and prevent regression.
 - Categorize your PR properly by using GitHub labels. This aids in the review process by informing reviewers about the type of change at a glance.
 - Make sure your code includes adequate comments. These should explain why certain decisions were made and how your changes work.
@@ -58,27 +61,29 @@ Specifically, pull requests must adhere to the following criteria:
 
 Generally, all pull requests must:
 
-  - Have a clear use case, fix a demonstrable bug or serve the greater good of the project (e.g. refactoring for modularisation).
-  - Be well peer-reviewed.
-  - Follow code style guidelines.
-  - Not break the existing test suite.
-  - Where bugs are fixed, where possible, there should be unit tests demonstrating the bug and also proving the fix.
-  - Change relevant comments and documentation when behaviour of code changes.
+- Have a clear use case, fix a demonstrable bug or serve the greater good of the project (e.g. refactoring for modularisation).
+- Be well peer-reviewed.
+- Follow code style guidelines.
+- Not break the existing test suite.
+- Where bugs are fixed, where possible, there should be unit tests demonstrating the bug and also proving the fix.
+- Change relevant comments and documentation when behaviour of code changes.
 
 #### Pull Request Process
 
 Please follow these steps to have your contribution considered by the maintainers:
 
-*Before* creating the PR:
+_Before_ creating the PR:
+
 1. Read the [development workflow](./DEVELOPMENT_WORKFLOW.md) defined for this repository to understand our workflow.
 2. Ensure your PR meets the criteria stated in the 'Pull Request Philosophy' section.
 3. Include relevant tests for any fixed bugs or new features as stated in the [testing guide](./TESTING.md).
 4. Ensure your commit messages are clear and concise. Include the issue number if applicable.
 5. If you have multiple commits, rebase them into a single commit using `git rebase -i`.
 6. Explain what your changes do and why you think they should be merged in the PR description consistent with the [style guide](./STYLE.md).
 
-*After* creating the PR:
-1. Verify that all [status checks](https://help.github.com/articles/about-status-checks/) are passing after you submit your pull request. 
+_After_ creating the PR:
+
+1. Verify that all [status checks](https://help.github.com/articles/about-status-checks/) are passing after you submit your pull request.
 2. Label your PR using GitHub's labeling feature. The labels help categorize the PR and streamline the review process.
 3. Document your code with comments that provide a clear understanding of your changes. Explain any non-obvious parts of your code or design decisions you've made.
 4. If your PR has extensive changes, consider splitting it into smaller, related PRs. This reduces the cognitive load on the reviewers and speeds up the review process.
@@ -99,7 +104,7 @@ After you submit a pull request, it will be reviewed by the maintainers. They ma
 > Note: Be sure to merge the latest from "upstream" before making a pull request:
 
 ```bash
-git remote add upstream https://github.com/opentensor/bittensor.git # TODO(developer): replace with your repo URL
+git remote add upstream https://github.com/masa-finance/masa-bittensor.git
 git fetch upstream
 git merge upstream/<your-branch-name>
 git push origin <your-branch-name>
@@ -165,29 +170,29 @@ Please track bugs as GitHub issues.
 
 Explain the problem and include additional details to help maintainers reproduce the problem:
 
-* **Use a clear and descriptive title** for the issue to identify the problem.
-* **Describe the exact steps which reproduce the problem** in as many details as possible. For example, start by explaining how you started the application, e.g. which command exactly you used in the terminal, or how you started Bittensor otherwise. When listing steps, **don't just say what you did, but explain how you did it**. For example, if you ran with a set of custom configs, explain if you used a config file or command line arguments. 
-* **Provide specific examples to demonstrate the steps**. Include links to files or GitHub projects, or copy/pasteable snippets, which you use in those examples. If you're providing snippets in the issue, use [Markdown code blocks](https://help.github.com/articles/markdown-basics/#multiple-lines).
-* **Describe the behavior you observed after following the steps** and point out what exactly is the problem with that behavior.
-* **Explain which behavior you expected to see instead and why.**
-* **Include screenshots and animated GIFs** which show you following the described steps and clearly demonstrate the problem. You can use [this tool](https://www.cockos.com/licecap/) to record GIFs on macOS and Windows, and [this tool](https://github.com/colinkeenan/silentcast) or [this tool](https://github.com/GNOME/byzanz) on Linux.
-* **If you're reporting that Bittensor crashed**, include a crash report with a stack trace from the operating system. On macOS, the crash report will be available in `Console.app` under "Diagnostic and usage information" > "User diagnostic reports". Include the crash report in the issue in a [code block](https://help.github.com/articles/markdown-basics/#multiple-lines), a [file attachment](https://help.github.com/articles/file-attachments-on-issues-and-pull-requests/), or put it in a [gist](https://gist.github.com/) and provide link to that gist.
-* **If the problem is related to performance or memory**, include a CPU profile capture with your report, if you're using a GPU then include a GPU profile capture as well. Look into the [PyTorch Profiler](https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html) to look at memory usage of your model.
-* **If the problem wasn't triggered by a specific action**, describe what you were doing before the problem happened and share more information using the guidelines below.
+- **Use a clear and descriptive title** for the issue to identify the problem.
+- **Describe the exact steps which reproduce the problem** in as many details as possible. For example, start by explaining how you started the application, e.g. which command exactly you used in the terminal, or how you started Bittensor otherwise. When listing steps, **don't just say what you did, but explain how you did it**. For example, if you ran with a set of custom configs, explain if you used a config file or command line arguments.
+- **Provide specific examples to demonstrate the steps**. Include links to files or GitHub projects, or copy/pasteable snippets, which you use in those examples. If you're providing snippets in the issue, use [Markdown code blocks](https://help.github.com/articles/markdown-basics/#multiple-lines).
+- **Describe the behavior you observed after following the steps** and point out what exactly is the problem with that behavior.
+- **Explain which behavior you expected to see instead and why.**
+- **Include screenshots and animated GIFs** which show you following the described steps and clearly demonstrate the problem. You can use [this tool](https://www.cockos.com/licecap/) to record GIFs on macOS and Windows, and [this tool](https://github.com/colinkeenan/silentcast) or [this tool](https://github.com/GNOME/byzanz) on Linux.
+- **If you're reporting that Bittensor crashed**, include a crash report with a stack trace from the operating system. On macOS, the crash report will be available in `Console.app` under "Diagnostic and usage information" > "User diagnostic reports". Include the crash report in the issue in a [code block](https://help.github.com/articles/markdown-basics/#multiple-lines), a [file attachment](https://help.github.com/articles/file-attachments-on-issues-and-pull-requests/), or put it in a [gist](https://gist.github.com/) and provide link to that gist.
+- **If the problem is related to performance or memory**, include a CPU profile capture with your report, if you're using a GPU then include a GPU profile capture as well. Look into the [PyTorch Profiler](https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html) to look at memory usage of your model.
+- **If the problem wasn't triggered by a specific action**, describe what you were doing before the problem happened and share more information using the guidelines below.
 
 Provide more context by answering these questions:
 
-* **Did the problem start happening recently** (e.g. after updating to a new version) or was this always a problem?
-* If the problem started happening recently, **can you reproduce the problem in an older version of Bittensor?** 
-* **Can you reliably reproduce the issue?** If not, provide details about how often the problem happens and under which conditions it normally happens.
+- **Did the problem start happening recently** (e.g. after updating to a new version) or was this always a problem?
+- If the problem started happening recently, **can you reproduce the problem in an older version of Bittensor?**
+- **Can you reliably reproduce the issue?** If not, provide details about how often the problem happens and under which conditions it normally happens.
 
 Include details about your configuration and environment:
 
-* **Which version of Bittensor Subnet are you using?**
-* **What commit hash are you on?** You can get the exact commit hash by checking `git log` and pasting the full commit hash.
-* **What's the name and version of the OS you're using**?
-* **Are you running Bittensor Subnet in a virtual machine?** If so, which VM software are you using and which operating systems and versions are used for the host and the guest?
-* **Are you running Bittensor Subnet in a dockerized container?** If so, have you made sure that your docker container contains your latest changes and is up to date with Master branch?
+- **Which version of Bittensor Subnet are you using?**
+- **What commit hash are you on?** You can get the exact commit hash by checking `git log` and pasting the full commit hash.
+- **What's the name and version of the OS you're using**?
+- **Are you running Bittensor Subnet in a virtual machine?** If so, which VM software are you using and which operating systems and versions are used for the host and the guest?
+- **Are you running Bittensor Subnet in a dockerized container?** If so, have you made sure that your docker container contains your latest changes and is up to date with Master branch?
 
 ### Suggesting Enhancements and Features
 
@@ -197,17 +202,17 @@ When you are creating an enhancement suggestion, please [include as many details
 
 #### Before Submitting An Enhancement Suggestion
 
-* **Check the [debugging guide](./DEBUGGING.md).** for tips — you might discover that the enhancement is already available. Most importantly, check if you're using the latest version of the project first.
+- **Check the [debugging guide](./DEBUGGING.md).** for tips — you might discover that the enhancement is already available. Most importantly, check if you're using the latest version of the project first.
 
 #### How Submit A (Good) Feature Suggestion
 
-* **Use a clear and descriptive title** for the issue to identify the problem.
-* **Provide a step-by-step description of the suggested enhancement** in as many details as possible.
-* **Provide specific examples to demonstrate the steps**. Include copy/pasteable snippets which you use in those examples, as [Markdown code blocks](https://help.github.com/articles/markdown-basics/#multiple-lines).
-* **Describe the current behavior** and **explain which behavior you expected to see instead** and why.
-* **Include screenshots and animated GIFs** which help you demonstrate the steps or point out the part of the project which the suggestion is related to. You can use [this tool](https://www.cockos.com/licecap/) to record GIFs on macOS and Windows, and [this tool](https://github.com/colinkeenan/silentcast) or [this tool](https://github.com/GNOME/byzanz) on Linux.
-* **Explain why this enhancement would be useful** to most users.
-* **List some other text editors or applications where this enhancement exists.**
-* **Specify the name and version of the OS you're using.**
+- **Use a clear and descriptive title** for the issue to identify the problem.
+- **Provide a step-by-step description of the suggested enhancement** in as many details as possible.
+- **Provide specific examples to demonstrate the steps**. Include copy/pasteable snippets which you use in those examples, as [Markdown code blocks](https://help.github.com/articles/markdown-basics/#multiple-lines).
+- **Describe the current behavior** and **explain which behavior you expected to see instead** and why.
+- **Include screenshots and animated GIFs** which help you demonstrate the steps or point out the part of the project which the suggestion is related to. You can use [this tool](https://www.cockos.com/licecap/) to record GIFs on macOS and Windows, and [this tool](https://github.com/colinkeenan/silentcast) or [this tool](https://github.com/GNOME/byzanz) on Linux.
+- **Explain why this enhancement would be useful** to most users.
+- **List some other text editors or applications where this enhancement exists.**
+- **Specify the name and version of the OS you're using.**
 
 Thank you for considering contributing to Bittensor! Any help is greatly appreciated along this journey to incentivize open and permissionless intelligence.

masa/api/server.py

@@ -22,6 +22,15 @@ def __init__(self, validator, config=None):
             allow_headers=["*"],
         )
 
+        self.app.add_api_route(
+            "/indexed_tweets",
+            self.show_indexed_tweets,
+            methods=["GET"],
+            dependencies=[Depends(self.get_self)],
+            response_description="Get indexed tweets",
+            tags=["scoring"],
+        )
+
         self.app.add_api_route(
             "/score",
             validator.scorer.score_miner_volumes,
@@ -161,6 +170,12 @@ async def show_volumes(self):
             return JSONResponse(content=serializable_volumes)
         return JSONResponse(content=[])
 
+    async def show_indexed_tweets(self):
+        tweets = self.validator.indexed_tweets
+        if len(tweets) > 0:
+            return JSONResponse(content=tweets)
+        return JSONResponse(content=[])
+
     def delete_volumes(self):
         self.validator.volumes = []
         return JSONResponse(

masa/base/validator.py

@@ -355,6 +355,7 @@ def save_state(self):
                 "hotkeys": self.hotkeys,
                 "volumes": self.volumes,
                 "versions": self.versions,
+                "indexed_tweets": self.indexed_tweets,
             },
             self.config.neuron.full_path + "/state.pt",
         )
@@ -372,12 +373,14 @@ def load_state(self):
             self.hotkeys = dict(state).get("hotkeys", [])
             self.volumes = dict(state).get("volumes", [])
             self.versions = dict(state).get("versions", [])
+            self.indexed_tweets = dict(state).get("indexed_tweets", [])
         else:
             self.step = 0
             self.scores = torch.zeros(self.metagraph.n)
             self.hotkeys = []
             self.volumes = []
             self.versions = []
+            self.indexed_tweets = []
             bt.logging.warning(
                 f"State file not found at {state_path}. Skipping state load."
             )

masa/validator/forwarder.py

@@ -31,7 +31,7 @@
 from masa.types.twitter import ProtocolTwitterTweetResponse
 
 
-TIMEOUT = 5
+TIMEOUT = 8
 
 
 class Forwarder:
@@ -139,9 +139,7 @@ async def get_miners_volumes(self):
         query = (
             f"({random_keyword.strip()}) since:{datetime.now().strftime('%Y-%m-%d')}"
         )
-        # TODO better define the frequency and count of volume queries
-        # TODO perhaps bring them all to a config or public file
-        request = RecentTweetsSynapse(query=query, count=2)
+        request = RecentTweetsSynapse(query=query, count=3)
         responses, miner_uids = await self.forward_request(
             request, sample_size=self.validator.config.neuron.sample_size_volume
         )
@@ -187,8 +185,9 @@ async def get_miners_volumes(self):
         )
         example_embedding = self.validator.model.encode(str(example_tweet))
 
+        all_valid_tweets = []
         for response, uid in zip(responses, miner_uids):
-            valid_tweets = 0
+            valid_tweets = []
             actual_response = dict(response).get("response", [])
             if actual_response is not None:
                 for tweet in actual_response:
@@ -201,8 +200,15 @@ async def get_miners_volumes(self):
                         )
                         bt.logging.info(f"Similarity: {similarity}, {tweet}")
                         if similarity >= 75:  # pretty strict
-                            valid_tweets += 1
-            self.validator.scorer.add_volume(int(uid), valid_tweets)
+                            valid_tweets.append(tweet)
+            self.validator.scorer.add_volume(int(uid), len(valid_tweets))
+            all_valid_tweets.extend(valid_tweets)
+
+        payload = {
+            "query": query,
+            "tweets": all_valid_tweets,
+        }
+        self.validator.indexed_tweets.append(payload)
 
     def check_tempo(self) -> bool:
         tempo = self.validator.subtensor.get_subnet_hyperparameters(