feat: improved reward system

[grantdfoster]

Description

Problem Statement
Currently, our Bittensor subnet's reward system for Twitter data scraping is based solely on whether the provided data is valid or not. This binary approach doesn't incentivize miners to continuously improve the quantity and quality of data they provide. As a result, we may be missing opportunities to gather more valuable data and encourage healthy competition among miners.

Proposed Solution
Implement a more sophisticated reward mechanism that takes into account multiple factors:

Quantity of valid tweets (we should not verify every tweet)
Quality of tweets (this needs thought)
Continuous improvement over time
A kurtosis-based distribution to reward top performers while maintaining broader participation

Notes for Reviewers

Signed commits

• Yes, I signed my commits.

[github-actions]

PR description is too short and seems to not fulfill PR template, please fill in

[teslashibe]

@grantdfoster added this here:

Refined Initial Solution: v0.8.0 release

Implement a more sophisticated reward mechanism that takes into account multiple factors that rewards higher performing miners:

1. Quantity of valid tweets as defined by the count returned in a request for the GET Tweets/Posts endpoint.
2. A validator requests continuous data from miners as a synthetic request job from a centralized queue for now
3. Organic requests can still be sent to the network through a validator
4. Round robin miner selection, if a miner returns 429 or no workers eligible, skip to the next miner. Issue here is latency will increase. We will worry about latency later. Set a parameter or constant that we can change for the number of miners selected in the round robin before return, "no miners available to fulfill the request"
5. Use a static representative JSON comparison using the LLM i.e. comparison >0.50
6. A kurtosis-based distribution to reward top performers while maintaining broader participation
7. Generate an Jupyter model showing the curves for the new Kurtosis curve

Next iteration: future release

1. Think about how similarity check can be compared to a source of truth for 'quality'
2. Quality of tweets (this needs thought)
3. Continuous improvement over time

How can miners self-optimize for higher rewards

Miners can work independently to setup private clusters/networks using the Masa Protocol that is configured as its own private network with its their own set of Twitter Scraper Nodes. This allows miners to optimize and grow their capacity on the network which benefits the subnet by rewarding miners that bring the greatest capacity to the network. This capacity is harnessed by validators to scrape static data sets that are ranked by download volume in HF as well as organic requests that are submitted to validator API endpoints by developers who want real time data access.

[teslashibe]

@grantdfoster implementation ticket to track this here: #227

[teslashibe]

Lets go