V1 - Define Reward Logic for Twitter Data Miner

[Luka-Loncar]

Objective:

To establish a clear and effective reward logic for the Twitter data miner within our Bittensor subnet, ensuring that the miner's responses are accurately scored against the correct criteria.

Tasks for V1:

• Define Correctness Criteria for Twitter Data #21
• Develop Reward Logic in reward.py #22

Tasks for V2:

• V2 - Integrate Response Time into Reward Calculation #23
• Implement Exponential Moving Average (EMA) for Rewards #24

Outstanding Questions:

• Correctness Criteria Specification
  • What specific elements should be checked to determine the correctness of Twitter profile information, recent tweets, and followers data?
  • How do we handle partial correctness or incomplete data in the miner's response?
• Reward Function Complexity
  • How complex should the reward function be to accurately reflect the quality of the miner's work?
  • Should the reward function consider additional factors such as data freshness or relevance?
• Response Time Weighting
  • What is the appropriate balance between response quality and response time in the reward calculation?
  • How should the reward logic adjust for varying complexities of data retrieval tasks?
• What Other Parameters Can We Use For Tuning
  • EMA?
    • What should be the decay factor for the EMA to ensure it accurately reflects the miner's performance trends?
    • How frequently should the EMA be updated to maintain an up-to-date performance score?

Acceptance Criteria:

• The reward logic clearly defines what constitutes a correct response from the miner.
• The reward function is implemented and tested to ensure it accurately reflects the quality of the miner's responses.
• The reward calculation includes considerations for response time and uses EMA to provide a stable performance score.
• Documentation is provided to explain the reward logic, including how correctness is determined and how rewards are calculated.

Resources:

• [reward.py](notion://www.notion.so/masafinance/template/validator/reward.py): Contains the logic for scoring miner responses.
• [Bittensor Docs](https://docs.bittensor.com/): Provides guidelines and examples for reward mechanisms in Bittensor subnets.

Notes:

• The reward logic must be robust enough to handle various edge cases and data inconsistencies.
• It is crucial to ensure that the reward logic does not incentivize gaming or exploitation by the miners.
• The reward system should be transparent and understandable to attract and retain high-performing miners.

By addressing these tasks and questions, we aim to create a fair and motivating environment for miners, driving the overall quality and efficiency of the Twitter data retrieval process within our Bittensor subnet.

Out of scope

Out of scope

Integrate Response Time into Reward Calculation

• Factor in the response time of the miner when calculating rewards to encourage efficiency.
• Decide on a method for penalizing slow responses without compromising data quality.

Implement Exponential Moving Average (EMA) for Rewards

• Apply EMA to smooth out reward fluctuations and obtain a more reliable estimate of the miner's performance over time.

[mudler]

Can we extend this to cover generically other services, and have specific reward implementation when required ? or do we intend to cover only twitter for now?

[Luka-Loncar]

I think this will be like that when we get to the end of this week. But I'd like to double click on this with eng team.

[mudler]

Closing as not relevant anymore