feat(twitter): Enhanced Twitter Worker Selection Algorithm

[teslashibe]

Description

This PR implements an improved worker selection algorithm for Twitter tasks in the Masa Oracle project. The goal is to balance between prioritizing high-performing workers and ensuring fair work distribution.

Key Changes

1. Modified GetEligibleWorkers function (pkg/workers/worker_selection.go)

• Now uses a specialized selection process for the Twitter category (pubsub.CategoryTwitter)
• Maintains existing behavior for all other worker categories

2. New getTwitterWorkers function (pkg/workers/worker_selection.go)

• Selects a larger pool of top-performing nodes
• Introduces controlled randomness by shuffling the pool of top performers
• Creates Worker objects from the shuffled pool, respecting the original limit

3. New calculatePoolSize function (pkg/workers/worker_selection.go)

• Determines the optimal pool size as the maximum of:
  • 5 (minimum pool size)
  • Double the requested limit
  • 20% of total nodes

4. New SortNodesByTwitterReliability function (pkg/pubsub/node_event_tracker.go)

• Sorts nodes based on their Twitter reliability using multiple criteria:
  1. More recent last returned tweet
  2. Higher number of returned tweets
  3. Longer time since last timeout
  4. Lower number of timeouts
  5. Earlier last not found time
  6. PeerId (for stable sorting when no performance data is available)

5. Updated NodeEventTracker.GetEligibleWorkerNodes (pkg/pubsub/node_event_tracker.go)

• Now uses SortNodesByTwitterReliability for the Twitter category

6. Enhanced logging

• Added informative log messages using logrus for debugging and monitoring

Implementation Details

Twitter Worker Selection Process

1. Get eligible worker nodes from NodeTracker.GetEligibleWorkerNodes(category)
2. For Twitter category:
   a. Calculate pool size using calculatePoolSize
   b. Select top performers based on the calculated pool size
   c. Shuffle the selected top performers
   d. Create Worker objects from the shuffled pool, respecting the original limit
3. For other categories:
   • Return all eligible workers without modification

Node Sorting for Twitter Reliability

The SortNodesByTwitterReliability function uses a multi-criteria approach to rank nodes:

1. Prioritizes nodes with more recent last returned tweet
2. Then by higher number of returned tweets
3. Considers the time since last timeout (longer time is better)
4. Then by lower number of timeouts
5. Deprioritizes nodes with more recent last not found time
6. Finally, sorts by PeerId for stability when no performance data is available

Benefits

• More efficient utilization of high-performing Twitter workers
• Fairer distribution of work among eligible nodes
• Improved overall system performance for Twitter-related tasks
• Maintained existing functionality for non-Twitter worker categories

TODO

Testing

• Added unit tests for the new implementation, including edge cases
• Ensured all existing tests pass
• Verified linting checks are successful

[restevens402]

The change looks great. Love the new twitter selection.