Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(twitter): Implement random sleep and improve login process #601

Conversation

teslashibe
Copy link
Contributor

Description

This PR introduces improvements to our Twitter scraping authentication process to make it more robust and less detectable as automated behavior.

Changes

  1. In config.go:

    • Introduced RandomSleep() function to add variability in request timing
    • Set sleep duration range from 500ms to 2s
    • Initialized a package-level random number generator for thread-safety
  2. In auth.go:

    • Updated NewScraper() to use RandomSleep() before and after login attempts
    • Improved error handling and logging in the login process

Rationale

These changes aim to make our scraping behavior more human-like and reduce the risk of being detected as a bot or having our accounts suspended. By introducing random delays between actions, we mimic natural user behavior more closely.

Testing

  • Tested login process with multiple accounts
  • Verified that random sleep durations are being applied correctly
  • Ensured that error handling and logging are working as expected

References

  • auth.go:
startLine: 9
endLine: 35
  • config.go:
startLine: 10
endLine: 32

Checklist

  • Code follows the project's coding standards
  • Comments have been added where necessary
  • All tests pass
  • Documentation has been updated (if applicable)

- Add RandomSleep function to introduce variability in request timing
- Update NewScraper to use RandomSleep before and after login attempts
- Adjust sleep duration range to 500ms - 2s for more natural behavior
- Improve error handling and logging in the login process
@teslashibe teslashibe self-assigned this Oct 15, 2024
@teslashibe teslashibe added the bug Something isn't working label Oct 15, 2024
@teslashibe teslashibe merged commit a8a77a6 into main Oct 15, 2024
9 of 11 checks passed
@teslashibe teslashibe deleted the fix(twiter)-randomize-steps-in-twitter-login-flow-for-bot-detection branch October 15, 2024 21:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant