Skip to content

About

Fast proxy scraper and checker written in Rust. Collects HTTP, SOCKS4 and SOCKS5 proxies from any number of sources, verifies each one really works, and writes JSON and plain text with response time, exit IP, ASN and geolocation. Single binary for Windows, Linux, macOS and Android, with an interactive TUI.

Topics

asyncclifree-proxygeolocationhttp-proxyproxyproxy-checkerproxy-grabberproxy-listproxy-scrapeproxy-scraperproxyscrapeproxyscraperrustsocks4socks4asocks5socks5htokiotui

Resources

Readme
MIT license
Activity

Stars

1.3k stars

Watchers

21 watching

Forks

281 forks
Report repository
BranchesTags

Repository files navigation

proxy-scraper-checker

TUI Demo

A proxy scraper and checker written in Rust.

It collects HTTP/SOCKS4/SOCKS5 proxies from any number of sources, verifies each one really works, and writes the survivors out with response times, geolocation and network ownership attached.

Features

  • Async Rust, shipped as a single binary for Windows, Linux, macOS, Android, FreeBSD, NetBSD, illumos and Solaris. Nothing to install.
  • Proxies are pattern-matched out of raw text, HTML or JSON, so any source works untouched. scheme://user:pass@host:port and CIDR ranges are understood, from URLs or local files.
  • Every proxy has to fetch a real URL in full to survive, so ones that connect and then stall get dropped. Results are deduplicated across sources.
  • Each working proxy comes with its response time, and with its exit IP when check_url returns one. ASN and city-level geolocation are looked up in offline MaxMind databases, so there are no per-proxy API calls.
  • The interactive TUI shows per-protocol progress and logs as the run goes.
  • Output is JSON with metadata, plus ready-to-paste plain text lists.

Related

If you want proxies without running anything, monosans/proxy-list is updated regularly using this tool.

Safety warning

Checking opens hundreds of simultaneous connections to untrusted hosts. Your ISP may read that as abusive traffic, and cheap routers can drop connectivity when their NAT table fills. Consider a VPN, and lower max_concurrent_checks if a run destabilizes your network.

Quick start

Every option is documented inline in config.toml. The binary reads config.toml from the current directory; set PROXY_SCRAPER_CHECKER_CONFIG to point it somewhere else.

Pre-built archives are produced by CI from the latest commit on main and published as the latest release, see Verifying downloads. Open the section for your system.

Windows

  1. Download the archive for your processor:

    Processor Download
    64-bit Intel or AMD (almost every PC) Download
    ARM64 (Snapdragon and other ARM laptops) Download
    32-bit Download

    Not sure? Open Settings → System → About and look at "System type": x64-based processor means 64-bit Intel or AMD, ARM-based processor means ARM64, and 32-bit operating system needs the 32-bit build whatever the processor.

  2. Extract the archive to a dedicated folder.

  3. Edit config.toml.

  4. Run proxy-scraper-checker.exe. If SmartScreen shows "Windows protected your PC", click "More info", then "Run anyway".

macOS

  1. Download the archive for your processor:

    Processor Download
    Apple Silicon (M1 and newer) Download
    Intel Download

    Not sure? Apple menu → About This Mac shows "Chip: Apple M…" on Apple Silicon and "Processor: … Intel …" on Intel.

  2. Extract the archive to a dedicated folder.

  3. Edit config.toml.

  4. Run it from Terminal in that folder:

    ./proxy-scraper-checker

    If macOS refuses because it cannot verify the developer, clear the download quarantine and run it again:

    xattr -d com.apple.quarantine proxy-scraper-checker
Linux

  1. Run uname -m and download the archive from the matching row. The glibc build suits most distributions (Ubuntu, Debian, Fedora, Arch and so on). Each glibc download notes the oldest glibc it runs on (ldd --version prints yours). Take the musl build on Alpine, on an older glibc, or if the glibc build refuses to start.

    Several rows share i686, armv7l, armv6l or riscv64. On i686, take the row without SSE2 only if grep -w sse2 /proc/cpuinfo prints nothing. On armv7l and armv6l, the system has hardware floating point if /lib/ld-linux-armhf.so.3 (glibc), /lib/ld-musl-armhf.so.1 or /lib/ld-musl-armv7.so.1 (musl) exists; otherwise take a row without hardware floating point. The Thumb-2 build also needs neon in /proc/cpuinfo. On riscv64, take the RVA23 build only if the processor implements the RVA23 profile.

    Processor uname -m glibc musl
    64-bit Intel or AMD x86_64 Download (2.17+) Download
    64-bit ARM (Raspberry Pi 3 and newer on a 64-bit OS, ARM servers) aarch64 Download (2.17+) Download
    32-bit ARMv7 (Raspberry Pi 2 and newer on a 32-bit OS) armv7l Download (2.17+) Download
    32-bit ARMv6 (Raspberry Pi 1, Zero, Zero W) armv6l Download (2.17+) Download
    32-bit Intel or AMD i686 Download (2.17+) Download
    64-bit RISC-V riscv64 Download (2.29+) Download
    LoongArch loongarch64 Download (2.36+) Download
    64-bit PowerPC, little-endian ppc64le Download (2.32+) Download
    64-bit PowerPC, big-endian ppc64 Download (2.39+) Download
    32-bit PowerPC ppc Download (2.39+) —
    IBM Z s390x Download (2.17+) —
    64-bit SPARC sparc64 Download (2.39+) —
    32-bit ARMv5 armv5tel Download (2.23+) Download
    32-bit x86 without SSE2 (Pentium III, Athlon XP and older) i686 Download (2.17+) Download
    32-bit ARMv7 without hardware floating point armv7l Download (2.27+) Download
    32-bit ARMv6 without hardware floating point armv6l Download (2.17+) Download
    32-bit ARMv7 with NEON, Thumb-2 build armv7l Download (2.23+) —
    64-bit RISC-V with the RVA23 profile riscv64 Download (2.39+) —

  2. Extract the archive to a dedicated folder.

  3. Edit config.toml.

  4. Run it from that folder:

    ./proxy-scraper-checker
Android / Termux

Install Termux from F-Droid, not Google Play (why?).

  1. Install with one command:

    bash <(curl -fsSL 'https://raw.githubusercontent.com/monosans/proxy-scraper-checker/main/termux.sh')

  2. Edit the config in a text editor:

    nano ~/proxy-scraper-checker/config.toml

  3. Run it:

    cd ~/proxy-scraper-checker && ./proxy-scraper-checker

The script picks the right archive by itself. To fetch one by hand, match the output of getprop ro.product.cpu.abi:

getprop ro.product.cpu.abi Download
arm64-v8a Download
armeabi-v7a Download (without NEON)
armeabi Download
x86_64 Download
x86 Download
FreeBSD

  1. Run uname -m and download the archive from the matching row:

    Processor uname -m Download
    64-bit Intel or AMD amd64 Download
    64-bit ARM arm64 Download
    32-bit Intel or AMD i386 Download

  2. Extract the archive to a dedicated folder.

  3. Edit config.toml.

  4. Run it from that folder:

    ./proxy-scraper-checker
NetBSD

  1. Download the 64-bit x86 archive. It needs NetBSD 10 or newer.

  2. Extract the archive to a dedicated folder.

  3. Edit config.toml.

  4. Run it from that folder:

    ./proxy-scraper-checker
illumos

  1. Download the 64-bit x86 archive.

  2. Extract the archive to a dedicated folder.

  3. Edit config.toml.

  4. Run it from that folder:

    ./proxy-scraper-checker
Solaris

  1. Run uname -p and download the archive from the matching row. It needs Solaris 11.4 or newer.

    Processor uname -p Download
    64-bit x86 i386 Download
    SPARC sparc Download

  2. Extract the archive to a dedicated folder.

  3. Edit config.toml.

  4. Run it from that folder:

    ./proxy-scraper-checker
Docker on Windows

The Docker image logs to stdout instead of showing the interactive TUI.

  1. Install Docker Desktop.

  2. Download the archive for your processor:

    Processor Download
    64-bit Intel or AMD (almost every PC) Download
    ARM64 (Snapdragon and other ARM laptops) Download

    Not sure? Open Settings → System → About and look at "System type": x64-based processor means 64-bit Intel or AMD, ARM-based processor means ARM64.

  3. Extract it to a folder and edit config.toml.

  4. Build and run from that folder:

    docker compose build
docker compose up --no-log-prefix --remove-orphans

Results land in out inside that folder; output.path is ignored in Docker.

Docker on Linux / macOS

The Docker image logs to stdout instead of showing the interactive TUI.

  1. Install Docker Compose.

  2. Run uname -m and download the archive from the matching row:

    Processor uname -m Download
    64-bit Intel or AMD (including Intel Macs) x86_64 Download
    64-bit ARM (Apple Silicon Macs, Raspberry Pi on a 64-bit OS) aarch64, arm64 on macOS Download
    32-bit ARMv7 armv7l Download
    32-bit Intel or AMD i686 Download
    64-bit PowerPC, little-endian ppc64le Download
    64-bit RISC-V riscv64 Download
    IBM Z s390x Download

  3. Extract it to a folder and edit config.toml.

  4. Build and run from that folder. The build arguments make the files in ./out belong to your user:

    docker compose build --build-arg UID=$(id -u) --build-arg GID=$(id -g)
docker compose up --no-log-prefix --remove-orphans

Results land in ./out; output.path is ignored in Docker.

Build from source

  1. Install the Rust toolchain, see https://rust-lang.org/tools/install/

  2. Clone the repository:

    git clone https://github.com/monosans/proxy-scraper-checker.git
cd proxy-scraper-checker

  3. Build and run it with the TUI:

    cargo run --features tui --release --locked

The binary lands in target/release/proxy-scraper-checker on Linux/macOS, or target\release\proxy-scraper-checker.exe on Windows.

Verifying downloads

Every archive is built by GitHub Actions from the public source and published by the CI workflow straight from GitHub's runners, never from someone's computer:

  • Each release links to a build provenance attestation that GitHub signs through Sigstore. It lists the SHA-256 of every archive together with the commit and the workflow run that built it, and only that workflow run could have produced the signature. Whatever went into a build, the source and the workflow included, is therefore in the public history of this repository.
  • Releases are immutable: once published, their files cannot be replaced.

To check a downloaded archive without installing anything:

  1. Compute its SHA-256: Get-FileHash <archive> in PowerShell, shasum -a 256 <archive> on macOS, sha256sum <archive> on Linux and Termux, sha256 <archive> on FreeBSD.
  2. Open the "Build provenance" link in the notes of the latest release and find the same hash next to the archive name. The page also shows the commit and the workflow run it was built from. Only the latest release is kept; attestations of older builds stay on the attestations page.

To check the signature itself, use the GitHub CLI:

gh attestation verify <archive> \
  --repo monosans/proxy-scraper-checker \
  --signer-workflow monosans/proxy-scraper-checker/.github/workflows/ci.yml

The Docker images are attested the same way, and the prebuilt Docker archives pin the image by digest, so docker compose build uses exactly the image CI built.

Output

out/
├── proxies.json          working proxies with metadata, compact
├── proxies_pretty.json   the same data, indented
└── proxies/
    ├── all.txt           every proxy, with its protocol prefix
    ├── http.txt          one file per enabled protocol, no prefix
    ├── socks4.txt
    └── socks5.txt

A text line is host:port, or username:password@host:port for a proxy that needs credentials, with protocol:// in front of either in all.txt.

{
  "protocol": "socks5",
  "username": null,
  "password": null,
  "host": "1.2.3.4",
  "port": 1080,
  "timeout": 0.42,
  "exit_ip": "5.6.7.8",
  "asn": {
    "autonomous_system_number": 12345,
    "autonomous_system_organization": "Example Networks"
  },
  "geolocation": {
    "country": { "iso_code": "US", "names": { "en": "United States" } },
    "city": { "names": { "en": "Chicago" } },
    "location": {
      "latitude": 41.85,
      "longitude": -87.65,
      "time_zone": "America/Chicago"
    }
  }
}

timeout is how long the whole request took, in seconds: connection, request and the complete response. That is also what sort_by_speed orders by. exit_ip is the address the target site actually saw, so comparing it with host spots proxies that forward through somewhere else.

exit_ip, asn and geolocation are only filled in when check_url returns the exit IP, so pointing it at an ordinary web page leaves all three null. geolocation mirrors the GeoLite2 City record (abbreviated above); fields the database has no data for are omitted, and names are English only.

Stopping a run early, with ESC or q in the TUI or Ctrl-C anywhere, writes out the working proxies found so far. If none has been found yet, the previous run's files are left as they were.

Sponsors

Want your name in this section? Support the project and it goes here.

Support this project

Star the repository so other people can find it. If you are interested in sponsoring, DM me on Telegram.

License

MIT

The binaries contain third-party code under its own licenses, listed in LICENSE-THIRD-PARTY. CI generates it for every build, and it comes with LICENSE in every release archive and next to the binary in /app of the published Docker image. An image you build yourself includes it only if the file is in your checkout.

The ASN and geolocation databases are not part of the binaries: the program downloads them at run time. They are GeoLite Data, covered by the GeoLite End User License Agreement.

This product includes GeoLite Data created by MaxMind, available from https://www.maxmind.com.

About

Fast proxy scraper and checker written in Rust. Collects HTTP, SOCKS4 and SOCKS5 proxies from any number of sources, verifies each one really works, and writes JSON and plain text with response time, exit IP, ASN and geolocation. Single binary for Windows, Linux, macOS and Android, with an interactive TUI.

Topics

asyncclifree-proxygeolocationhttp-proxyproxyproxy-checkerproxy-grabberproxy-listproxy-scrapeproxy-scraperproxyscrapeproxyscraperrustsocks4socks4asocks5socks5htokiotui

Resources

Readme
MIT license
Activity

Stars

1.3k stars

Watchers

21 watching

Forks

281 forks
Report repository

Releases

Packages

Used by

Contributors

Languages