test-proxy: An advanced Proxy and WAF Detection Tool

Welcome to the Advanced Proxy and WAF Detection Tool! This powerful and flexible tool is designed to analyze potential proxy servers, load balancers, and Web Application Firewalls (WAFs) by examining open ports, SSL certificates, HTTP headers, and various other indicators.

Overview

This advanced script performs a comprehensive analysis of target hosts, including:

1. Asynchronous Port Scanning with IPv6 Support: Rapidly scans custom port ranges on both IPv4 and IPv6 addresses using asynchronous I/O.
2. SSL/TLS Certificate Analysis: Retrieves detailed SSL/TLS certificate information, including cipher suites, protocol versions, and validity checks.
3. HTTP/HTTPS Header Inspection: Sends requests to both HTTP and HTTPS endpoints and thoroughly examines the headers.
4. Proxy/Load Balancer Detection: Analyzes headers for a wide range of proxy and load balancer indicators, loaded dynamically from external files.
5. Web Application Firewall (WAF) Detection: Identifies potential WAFs based on specific header signatures, also loaded dynamically.
6. Redirect Chain Analysis: Tracks and reports on HTTP and HTTPS redirect chains.
7. GeoIP Lookup: Provides geolocation information for target IP addresses.
8. Banner Grabbing: Retrieves service banners on open ports to identify running services.
9. Customizable Output Formats: Supports text, JSON, and CSV output formats for flexibility in data analysis.
10. Advanced Logging Control: Allows setting of logging levels and offers verbose output for in-depth analysis.

Features

• High-Performance Asynchronous Scanning: Utilizes asyncio for efficient port scanning and analysis.
• IPv4 and IPv6 Support: Capable of analyzing both IPv4 and IPv6 addresses.
• Advanced Rate Limiting: Implements configurable rate limiting to prevent overwhelming target servers (1-100 concurrent connections).
• Custom Port Ranges: Allows users to specify custom port ranges or additional common ports with comprehensive validation.
• Secure SSL/TLS Analysis: Provides detailed SSL/TLS certificate data with optional strict verification to detect certificate issues.
• Advanced Input Validation: Comprehensive security validation for hostnames, ports, and file paths to prevent injection attacks.
• Advanced HTTP(S) Header Analysis: Examines a wide range of headers to detect proxies, load balancers, and WAFs, with dynamic lists.
• Banner Grabbing: Retrieves service banners to identify running services on open ports with rate limiting.
• Flexible Output Options: Supports text, JSON, and CSV output formats.
• Redirect Chain Tracking: Follows and reports on HTTP and HTTPS redirects.
• WAF Detection: Identifies common Web Application Firewalls based on specific headers, with dynamic lists.
• Verbose Logging and Logging Levels: Offers detailed logging options and allows setting of logging levels.
• Multiple Target Support: Can analyze multiple targets provided via a file.
• GeoIP Lookup: Provides geolocation information for target IP addresses.
• Modular Design: Code is organized into functions and modules for better readability and maintainability.
• Dynamic Indicator Lists: Loads proxy and WAF indicators from external files for easy updates.
• Security Hardening: Built-in protection against common security vulnerabilities including directory traversal and DoS attacks.

Requirements

• Python 3.10+ (for optimal type hinting support)
• Required Python libraries:
  • requests
  • urllib3
  • cryptography
  • asyncio (built-in with Python 3.4+)
  • csv (built-in)
  • datetime (built-in)
  • typing (built-in)

Install the required libraries using:

pip install -r requirements.txt

requirements.txt:

requests
cryptography
urllib3

Setup

It's recommended to use a virtual environment to manage dependencies:

python3 -m venv venv
source venv/bin/activate  # On Windows use `venv\Scripts\activate`
pip install -r requirements.txt

Usage

Clone the repository and navigate to the project directory:

git clone https://github.com/geeknik/test-proxy.git
cd test-proxy

Run the script with various options:

1. Basic usage:

   python testproxy.py -t example.com

2. Analyze multiple targets from a file:

   python testproxy.py -T targets.txt

3. Specify custom port ranges:

   python testproxy.py -t example.com -p 80,443,8000-8100

4. JSON output:

   python testproxy.py -t example.com -of json

5. CSV output:

   python testproxy.py -T targets.txt -of csv -f results.csv

6. Save results to a file:

   python testproxy.py -t example.com -of json -f results.json

7. Verbose output:

   python testproxy.py -t example.com -v

8. Set logging level to DEBUG:

   python testproxy.py -t example.com -l DEBUG

Command-line Arguments

• -t, --target: The IP address or hostname to analyze.
• -T, --target-file: File containing a list of targets to analyze.
• -p, --ports: Comma-separated list of ports or port ranges (e.g., 80,443,8000-8100).
• -o, --output: Output format, either 'text' (default) or 'json'.
• -of, --output-format: Output format, choices are 'text', 'json', or 'csv'.
• -f, --file: Output file path to save results.
• -l, --log-level: Set the logging level, choices are 'DEBUG', 'INFO', 'WARNING', 'ERROR' (default: 'INFO').
• -v, --verbose: Enable verbose output (equivalent to --log-level DEBUG).
• --verify-ssl: Enable SSL certificate verification (default: disabled).
• --rate-limit: Maximum concurrent connections (default: 5, range: 1-100).
• --rate-window: Rate limiting time window in seconds (default: 1.0).
• -h, --help: Show help message and exit.

Note: You must specify either -t/--target or -T/--target-file.

Example Output

Analyzing www.mapbox.com...
Resolved www.mapbox.com to IP: 146.75.104.143
Geolocation info: {'country': 'Sweden', 'state': None, 'city': None, 'latitude': 59.3247, 'longitude': 18.056}
Open ports: []

HTTP Headers (Status: 200):
  Connection: keep-alive
  Content-Type: text/html
  Content-Encoding: gzip
  CF-Ray: 97e96c3e8c00f07a-DFW
  CF-Cache-Status: DYNAMIC
  Age: 74245
  Content-Language: en
  Link: <https://www.mapbox.com>; rel="canonical"
  content-security-policy: frame-ancestors 'self'
  processed-by: Weglot
  Weglot: id.97e96c3e8c00f07a, p.cf, cs
  weglot-translated: true
  x-lambda-id: e3b8133e-4123-4b1d-8685-e9c5ed9ce63d
  Server: cloudflare
  Accept-Ranges: bytes
  Date: Sat, 13 Sep 2025 17:39:14 GMT
  Via: 1.1 varnish
  X-Frame-Options: SAMEORIGIN
  X-Served-By: cache-dfw-kdfw8210095-DFW, cache-dfw-kdfw8210066-DFW
  X-Cache: HIT, MISS
  X-Cache-Hits: 2, 0
  X-Timer: S1757785154.316774,VS0,VE164
  Vary: x-wf-forwarded-proto, Accept-Encoding
  Cross-Origin-Opener-Policy: same-origin
  Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
  X-Content-Type-Options: nosniff
  X-Download-Options: noopen
  X-XSS-Protection: 1; mode=block

HTTPS Headers (Status: 200):
  Connection: keep-alive
  Content-Type: text/html
  Content-Encoding: gzip
  CF-Ray: 97e96c40ff096a08-DFW
  CF-Cache-Status: DYNAMIC
  Age: 74245
  Content-Language: en
  Link: <https://www.mapbox.com>; rel="canonical"
  content-security-policy: frame-ancestors 'self'
  processed-by: Weglot
  Weglot: id.97e96c40ff096a08, p.cf, cs
  weglot-translated: true
  x-lambda-id: e3b8133e-4123-4b1d-8685-e9c5ed9ce63d
  Server: cloudflare
  Accept-Ranges: bytes
  Date: Sat, 13 Sep 2025 17:39:14 GMT
  Via: 1.1 varnish
  X-Frame-Options: SAMEORIGIN
  X-Served-By: cache-dfw-kdfw8210029-DFW, cache-dfw-kdfw8210167-DFW
  X-Cache: HIT, MISS
  X-Cache-Hits: 0, 0
  X-Timer: S1757785155.700663,VS0,VE109
  Vary: x-wf-forwarded-proto, Accept-Encoding
  Cross-Origin-Opener-Policy: same-origin
  Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
  X-Content-Type-Options: nosniff
  X-Download-Options: noopen
  X-XSS-Protection: 1; mode=block

HTTP Redirects:
http://www.mapbox.com/

Potential proxy/load balancer detected. Indicators found: Via, CF-RAY, X-Served-By, X-Cache, X-Timer, X-Cache

WAF detected: Generic WAF, Cloudflare WAF

Summary of findings:
  Host: www.mapbox.com
  IP: 146.75.104.143
  Open ports: []
  Proxy/load balancer indicators: Via, CF-RAY, X-Served-By, X-Cache, X-Timer, X-Cache
  WAF detected: Generic WAF, Cloudflare WAF
  Redirects detected: 1

Analysis completed in 7.89 seconds.

Security Features

This tool includes comprehensive security hardening to protect against common vulnerabilities and ensure safe scanning:

Input Validation & Sanitization

• Hostname Validation: Prevents malicious hostnames, blocks localhost/private IPs (127.0.0.1, ::1), validates IP formats, and checks IDNA encoding
• Port Security: Restricts to valid port ranges (1-65535), prevents oversized ranges, blocks duplicates, and limits range sizes to 1000 ports
• File Path Protection: Prevents directory traversal attacks, validates path lengths, and resolves symbolic links safely

Advanced Rate Limiting

• Configurable Concurrency: Set maximum concurrent connections (1-100) using --rate-limit
• Sliding Window Control: Adjust time window for rate limiting using --rate-window (default: 1.0 seconds)
• Thread-Safe Implementation: Uses advanced rate limiting class with proper locking mechanisms
• DOS Protection: Built-in protection against resource exhaustion through controlled concurrent operations

SSL/TLS Security

• Selective SSL Verification: Enable/disable SSL certificate verification with --verify-ssl flag
• Enhanced Error Reporting: Differentiates between SSL verification failures and connection errors
• Certificate Analysis: Maintains full SSL/TLS certificate information with optional strict validation

Security Hardening

• Resource Limits: Prevents DoS through input size restrictions (1000 char ports, 50 max port ranges)
• Memory Protection: Bounded buffer sizes and controlled memory usage
• Injection Prevention: Multiple layers of input validation to prevent command injection
• DNS Rebinding Protection: Hostname validation prevents DNS rebinding attacks

Usage Examples with Security Options

# Enable SSL verification for production scanning
python testproxy.py -t example.com --verify-ssl

# Custom rate limiting for large-scale scanning
python testproxy.py -t example.com --rate-limit 20 --rate-window 2.0

# Secure scanning with combined security options
python testproxy.py -t example.com --verify-ssl --rate-limit 10 --rate-window 1.5 -v

Best Practices for Security

1. Always use --verify-ssl in production environments
2. Adjust rate limiting based on network capacity and target tolerance
3. Validate inputs - the tool provides extensive built-in validation
4. Use appropriate logging levels - avoid verbose output in sensitive environments
5. Keep dependencies updated - security updates for cryptography libraries are crucial

Dynamic Indicator Lists

The script uses external files for proxy and WAF indicators, allowing for easy updates:

• Proxy Indicators File (proxy_indicators.txt): Contains a list of proxy indicator headers, one per line.
• WAF Indicators File (waf_indicators.txt): Contains WAF indicator headers and their corresponding WAF names in the format Header:WAF Name.

Ensure these files are placed in the same directory as the script.

Contribution

We welcome contributions! If you have ideas for improvements, new features, or bug fixes, please open an issue or submit a pull request. Make sure to follow the existing code style and add tests for new functionality.

License

This project is licensed under the GPLv3 License. See the LICENSE file for details.

Acknowledgments

• Thanks to the requests, urllib3, and cryptography libraries for their powerful features.
• Inspired by various cybersecurity tools and the need for comprehensive proxy and WAF detection.
• Thanks to cryptoscuttlebutt for their contributions.

Disclaimer

This tool is for educational and informational purposes only. Ensure you have permission before scanning any networks or systems you do not own or have explicit permission to test.

Recent Improvements

v2.x.x Code Quality Enhancements

• Comprehensive Type Hints: Full Python type annotations for better IDE support and code maintainability
• Performance Optimizations: Parallelized banner grabbing using asyncio and thread pools for faster scanning
• Enhanced Error Handling: Improved exception management and null safety throughout the codebase
• Security Hardening: Better SSL certificate validation and input sanitization
• Configuration Management: Extracted hardcoded constants to centralized, configurable defaults

v2.x.x Features

• Improved Performance: Concurrent banner scanning reduces analysis time by up to 60%
• Enhanced Reliability: Better timeout handling and connection management
• Code Maintainability: Modular design with clear type annotations makes future development easier

Notes

• Always ensure you have proper authorization before scanning or analyzing targets to comply with legal and ethical guidelines.
• Be cautious when scanning multiple targets or using custom port ranges to avoid potential network issues or abuse reports.