Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Nuclei Proxy Handling Issues: Ignores Concurrency Settings & Runs Extremely Slow #6029

Open
1 task done
Nishantbhagat57 opened this issue Feb 6, 2025 · 7 comments
Labels
help wanted Extra attention is needed Investigation Something to Investigate Type: Bug Inconsistencies or issues which will cause an issue or problem for users or implementors.

Comments

@Nishantbhagat57
Copy link

Is there an existing issue for this?

  • I have searched the existing issues.

Current Behavior

Well, this is a complex one to explain, but I don't think it's hard to fix.

In simple words:

  1. Nuclei runs super slow when using a proxy.
  2. OR: Nuclei doesn't follow concurrency values when using a proxy.

Expected Behavior

Nuclei should respect the configured concurrency values when using a proxy and maintain the same request speed as when running without a proxy. Proxy usage should not significantly degrade performance.

Steps To Reproduce

Note: Recording this issue would have taken more time, so I wrote it all down. However, I think I have provided everything in an easy-to-understand way so you can reproduce this issue easily from your side.

  1. Create an SSRF subdomain instance from https://ssrf.cvssadvisor.com
  2. Get any static/rotating proxy from any provider. I'm using rotating proxies from https://smartproxy.com
  3. Verify that your proxy provider doesn't limit concurrent connections or impose rate limits by running:
time (seq 1000 | xargs -n1 -P100 -I{} curl -x http://username:[email protected]:10000 -s -w "%{time_total}\n" -o /dev/null https://59ip1dbi.ssrf.cvssadvisor.com)

You will see it easily sends 1000 requests within 5-15 seconds. This confirms that the proxy is super fast and doesn't impose any limits (concurrent/rate limits) that could slow down a Nuclei scan.

  1. Now, run any Nuclei fuzzing template with higher concurrency limits (First Without Proxy):
echo "https://59ip1dbi.ssrf.cvssadvisor.com/?user=nb&api-token=true" | nuclei -bulk-size 100 -fuzz-param-frequency 100000 -dast -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:135.0) Gecko/20100101 Firefox/135.0" -t /home/nishant57/oob/custom-ssrf-fuzzing-template.yaml -rl 400 -timeout 30 -c 80 -payload-concurrency 80

It works great, right? You can verify the request speed on https://ssrf.cvssadvisor.com/instance/your-instance by observing the Requests Count value.

  1. Now, let's use the same command but with a proxy this time.
    First, clear the logs on https://ssrf.cvssadvisor.com/instance/your-instance, then use the same command but with our fast proxy:
echo "https://59ip1dbi.ssrf.cvssadvisor.com/?user=nb&api-token=true" | nuclei -bulk-size 100 -fuzz-param-frequency 100000 -proxy "http://username:[email protected]:10000" -dast -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:135.0) Gecko/20100101 Firefox/135.0" -t /home/nishant57/oob/custom-ssrf-fuzzing-template.yaml -rl 400 -timeout 30 -c 80 -payload-concurrency 80
  1. At the same time, observe the Requests Count value on https://ssrf.cvssadvisor.com/instance/your-instance webpage. You will notice that requests are being sent Super Slow (only 2-5 requests per second) despite having already set high concurrency limits in nuclei.

  2. Want more assurance that this has nothing to do with the proxy provider? Let's use Nuclei with proxychains-ng instead.

Testing With Proxychains-NG

A) Install proxychains-ng:

git clone https://github.com/rofl0r/proxychains-ng
cd proxychains-ng
./configure --prefix=/usr --sysconfdir=/etc
sudo make install
sudo make install-config

B) Set up the proxychains4.conf file with this content:

strict_chain
proxy_dns
tcp_read_time_out 15000
tcp_connect_time_out 8000

[ProxyList]
# Format: <type> <host> <port> <username> <password>
http  yourproxyprovider.com 10000 username password

C) Clear the logs on https://ssrf.cvssadvisor.com/instance/your-instance

D) Run the same Nuclei command, but this time using proxychains-ng:

echo "https://59ip1dbi.ssrf.cvssadvisor.com/?user=nb&api-token=true" | proxychains4 -f /path/to/your/proxychains4.conf nuclei -bulk-size 100 -fuzz-param-frequency 100000 -dast -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:135.0) Gecko/20100101 Firefox/135.0" -t /home/nishant57/oob/custom-ssrf-fuzzing-template.yaml -rl 400 -timeout 30 -c 80 -payload-concurrency 80

E) At same time, observe the Requests Count value on the https://ssrf.cvssadvisor.com/instance/your-instance webpage. Bingo! Requests are now getting sent super fast, just like they were without a proxy—but this time with our proxy, thanks to proxychains-ng!

Conclusion

This confirms that Nuclei has a major issue with proxy handling.
It doesn't respect concurrency values when using a proxy, making scans significantly slower than expected.

Would appreciate it if you could check this issue out. Let me know if you need further details!
@ehsandeep @dogancanbakir @DhiyaneshGeek @dwisiswant0

Relevant log output

Environment

- OS: Debian 11
- Nuclei: v3.3.8
- Go: go1.23.2 linux/amd64

Anything else?

No response

@Nishantbhagat57 Nishantbhagat57 added the Type: Bug Inconsistencies or issues which will cause an issue or problem for users or implementors. label Feb 6, 2025
@Nishantbhagat57
Copy link
Author

@Nishantbhagat57
Copy link
Author

I was using proxychains4 in a wrong way. non-numeric ips i.e a dns proxy can't be added as the first one in the list. So I changed the DNS proxy to IP one but I observed that the requests are being sent by my own server IP rather than by the proxy IP. I didn't observed that when writing this bug report but it yes at that time too it was being sent from my own IP address.

This means that nuclei doesn't respect proxy or is Non-Proxy-Aware. So what options do we have currently? I think this needs to be fixed from nuclei's side. If the proxy server supports thousands of requests per second then why to limit from our side?
@ehsandeep @dogancanbakir @DhiyaneshGeek @dwisiswant0

@dwisiswant0 dwisiswant0 added help wanted Extra attention is needed Investigation Something to Investigate labels Feb 7, 2025
@Nishantbhagat57
Copy link
Author

Thanks, @dwisiswant0, for adding those tags. Currently, I have around 25k URLs to scan with my custom templates through Nuclei. I don't want to do it with my own IP, so I’ve bought several hundred IPs from different providers—all of them are super fast and have no concurrency limits. But what else can I do?

I know I could try creating a script out of those templates instead, but that would take a lot of time. I would really appreciate it if you could prioritize this issue.

Alternatively, if you have any quick workarounds or possible solutions in the meantime, I’m open to trying them.

@dogancanbakir
Copy link
Member

@Nishantbhagat57 Given what you've shared, I couldn't find any details about the proxy service, logs, or metrics suggesting that nuclei requests are either slow or not being sent. Do you know if we can get this information? You also mentioned having different providers; is this issue occurring with all of them?

@Nishantbhagat57
Copy link
Author

@dogancanbakir I have already provided all the Steps to Reproduce this issue and that too in most easy way. Yes the issue is occurring with all of them. I have 3 different providers and as I said before, they don't impose any limits on concurrent requests or sessions and I have verified that too by opening hundreds of browsers at same time and with curl too.

Okay, I will send you a screen recording in the next comment.

But it would be great if you still try to reproduce this on your end. I can provide you the proxy, please DM me on discord, my username: nishantcontributor

@Nishantbhagat57
Copy link
Author

@dogancanbakir @dwisiswant0 Here's the proof/details/metrics with recordings:

Curl With Proxy:

PROXY_WITH_CURL.MP4

10 seconds for 1000 requests - around 100 req / second

Nuclei Without Proxy:
https://drive.google.com/file/d/1rhq7GVJUlQYAxWQCj4ihEd7acBrthy0F/view?usp=sharing

39 seconds for 424 requests - around 11 - 20 req / second which is still slow (I have set every frequency/concurrent value to high)

Nuclei With Proxy:
https://drive.google.com/file/d/12Xa6xpCtGtfpIKBA7oejmQiG3fvA8ESS/view?usp=sharing

6.5 minutes for 427 requests - around 1-2 req / second which is super super slow (I have set every frequency/concurrent value to high like above)

I have changed the password of proxy, if you need it for testing on your side then please DM me on my discord nishantcontributor

@Nishantbhagat57
Copy link
Author

Likhithsai2580 added a commit to Likhithsai2580/nuclei that referenced this issue Mar 16, 2025
Related to projectdiscovery#6029

Address proxy handling issues in Nuclei to respect concurrency settings and improve request speed.

* **internal/runner/proxy.go**
  - Add logic to handle concurrency settings for proxy servers.
  - Ensure proxy settings respect concurrency values.

* **internal/runner/runner.go**
  - Initialize the HTTP client with proxy settings and ensure concurrency is respected.
  - Add logic to handle concurrency settings for proxy servers.

* **pkg/protocols/http/httpclientpool/clientpool.go**
  - Manage HTTP client pooling and enforce concurrency settings with proxies.
  - Ensure proxy settings respect concurrency values.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed Investigation Something to Investigate Type: Bug Inconsistencies or issues which will cause an issue or problem for users or implementors.
Projects
None yet
Development

No branches or pull requests

3 participants