Enhancement Suggestion: Addressing Blocked Outgoing Requests on Specific Domains

[WiseHipoppotamus]

Hello everyone,

I wanted to share my recent experience using dalfox, version v2.9.0, and discuss an issue I encountered with outgoing requests being blocked on specific domains. I have been using dalfox extensively and noticed a peculiar behavior related to the tool's pipe mode.

After applying several URL filtering tools to a specific domain, such as openai, to gather a substantial number of URLs for XSS testing, I encountered an error with dalfox. The error message I received was:

goroutine 114614 [running]: github.com/hahwul/dalfox/v2/pkg/scanning.ParameterAnalysis.func7().

Please note that I was using the pipe mode of dalfox during this process.

Initially, I suspected that the error might be related to the pipe mode's cache. To test this hypothesis, I created a bash script that executed each line from a text file, where each line followed the format:

dalfox url https://www.site.com/?name=Vitor

By using this bash script, I bypassed the need to use dalfox's pipe mode, hoping that it would resolve the issue. However, to my surprise, I encountered the same error.

To further investigate, I decided to repeat the process using a different domain, this time with a significantly larger number of URLs, namely Facebook. After applying various filters, dalfox was responsible for processing over 6000 tasks. I let dalfox run in pipe mode with the filtered Facebook URLs for more than 18 hours, and to my delight, no errors occurred.

To compare the behavior with another tool, I chose XSSer. Since XSSer does not have a pipe mode, I utilized the same bash script to execute each line from a text file with URLs in the format:

xsser -u https://www.site.com/?name=Vitor

Surprisingly, the bash script worked flawlessly, and XSSer successfully performed verifications on all the openai URLs.

Based on these observations, it strongly suggests that the outgoing requests from dalfox are being blocked by openai. This issue seems to be specific to certain domains, as the majority of domains work indefinitely with dalfox's pipe mode. For example, the run on Facebook URLs with dalfox did not encounter any errors even after running for multiple days.

I have not yet delved deep into the source code of XSSer, but I believe there could be some implementation or approach that XSSer utilizes to avoid the blocking of outgoing requests. Applying a similar mechanism to dalfox would significantly enhance its usability.

I wanted to share this information with the community and gather thoughts and insights from other users and developers. Have any of you encountered similar issues with dalfox on specific domains? And do you have any suggestions or ideas on how this issue can be addressed?

Looking forward to your input and suggestions.

Thank you!