Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tlsi net/http: HTTP/1.x transport connection broken: malformed HTTP response #5373

Open
SuperXiaoxiong opened this issue Jul 5, 2024 · 8 comments · May be fixed by projectdiscovery/retryablehttp-go#282
Assignees
Labels
Type: Enhancement Most issues will probably ask for additions or changes.

Comments

@SuperXiaoxiong
Copy link

SuperXiaoxiong commented Jul 5, 2024

use tlsi parameter will cause transport connection broken: malformed HTTP response

.\nuclei.exe -tlsi -v -u target -t "rabbitmq1.yaml" -duc -debug

[WRN] [c027dc71-857c-4298-affc-231355ebd25d] Could not execute step: [:RUNTIME] got err while executing xxx <- GET xxxx giving up after 2 attempts: Get "xxxx": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x80\x00\x04\x00\x01\x00\x00\x00\x05\x00\xff\xff\xff\x00\x00\x04\b\x00\x00\x00\x00\x00\x7f\xff\x00\x00\x00\x00\b\a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01"
[INF] No results found. Better luck next time!

no tlsi parameter will get the correct result
[c027dc71-857c-4298-affc-231355ebd25d:word-1] [http] [low] xxx

if without -debug and -v

[WRN] Found 28 template[s] loaded with deprecated paths, update before v3 for continued support.
[INF] Current nuclei version: v3.2.9 (outdated)
[INF] Current nuclei-templates version: v9.9.0 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 164
[INF] Templates loaded for current scan: 1
[WRN] Loading 1 unsigned templates for scan. Use with caution.
[INF] Targets loaded for current scan: 1
[INF] No results found. Better luck next time!

Expect :

If no random ja request retry should follow a failed random tlsi request

@SuperXiaoxiong SuperXiaoxiong added the Type: Enhancement Most issues will probably ask for additions or changes. label Jul 5, 2024
@RamanaReddy0M
Copy link
Contributor

@SuperXiaoxiong can you provide template?

@SuperXiaoxiong
Copy link
Author

SuperXiaoxiong commented Jul 5, 2024

id: c027dc71-857c-4298-affc-231355ebd25d

info:
  name: RabbitMQ
  author: superx
  description: RabbitMQ
  severity: info
  tags: panel

http:
  - method: GET
    path:
      - "{{BaseURL}}"

    host-redirects: true
    max-redirects: 2

    matchers:
      - type: word
        words:
          - "<title>RabbitMQ Management"
        part: body

  - method: GET
    path:
      - "{{BaseURL}}/favicon.ico"

    matchers:
      - type: dsl
        name: "rabbitmq"
        dsl:
          - "status_code==200 && (\"1064742722\" == mmh3(base64_py(body)))"

@jimen0
Copy link
Contributor

jimen0 commented Jul 5, 2024

Seems to work without -tlsi. I can reproduce this with the provided template and the official docker image on the current dev branch. Tested on both Linux and Windows 10 x86_64 Enterprise. Sometimes it just works though.

$ docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3.13-management
PS C:\Users\user\Desktop> .\nuclei.exe -tlsi -t .\template.yaml -u https://... -debug
...
[INF] [c027dc71-857c-4298-affc-231355ebd25d] Dumped HTTP request for https://...

GET / HTTP/1.1
Host: ...
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36 Edg/108.0.1462.42
Connection: close
Accept: */*
Accept-Language: en
Accept-Encoding: gzip

[WRN] [c027dc71-857c-4298-affc-231355ebd25d] Could not execute step: [:RUNTIME] got err while executing https://1.... <- GET ... giving up after 2 attempts: Get "https://...": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x1e\x04\x00\x00\x00\x00\x00\x00\x05\x00\x10\x00\x00\x00\x03\x00\x00\x00\xfa\x00\x06\x00\x10\x01@\x00\x01\x00\x00\x10\x00\x00\x04\x00\x10\x00\x00"

Without the tlsi flag it just works. Might be the same as refraction-networking/utls#16.

@ehsandeep
Copy link
Member

@SuperXiaoxiong -tlsi is experimental and it's possible server reject the request because of change in tls connection.

@jimen0
Copy link
Contributor

jimen0 commented Jul 5, 2024

@SuperXiaoxiong As a veeeeery ugly and never production-ready workaround and only if your target HTTP server supports HTTP2 you can address this by rebuilding nuclei with a go mod replace directive on retryablehttp-go and forcing the client (do.go) to use HTTP2:

// Do wraps calling an HTTP method with retries.
func (c *Client) Do(req *Request) (*http.Response, error) {
...
		} else {
			// Attempt the request with standard behavior
			resp, err = c.HTTPClient2.Do(req.Request)
		}

Note this might break a lot of stuff and has only been tested against a docker container behind a reverse proxy that supports HTTP/2. Unfortunately just using the -fh2 nuclei flag isn't enough.

@jimen0
Copy link
Contributor

jimen0 commented Jul 5, 2024

@ehsandeep this is a bit cleaner but still hacky.

diff --git a/do.go b/do.go
index 8ca57f7..09acaae 100644
--- a/do.go
+++ b/do.go
@@ -53,6 +53,12 @@ func (c *Client) Do(req *Request) (*http.Response, error) {
                // Check if we should continue with retries.
                checkOK, checkErr := c.CheckRetry(req.Context(), resp, err)
 
+               // if err is about malformed responses then an HTTP/1.1 transport received an HTTP/2 response. Retry with http/2.
+               if err != nil && strings.Contains(err.Error(), "net/http: HTTP/1.x transport connection broken: malformed HTTP response") {
+                       resp, err = c.HTTPClient2.Do(req.Request)
+                       checkOK, checkErr = c.CheckRetry(req.Context(), resp, err)
+               }
+
                // if err is equal to missing minor protocol version retry with http/2
                if err != nil && strings.Contains(err.Error(), "net/http: HTTP/1.x transport connection broken: malformed HTTP version \"HTTP/2\"") {
                        resp, err = c.HTTPClient2.Do(req.Request)

Seems to work for this particular template. If this is something PD wants to battle-test before merging I could open a PR on retryablehttp-go for it.

@SuperXiaoxiong
Copy link
Author

Here are the same case

GET / HTTP/1.1
Host: sonarqube.porsche-cloud.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.6.22
Connection: close
Accept: */*
Accept-Language: en
Accept-Encoding: gzip

[WRN] [0cc49ffc-8b49-4eca-b015-7bd053743fe8] Could not execute request for https://sonarqube.porsche-cloud.com: [:RUNTIME] got err while executing https://sonarqube.porsche-cloud.com <- GET https://sonarqube.porsche-cloud.com giving up after 2 attempts: Get "https://sonarqube.porsche-cloud.com": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x18\x04\x00\x00\x00\x00\x00\x00\x05\x00\x10\x00\x00\x00\x03\x00\x00\x00\xfa\x00\x06\x00\x10\x01@\x00\x04\x00\x10\x00\x00"
[INF] No results found. Better luck next time!

the template

id: 0cc49ffc-8b49-4eca-b015-7bd053743fe8

info:
  name: SonarQube panel detect
  author: x
  description: x
  severity: info
  tags: panel

http:
  - method: GET
    path:
      - "{{BaseURL}}"

    host-redirects: true
    max-redirects: 2

    matchers:
      - type: word
        words:
          - "<title>SonarQube</title>"
        part: body

@tarunKoyalwar
Copy link
Member

tarunKoyalwar commented Jul 8, 2024

resolved via retryablehttp-go projectdiscovery/retryablehttp-go#282

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: Enhancement Most issues will probably ask for additions or changes.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants