Goreplay was unable to read Solr requests after a certain length.

[M-Somtirth]

I am using Gor on my Linux system where I am running simply this command: sudo ./gor --input-raw:8985 --output-stdout
I have an active node of Solr Cloud on port 8985 while querying found that for this query

The Query I made on my Solr node is given below:
http://Solr-IP:8985/solr/search-goreplay/select?". . ."
The query length is ~2800 characters
Gor was unable to read the request not printing any output.

Decreasing the length of the query further upto length:

http://Solr-IP:8985/solr/search-goreplay/select? ". . ."
The Query Length is ~1200 characters in all total
Gor was working without any issue for all queries around that length.

Questions:

1. Is this an issue of my system or is it like Gor reads requests up to a certain limit?
2. If so is there any option to dynamically increase the request length supported by Gor?

[dosubot]

🤖

Hello @M-Somtirth! I'm Dosu, a friendly bot here to assist you with your GoReplay questions and issues while we wait for a human maintainer. I'm ready to help solve bugs, answer questions, and even guide you on becoming a contributor. Let's dive into your issue together!

Based on the provided context, GoReplay does not explicitly mention a limit on the length of the requests it can handle. However, there is a limit on the response body that the internal HTTP client will fetch, which is 200kb by default. This limit can be increased using the --output-http-response-buffer option as mentioned in the Replaying Http Traffic section of the GoReplay documentation.

As for the specific error or exception that GoReplay throws when it encounters a request that exceeds its limit, the provided context does not provide information on this. It's possible that the issue you're experiencing is due to a limitation in GoReplay, your system, or the specific configuration you're using.

To better understand the issue, it would be helpful to have more information. For example, are there any error messages displayed when GoReplay fails to read the request? Additionally, could you provide more details about your system configuration and the specific version of GoReplay you're using?

If you have already tried increasing the --output-http-response-buffer limit and the issue persists, it might be worth considering other factors that could be causing this issue. For instance, there could be a limit set by the operating system or the network infrastructure that is causing the issue.

I hope this helps! If you have any more questions or need further clarification, feel free to ask.

Sources

• Replaying Http Traffic

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[M-Somtirth]

Firstly, there is no output when the query size is large, Gor simply doesn't produce any output.
Secondly after using the suggestions stated in the above comment got the same result.
When I have tried using the command: sudo ./gor --input-raw :8985 --output-stdout --output-http-response-buffer 1024kb
but couldn't observe any change.

[buger]

@M-Somtirth try the --input-raw-allow-incomplete flag. Also I recommend increasing buffer even more.
And also set --input-raw-buffer-size 10485760

Here is related page https://github.com/buger/goreplay/wiki/Dealing-with-missing-requests-and-responses

[M-Somtirth]

@buger Even while using this I have found the number of queries fired is not exactly fired on the target Solr Server.
While saving the requests in a file for replaying later I can find the number to match exactly.
Is there any way possible to keep the logs for Go-Replay so that I can check and verify that which query got skipped while redirecting requests at real-time?