Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix CI error of nightly eval #2701

Closed
wants to merge 2 commits into from
Closed

Fix CI error of nightly eval #2701

wants to merge 2 commits into from

Conversation

zhaochenyang20
Copy link
Collaborator

Motivation

We rewrite and fix the CI of human_eval and math.

Modifications

Checklist

  • Format your code according to the Contributor Guide.
  • Add unit tests as outlined in the Contributor Guide.
  • Update documentation as needed, including docstrings or example tutorials.

zhyncs
zhyncs reviewed Jan 2, 2025
View reviewed changes
python/sglang/test/test_utils.py
@@ -822,3 +822,61 @@ def run_one(_):
def write_github_step_summary(content):
with open(os.environ["GITHUB_STEP_SUMMARY"], "a") as f:
f.write(content)


def launch_server_in_nightly_test(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't use this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May you update https://github.com/sgl-project/sglang/blob/main/test/srt/experiment_runner.py

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The old style of nightly evaluation is not good. Let's switch to a configurable approach to support any third-party tools. Something like this https://github.com/sgl-project/sglang/blob/main/test/srt/configs/sharegpt_config.yaml

github-advanced-security[bot]
github-advanced-security bot found potential problems Jan 2, 2025
View reviewed changes
python/sglang/test/test_utils.py

try:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(("", parsed.port))

Check warning

Code scanning / CodeQL

Binding a socket to all network interfaces Medium test

'' binds a socket to all interfaces.

Copilot Autofix AI 3 days ago

To fix the problem, we need to bind the socket to a specific interface instead of all interfaces. This can be done by replacing the empty string ('') with a specific IP address, such as 127.0.0.1, which binds the socket to the localhost interface. This change will limit the socket to accept connections only from the local machine, reducing the security risk.

Suggested changeset 1
python/sglang/test/test_utils.py

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/python/sglang/test/test_utils.py b/python/sglang/test/test_utils.py
--- a/python/sglang/test/test_utils.py
+++ b/python/sglang/test/test_utils.py
@@ -873,3 +873,3 @@
         sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-        sock.bind(("", parsed.port))
+        sock.bind(("127.0.0.1", parsed.port))
         sock.close()
@@ -878,3 +878,3 @@
         sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-        sock.bind(("", 0))
+        sock.bind(("127.0.0.1", 0))
         _, new_port = sock.getsockname()
EOF
@@ -873,3 +873,3 @@
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(("", parsed.port))
sock.bind(("127.0.0.1", parsed.port))
sock.close()
@@ -878,3 +878,3 @@
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(("", 0))
sock.bind(("127.0.0.1", 0))
_, new_port = sock.getsockname()
Copilot is powered by AI and may make mistakes. Always verify output.
Positive Feedback
Negative Feedback

Provide additional feedback

Please help us improve GitHub Copilot by sharing more details about this comment.

Please select one or more of the options
python/sglang/test/test_utils.py
return url
except OSError:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(("", 0))

Check warning

Code scanning / CodeQL

Binding a socket to all network interfaces Medium test

'' binds a socket to all interfaces.

Copilot Autofix AI 3 days ago

To fix the problem, we need to bind the socket to a specific interface instead of all interfaces. This can be achieved by replacing the empty string '' with a specific IP address, such as 127.0.0.1, which binds the socket to the localhost interface. This change will ensure that the socket only accepts connections from the local machine, reducing the security risks.

Suggested changeset 1
python/sglang/test/test_utils.py

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/python/sglang/test/test_utils.py b/python/sglang/test/test_utils.py
--- a/python/sglang/test/test_utils.py
+++ b/python/sglang/test/test_utils.py
@@ -873,3 +873,3 @@
         sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-        sock.bind(("", parsed.port))
+        sock.bind(("127.0.0.1", parsed.port))
         sock.close()
@@ -878,3 +878,3 @@
         sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-        sock.bind(("", 0))
+        sock.bind(("127.0.0.1", 0))
         _, new_port = sock.getsockname()
EOF
@@ -873,3 +873,3 @@
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(("", parsed.port))
sock.bind(("127.0.0.1", parsed.port))
sock.close()
@@ -878,3 +878,3 @@
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(("", 0))
sock.bind(("127.0.0.1", 0))
_, new_port = sock.getsockname()
Copilot is powered by AI and may make mistakes. Always verify output.
Positive Feedback
Negative Feedback

Provide additional feedback

Please help us improve GitHub Copilot by sharing more details about this comment.

Please select one or more of the options
XiaotongJiang
XiaotongJiang reviewed Jan 2, 2025
View reviewed changes
test/srt/test_nightly_human_eval.py
)


def parse_humaneval_results(output_text):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we add a todo for eval res assertion?

@zhaochenyang20
Copy link
Collaborator Author

@zhyncs Leave this to you!

@zhyncs
Copy link
Member

zhyncs commented Jan 2, 2025

As discussed in Slack, I'll handle this refactor. Thanks for your contributions, @zhaochenyang20 and @XiaotongJiang. I'll list you as co-authors. Stay tuned, it should be finished in a day or two.

@zhyncs zhyncs closed this Jan 2, 2025
@zhyncs zhyncs self-assigned this Jan 2, 2025
@zhyncs zhyncs deleted the night_eval branch January 2, 2025 18:07
@zhaochenyang20
Copy link
Collaborator Author

cool! Thanks yineng @zhyncs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@XiaotongJiang XiaotongJiang XiaotongJiang left review comments

@zhyncs zhyncs zhyncs left review comments

@merrymercy merrymercy Awaiting requested review from merrymercy merrymercy is a code owner

@Ying1123 Ying1123 Awaiting requested review from Ying1123 Ying1123 is a code owner

Assignees

@zhyncs zhyncs

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@zhaochenyang20 @zhyncs @XiaotongJiang @zhaochen20