Skip to content

[WIP][core] Fix crash during shutdown #53002

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
2 changes: 1 addition & 1 deletion python/ray/_private/worker.py
Original file line number Diff line number Diff line change
Expand Up @@ -938,7 +938,7 @@ def main_loop(self):

def sigterm_handler(signum, frame):
raise_sys_exit_with_custom_error_message(
"The process receives a SIGTERM.", exit_code=1
"The process received a SIGTERM", exit_code=1
)
# Note: shutdown() function is called from atexit handler.

Expand Down
5 changes: 3 additions & 2 deletions python/ray/_raylet.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -2368,12 +2368,13 @@ cdef CRayStatus task_execution_handler(
# This means the system exit was
# normal based on the python convention.
# https://docs.python.org/3/library/sys.html#sys.exit
msg = f"Worker exits with an exit code {e.code}."
print("RETURNING FROM HERE!")
msg = f"The worker exited with exit code {e.code}."
if hasattr(e, "ray_terminate_msg"):
msg += (f" {e.ray_terminate_msg}")
return CRayStatus.IntentionalSystemExit(msg)
else:
msg = f"Worker exits with an exit code {e.code}."
msg = f"The worker exited with exit code {e.code}."
# In K8s, SIGTERM likely means we hit memory limits, so print
# a more informative message there.
if "KUBERNETES_SERVICE_HOST" in os.environ:
Expand Down