Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Release test dask_on_ray_100gb_sort.aws failed #48518

Closed
can-anyscale opened this issue Nov 3, 2024 · 9 comments · Fixed by #48858
Closed

Release test dask_on_ray_100gb_sort.aws failed #48518

can-anyscale opened this issue Nov 3, 2024 · 9 comments · Fixed by #48858
Assignees
@rynewang
Labels
bug Something that is supposed to be working; but isn't core Issues that should be addressed in Ray Core P0 Issues that should be fixed in short order ray-test-bot Issues managed by OSS test policy release-test release test stability unstable-release-test weekly-release-blocker Issues that will be blocking Ray weekly releases

Comments

@can-anyscale
Copy link
Collaborator

Release test dask_on_ray_100gb_sort.aws failed. See https://buildkite.com/ray-project/release/builds/25275#0192ef85-8fcc-4f3a-bd0c-e9ccd433aee5 for more details.

Managed by OSS Test Policy
@can-anyscale can-anyscale added bug Something that is supposed to be working; but isn't core Issues that should be addressed in Ray Core P0 Issues that should be fixed in short order ray-test-bot Issues managed by OSS test policy release-test release test stability triage Needs triage (eg: priority, bug/not-bug, and owning component) unstable-release-test labels Nov 3, 2024
@can-anyscale
Copy link
Collaborator Author

Test has been failing for far too long. Jailing.

@can-anyscale can-anyscale added the jailed-test Indicate that this test is jailed and will stop running in CI/CD label Nov 4, 2024
@jjyao jjyao removed the triage Needs triage (eg: priority, bug/not-bug, and owning component) label Nov 18, 2024
@jjyao jjyao assigned jjyao and unassigned rynewang Nov 21, 2024
@jjyao jjyao added the weekly-release-blocker Issues that will be blocking Ray weekly releases label Nov 21, 2024
@jjyao jjyao assigned rynewang and unassigned jjyao Nov 22, 2024
@jjyao
Copy link
Collaborator

jjyao commented Nov 22, 2024

There is a real Ray bug that @rynewang is fixing.

@rynewang rynewang mentioned this issue Nov 22, 2024
Merged
@jjyao jjyao removed the jailed-test Indicate that this test is jailed and will stop running in CI/CD label Nov 22, 2024
@can-anyscale can-anyscale reopened this Nov 23, 2024
@can-anyscale
Copy link
Collaborator Author

Re-opening issue as test is still failing. Latest run: https://buildkite.com/ray-project/release/builds/26725#019357ec-9b55-4574-a31c-0b4e89e0bef0

@aslonnie
Copy link
Collaborator

it was passing at https://buildkite.com/ray-project/release/builds/26677#019355a0-0d21-4a68-83d1-603494a84b46

@aslonnie
Copy link
Collaborator

why this test keeps getting worker killed?

@aslonnie
Copy link
Collaborator

passed with dask upgraded:
https://buildkite.com/ray-project/release/builds/26783#01935ce0-6b4e-4c3d-b048-867270d5de05
#48898

though I don't think the upgrade played that much of difference.

@aslonnie
Copy link
Collaborator

I am closing it to let it run again.

@aslonnie
Copy link
Collaborator

it was passing at https://buildkite.com/ray-project/release/builds/26677#019355a0-0d21-4a68-83d1-603494a84b46

rerun at the same commit failed:
https://buildkite.com/ray-project/release/builds/26785#01935cfa-2616-4161-8efc-b5125de3747c

@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/release/builds/26807#01936232-1ff7-4d1c-8988-e067fd709b8f

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rynewang rynewang

Labels
bug Something that is supposed to be working; but isn't core Issues that should be addressed in Ray Core P0 Issues that should be fixed in short order ray-test-bot Issues managed by OSS test policy release-test release test stability unstable-release-test weekly-release-blocker Issues that will be blocking Ray weekly releases
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[core] Erases PopWorkerRequest on registration timeout. rynewang/ray
4 participants
@jjyao @rynewang @aslonnie @can-anyscale