Random shuffle of lists #10281
Draft
Random shuffle of lists #10281
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
RaimoNiskanen
added
team:VM
CT Test ResultsNo tests were run for this PR. This is either because the build failed, or the PR is based on a branch without GH actions tests configured. Results for commit 0280798 To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass. See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally. Artifacts
// Erlang/OTP Github Action Bot |
bjorng
reviewed
Oct 15, 2025
RaimoNiskanen
force-pushed
the
raimo/stdlib/rand-shuffle
branch
from
70efe49 to
aba9094
Compare
jhogberg
reviewed
Oct 15, 2025
RaimoNiskanen
force-pushed
the
raimo/stdlib/rand-shuffle
branch
from
aba9094 to
0280798
Compare
Write a few shuffle algorithms for comparison. I have found no formal statement that it is bias free, but have tried to reason around it. The algorithm should be equivalent to generating more random decimals to decide the shuffle order for elements with the same random number. It should make no difference if the random decimals are generated always and ignored, or when needed. Speed: 1.2 s for 2^20 integers on my laptop. The classical textbook shuffle. Speed: 5 s for 2^20 integers on my laptop. Quite a beautiful algorithm since the `gb_tree` has all the functionality in itself. Speed: 5 s for 2^20 integers on my laptop. The same as the `gb_tree` above, but with a map. Uses the map key order instead of the general term order, which works just fine. Speed: 2 s for 2^20 integers on my laptop. Suggested by Richard A. O'Keefe on ErlangForums as "a random variant of Quicksort". Shall we name it Quickshuffle? Really fast. Uses random numbers efficiently by looking at individual bits for the random split. Has no overhead for tagging. Just creates intermediate lists as garbage. This generator appears to be equivalent with shuffle1, using a random number generator with 1 bit. Speed: 0.8 s for 2^20 integers on my laptop. The classical textbook shuffle. Our standard `array` module here outperforms map, probably because keys does not have to be stored, they are implicit. Speed: 2 s for 2^20 integers on my laptop. shuffle3 and shuffle4 have the theoretical limitation that when the length of the list approaches the generator size, it will take catastrophically much longer time to generate a random number that has not been used. There is no check for the list length being larger than the generator size in which case it will be impossible to generate unique random numbers for all list elements, and the algorithm will simply keep on failing forever. This is for now a theoretical problem since already for a list length with log half the generator size (e.g 2^28 with a generator size 2^56), my laptop runs out of memory with a VM of about 30 GB. shuffle1 and shuffle5 avoids that limitation. shuffle1 by recursing over the duplicates sublists so it is not affected much by fairly long lists of duplicates, shuffle5 by using only individual bits and ranges 2, 6, or 24. The classical Fisher-Yates algorithm in shuffle2 and shuffle6 does not have this limitation, but generating random numbers of unlimited length gets increasingly expensive, but should not be any problem for 2 or even 4 times the generator length, that is list lengths of well over 2^200, which is well over ridiculous.
RaimoNiskanen
force-pushed
the
raimo/stdlib/rand-shuffle
branch
from
0280798 to
a536947
Compare
|
New algorithm selected. "Quickshuffle"? |
bjorng
reviewed
Oct 16, 2025
| %% | ||
| %% Randomly split the list in two lists, | ||
| %% recursively shuffle the two smaller lists, | ||
| %% randomize the order between the lists according to their size. |
There was a problem hiding this comment.
This line is no longer correct.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds functions
rand:shuffle/1andrand:shuffle_s/2due to a discussion on ErlangForums: https://erlangforums.com/t/random-sort-should-be-included-in-the-lists-module/5125There are 4 algorithms in the first commit. The suggested winner is the one remaining in the second commit.
Documentation and test cases are still missing...