Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

draft changes for multilocale suffix array construction #38

Open
wants to merge 106 commits into
base: main
Choose a base branch
from
Open

draft changes for multilocale suffix array construction #38

wants to merge 106 commits into from

Conversation

mppf
Copy link
Contributor

@mppf mppf commented Dec 3, 2024

Achieved limited scalability with bc3ab70

  • put back in random access

mppf added 21 commits December 9, 2024 15:29
@mppf
Add timing printouts when compiled with --set TIMING=true
2f36dbb 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Use block distributed, and use integral SA elements
3aa74d6 
only use offsetAndCached when the cached type != nothing;
otherwise just use integral offsets

---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Use Block distribution in partitioning, more distributed ssort
28586e3 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Hide warning, add bucket statistics, enhance trace
50dea99 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Replicate splitters
05c28b9 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Use Block arrays only when CHPL_COMM != none
5f9e0f6 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Replicate splitters only with CHPL_COMM!=none
6d5da0e 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Adjust comments
236326e 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Add timing inside sort buckets & avoid unneeded Block there
cc2d748 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Add nextCoverIndex
69689fe 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Allow partition to have different output type
8f219a3 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Experimental: separating lookup phase in final sort
e2a5a23 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Don't separate lookup phase
9e98e9a 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Replicate sample ranks
f942712 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Also replicate the text
8bf935c 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Improve replicating text and sample ranks
8d8f8bc 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Prototype work for no-random-access version
bfa17ad 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Add more testing, fix a bug
39eeb70 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Hide some communication
a66c275 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Avoid comms for accessing difference cover
6187e04 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Don't make a local copy
f18eebe 
---
Signed-off-by: Michael Ferguson <[email protected]>
mppf added 8 commits December 12, 2024 12:03
@mppf
Tidy up partition()
40e3048 
* make it driven by a possibly-distributed Domain
* support that with 'Utility.divideIntoTasks' iterator
* use replicated in the partition code

---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Add mechanism to pack input
2174e2b 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Get divide by buckets working multilocale & testing
7c420fc 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Fix a comment
533ea8b 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Add filter mechanism to partition & test stability
5659077 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Add a simpler test of the divideByBuckets iterator
0dfb366 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Add getBit/setBit
4905a2a 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Add implementation of insertion sort
ae71f56 
---
Signed-off-by: Michael Ferguson <[email protected]>
mppf added 30 commits January 19, 2025 10:22
@mppf
Fix a bug & fix multilocale compilation
378610d 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Use radix sort for initial naming process
9316410 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Small changes
e7fb031 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Fix a bug & time copy-to-local-and-sort
910a4d5 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Fix a bug and include serial bucket stats in trace
dfb6cc4 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Fix problem with serial splitters
1846ecb 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Improve timing, time more parts
b4651fa 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
'cached' stores two words instead of one
9618a9f 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Tidy up some of sortAndNameSampleOffsets
39c0568 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Switch to final phase using parallel partitions & local copies
7ff9b02 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Re-enable code for different character bits
d686081 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Use default period of 57 based on experiments
60c4c77 
Also improve trace output

---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Reduce memory usage of naming portion
62d76bd 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Improve sequence reading process
a5e2228 
Make it parallel to improve performance

---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Parallelize reverseComplement
521144c 
and further improve fasta reading

---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Add ability to truncate input for SuffixSort
7293a02 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Time reading input
d0ce2b8 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Time gatherSplitters, make it more parallel
7fbf725 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Use one task in placess within a parallel region
c72311f 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Small comms opts
a4fc74e 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Add bulkCopy helper
38756e3 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Enable bulkCopy to work with two distributed arrays
d101c3d 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Adjust TestPartitioning for a previous change
8b6afb5 
Adjusts testing for "Fix problem with serial splitters"

---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Fix a bug in bulkCopy
164ad20 
when working with a remote default array and a local one

---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Use bulkCopy in Partitioning
6e29e2d 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
use bulkCopy in SuffixSortImpl
87e6f2f 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Add helper iterators
de5d105 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Make bulkCopy parallel
4acf368 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
divideIntoPages does not yield empty ranges
e100b88 
---
Signed-off-by: Michael Ferguson <[email protected]>
@mppf
Fix bug in bulkCopy
d1084ab 
---
Signed-off-by: Michael Ferguson <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@mppf