building revindex databases at scaled 10_000 is fast & (reasonably) low memory!

[ctb]

building a combined revindex of host + plants + gtdb rs220 at k=21:

        Command being timed: "snakemake -j 1"
        User time (seconds): 8456.80
        System time (seconds): 3853.24
        Percent of CPU this job got: 494%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 41:31.08
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 24972504
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 167019
        Minor (reclaiming a frame) page faults: 11090432
        Voluntary context switches: 246040798
        Involuntary context switches: 283432
        Swaps: 0
        File system inputs: 50183808
        File system outputs: 51949592
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 1

the memory usage is probably almost entirely because we are indexing off a manifest, so all the signatures need to be loaded into memory at once.

see /group/ctbrowngrp5/sourmash-db/gtdb+host-2025.01.02 for the snakefile.

[ctb]

database link: sourmash-bio/sourmash#3467 (comment)