Skip to content

Commit f1a40f7

Browse files
committed
fixup! Add clipping to merge_corpus
1 parent 52d3ae0 commit f1a40f7

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

pipeline/clean/merge-corpus.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ get_seeded_random() {
4949

5050
paste <(zstdmt -dc "$merged_src") <(zstdmt -dc "$merged_trg") |
5151
${BIN}/dedupe |
52-
shuf --random-source=<(echo "merge-corpus-${SRC}-${TRG}") "${shuf_params[@]}" |
52+
shuf --random-source=<(get_seeded_random) "${shuf_params[@]}" |
5353
zstdmt >"$deduplicated_corpus"
5454

5555
rm "$merged_src" "$merged_trg"

0 commit comments

Comments
 (0)