Skip to content

data.md

Latest commit

 

History

History
278 lines (204 loc) · 10.9 KB

data.md

File metadata and controls

278 lines (204 loc) · 10.9 KB
Raw
sidebar_position 7

data

Manage the full data lifecycle on abc-cluster: move data between your machine and cluster storage (MinIO), acquire data from external sources, share within your group, delete safely, and run S3 plumbing operations.

Credentials and the S3 endpoint are always resolved from the active context — you never pass raw S3 credentials to any command.

The surface is organised in two layers:

  • Porcelain — high-level, task-oriented commands.
  • Plumbing — thin wrappers over local tools (s5cmd, mcli, aria2c, rclone) that mirror the AWS S3 / s5cmd surface. All flags after the arguments pass through to the underlying tool verbatim.

Moving data: you ↔ cluster storage

Command Direction Tool Notes
abc data upload <file> local → MinIO tus resumable, tracked across CLI restarts; best on unreliable networks
abc data download <s3-uri> MinIO → local aria2c + presigned URL resumable; no raw S3 creds on the client
abc data push <file> <s3-uri> local → MinIO s5cmd fast, parallel; no tracking
abc data pull <s3-uri> MinIO → local s5cmd fast, checksum-verified 
# Resumable upload (survives interruption — re-run to resume):
abc data upload ./genome.fa

# Fast push of a directory:
abc data push ./results/ s3://su-mbhg-hostgen/user/calm-dassie/results/

# Pull results back, checksum-verified (re-run skips unchanged files):
abc data pull s3://su-mbhg-hostgen/user/calm-dassie/results/ --destination ./out/

push flags: --parallel <n> (workers), --checksum (skip files whose checksum matches the remote), --dry-run.

Compress & decompress (zstd)

Command What it does
abc data compress <path> zstd-compress a raw file/folder to .zst (BGZF-aware); already-compressed inputs pass through
abc data decompress <path> expand a .zst (or any stock zstd) file/folder, verifying integrity

zstd compresses raw genomic text well (≈8–10× on real VCF, ≈25–30× on TSV/TXT). Already-compressed inputs — gzip, BGZF (.bam, tabix-indexed .vcf.gz), zstd — are detected by magic bytes and passed through unchanged; recompressing BGZF would break tabix random access. Each .zst embeds an integrity frame (original name, size, SHA-256), so decompress verifies the result and compress --replace deletes the source only after a verified round-trip. Output is readable by stock zstd -d.

# Compress a raw VCF; keep only the .zst once verified:
abc data compress ./calls.vcf --level best --replace

# Restore it (integrity-checked):
abc data decompress ./calls.vcf.zst

The same capability is available inline on the transfer commands: abc data upload --compress (compresses before encryption), abc data push --compress, and abc data pull --decompress.

Acquiring data: external → cluster

Command Source Mechanism
abc data fetch <url|accession> HTTP(S) URL, public S3, or accession (SRR, GSE, GCF_, …) server-side Nomad job
abc data download <accession> scientific database by accession redirects to fetch 
# Fetch a file from the internet into your MinIO bucket (runs on the cluster):
abc data fetch https://example.com/data.tar.gz

# Fetch a public sequence dataset by accession (auto-detects SRA → nf-core/fetchngs):
abc data fetch SRR000001

Within the cluster

Command What it does
abc data stage <s3-uri> copy a MinIO object into your workbench ~/data/ (Nomad job on the platform node)
abc data share <s3-uri> --to shared server-side copy into your group's shared/<your-slot>/
abc data share <s3-uri> --to common server-side copy into common/ (admin only)
abc data presign <s3-uri> generate a time-limited presigned URL for any accessible object 
# Make a MinIO file appear in your JupyterLab ~/data/ folder:
abc data stage s3://su-mbhg-hostgen/user/calm-dassie/genome.fa

# Share a result with your group (appears in everyone's ~/shared/ within ~10s):
abc data share s3://su-mbhg-hostgen/user/calm-dassie/results.vcf --to shared

# Hand an external collaborator a 48-hour download link (no cluster account needed):
abc data presign s3://su-mbhg-hostgen/user/calm-dassie/report.pdf --expires 48h

presign flags: --expires <duration> (e.g. 30m, 4h, 48h; max 7d), --method GET|PUT.

Deletion — three tiers

Deletion has three tiers with increasing finality. Choose by intent:

Naming (changed 2026-06-05): delete/del is now the safe, recoverable verb (→ trash, like a desktop "move to Trash"), and remove/rm is the permanent one (like Unix rm). This inverts the earlier mapping — see abc-universe brainstorm abc-data-platform/2026-06-05-invert-remove-delete.md.

Command Alias Tier Reversible? Confirmation
abc data delete <s3-uri> del soft delete to trash Yes (until trash expiry) none
abc data remove <s3-uri> rm permanent, current version No prompts (--yes to skip)
abc data purge <s3-uri> — permanent, all versions No must type purge (--yes to skip)

delete (del) — the safe default

Moves the object to s3://<bucket>/trash/<your-slot>/<original-key>. Recoverable with abc data trash restore until the trash lifecycle rule expires it (default 30 days).

abc data delete s3://su-mbhg-hostgen/user/calm-dassie/old.vcf
# → trashed: …/old.vcf → s3://su-mbhg-hostgen/trash/calm-dassie/user/calm-dassie/old.vcf

--overwrite replaces an object of the same name already in trash (relevant on unversioned buckets).

remove (rm) — permanent, current version

Permanently removes the current version (Unix rm convention). Does not go through trash. Prompts for confirmation unless --yes.

abc data remove s3://su-mbhg-hostgen/user/calm-dassie/scratch.bam
abc data remove s3://su-mbhg-hostgen/user/calm-dassie/tmp/ --recursive --yes

Flags: --yes (skip prompt), --recursive, --dry-run.

--recursive is required for a prefix. A target ending in / names a prefix, and removing one is opt-in: without --recursive the command stops and says so. With it, the prefix is expanded to the glob s5cmd needs (…/tmp/ → …/tmp/*), anchored so it cannot reach a sibling — …/tmp/* never matches …/tmp-old/.

A path with no trailing slash (…/tmp) is ambiguous — object or prefix — and is left alone even under --recursive. Add the slash to mean the prefix; a delete should not widen on a guess.

Preview first with --dry-run. It lists the exact objects that would be removed and deletes nothing:

abc data remove s3://su-mbhg-hostgen/user/calm-dassie/tmp/ --recursive --dry-run
# rm s3://su-mbhg-hostgen/user/calm-dassie/tmp/a.bam
# rm s3://su-mbhg-hostgen/user/calm-dassie/tmp/b.bam

purge — complete erasure, all versions

Removes all versions of an object (or prefix), including delete markers — the operation that satisfies a GDPR/POPIA right-to-erasure request. Requires typing the word purge to confirm. On an unversioned bucket this is equivalent to remove (the CLI says so explicitly).

abc data purge s3://su-mbhg-hostgen/user/calm-dassie/cohort-a/sample01.vcf
# WARNING: purge permanently erases ALL versions … Type 'purge' to confirm:

Flags: --yes (skip the typed confirmation), --dry-run (lists what would be purged, erases nothing).

trash — manage soft-deleted objects

abc data trash list                          # what's in your trash
abc data trash restore <trash-s3-uri>        # move back to the original key
abc data trash empty [--yes]                 # permanently delete all of your trash

trash restore recovers the original key by stripping trash/<slot>/ from the trash path, so the object returns to exactly where it was. --overwrite replaces the original location if it already exists.

Plumbing (S3 / s5cmd surface)

Thin wrappers; flags after the arguments pass through to the underlying tool. Canonical names are full English words; the familiar short forms are registered aliases.

Canonical Alias Wraps Purpose
list ls s5cmd ls list objects or buckets
sync — s5cmd sync sync two prefixes or directories
cat — s5cmd cat stream an object to stdout
pipe — s5cmd pipe upload stdin to an object
disk-usage du s5cmd du storage usage for a prefix
make-bucket mb s5cmd mb create a bucket
remove-bucket rb s5cmd rb remove a bucket
stat — head-object object metadata
copy cp rclone (Nomad job) server-side copy
move mv rclone (Nomad job) server-side move 
abc data list s3://su-mbhg-hostgen/user/calm-dassie/
abc data cat s3://su-mbhg-hostgen/user/calm-dassie/manifest.json | jq .
samtools view -bS in.sam | abc data pipe s3://su-mbhg-hostgen/user/calm-dassie/out.bam
abc data du s3://su-mbhg-hostgen/user/calm-dassie/

# Pass flags through to the underlying tool:
abc data list s3://su-mbhg-hostgen/ --show-fullpath

Tool resolution

Each plumbing command locates its tool in this order:

  1. ABC_BIN_<TOOL> environment variable (e.g. ABC_BIN_S5CMD)
  2. ~/.abc/binaries/<tool> (managed by abc operator tools fetch)
  3. system PATH

If the tool is missing, the command fails with an install hint. mcli is detected even when installed as mc, by verifying it is MinIO Client (not GNU Midnight Commander, which shares the mc name on some systems).

Encrypt / decrypt

Encryption uses the age envelope format — output is decryptable by stock age as well as by abc. There are two independent ways to encrypt, depending on whether the key is yours alone or shared with your group:

Local file, your own passphrase — encrypt/decrypt a file on disk without uploading, using a password only you know:

abc data encrypt ./data.csv --crypt-password "my-secret"   # → data.csv.age
abc data decrypt ./data.csv.age --crypt-password "my-secret"  # → data.csv

If you lose the password, the data cannot be recovered — abc never stores it server-side in this mode. (A control-plane-managed default for this local command is planned but not yet available; today --crypt-password is required, or use --unsafe-local to reuse credentials already saved in ~/.abc/config.yaml.)

Managed, on upload, to your group's key — abc data upload --encrypt encrypts client-side with your active context's group key (native age X25519), recoverable via the platform's key broker — no password to remember or lose:

abc data upload ./results/ --encrypt

Requires an authenticated broker context (cred_source: seedling/v1). The key is scoped to your active group — switch context (abc auth context use <name>) to encrypt against a different group's key; there is no separate --group flag.