diff --git a/CHANGELOG.md b/CHANGELOG.md
index 833a4ef6..d04f98c8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,15 @@
#### ๐ Enhancement
+- Added three "scaled" usage metrics to the per-Dandiset table, normalizing raw usage by how much content each Dandiset actually holds: "Views / Asset", "Downloads / Asset" and "Bytes / Size" (bytes sent per byte stored). Their denominators come from two new data sources, [dandi-cache/dandiset-id-to-number-of-assets](https://github.com/dandi-cache/dandiset-id-to-number-of-assets) and [dandi-cache/dandiset-id-to-total-size](https://github.com/dandi-cache/dandiset-id-to-total-size), which are gzipped JSONL files fetched from the GitHub raw CDN and decompressed in the browser. A failure to load either one leaves the rest of the page working. ([#241](https://github.com/dandi/usage-page/pull/241))
+- Added "Total Size" and "Total Assets" columns to the end of the per-Dandiset table, so the denominators behind the scaled metrics are visible alongside them. Both show "--" for Dandisets missing from the content derivatives. ([#241](https://github.com/dandi/usage-page/pull/241))
+- Added an "Ignore testing datasets" checkbox to the per-Dandiset section's gear-wheel settings panel, which leaves out the Dandisets listed in `TESTING_DANDISET_IDS` โ those whose usage is dominated by automated testing of the archive, and which otherwise dominate the scaled metrics. It is on by default, so a first visit shows research usage rather than testing traffic; turning it off is remembered in the URL (`ignore_testing=false`). It applies to the table (including its download) rather than the plot, and is shown only for the archive-wide selection, the only one whose rows are Dandisets. ([#241](https://github.com/dandi/usage-page/pull/241))
+- Stopped disabling the per-Dandiset gear-wheel settings button in table view, since its panel now holds a table setting as well as the plot-only "Plot Type"; both entries carry an info icon naming the view they apply to, and the panel is titled "Settings" rather than "Plot Settings". ([#241](https://github.com/dandi/usage-page/pull/241))
+- Kept the sort a user has chosen for a table when that table is re-rendered with new rows โ toggling "Ignore testing datasets" or the binary/decimal prefix no longer snaps the table back to its default sort. The remembered sort is tied to the table's container and column set, so a container that goes on to show a different table (the per-asset table replacing the per-Dandiset one) still starts from that table's own default. ([#241](https://github.com/dandi/usage-page/pull/241))
+- Replaced the "Download raw file" item of every table's "Data โพ" menu with "Download table", which hands back a TSV of the table as displayed โ the same columns, the same formatting, and the sort order currently in effect โ rather than the source file, which no longer contains the derived columns. The source file itself is still one click away under "View file on GitHub". ([#241](https://github.com/dandi/usage-page/pull/241))
+- Ordered the per-Dandiset table so each scaled metric sits beside what it is divided by: the per-asset rates with "Total Assets", then the usage totals, closing on "Bytes / Size" and the directly comparable "Total Bytes" and "Total Size" pair. Renamed the raw totals to "Total Bytes", "Total Views" and "Total Downloads", dropped the request columns (requests remain in the plot's hover text and in every other table), and let that table use the full page width (instead of the 1100px the other tables are capped at) so all of its columns are visible at once on a wide screen. Table columns gained an optional `default_sort` flag, used to keep "Total Bytes" the column the table is sorted by on load now that it is no longer the leftmost metric. ([#241](https://github.com/dandi/usage-page/pull/241))
+- Rendered a scaled metric as "--" when its denominator is unknown (a Dandiset missing from the content derivatives, such as 'undetermined') or zero, rather than as a blank, zero, or infinite value. Table rows without a value for the sorted column now always sink to the bottom, in both sort directions. ([#241](https://github.com/dandi/usage-page/pull/241))
+
- Surfaced the view counts now published by [dandi/access-summaries](https://github.com/dandi/access-summaries) (`number_of_views` in `by_day.tsv`, `by_region.tsv` and `by_asset.tsv`; `total_number_of_views` in `totals.json` and `archive_totals.json`) throughout the page: a new "Views" column in every sortable table (over-time, per-Dandiset histogram, per-asset histogram, per-region, AWS regions), a `Views` entry in the hover text of every plot (over-time single-series and both grouped modes, per-Dandiset and per-asset histograms, geographic scatter and choropleth), and a view count in the totals sentence above the plots with an explanatory footnote. ([#240](https://github.com/dandi/usage-page/pull/240))
- Renamed the "Usage" column of every table view to "Bytes". ([#240](https://github.com/dandi/usage-page/pull/240))
- Ordered the metric columns of every table view as "Views", "Downloads", "Requests" (after the leading "Bytes" column). ([#240](https://github.com/dandi/usage-page/pull/240))
@@ -21,12 +30,16 @@
#### ๐ Internal
+- Renamed `AGENTS.md` to `CLAUDE.md` and recorded the American-English spelling convention in it. ([#241](https://github.com/dandi/usage-page/pull/241))
+
- Updated the version-check CI workflow to not trigger on `package-lock.json` changes; updated `AGENTS.md` to match. ([#174](https://github.com/dandi/usage-page/pull/174))
- Updated the version-check workflow to allow dependency-only `package.json` changes (for example Dependabot dependency bumps) without requiring a package version bump. ([#189](https://github.com/dandi/usage-page/pull/189))
- Fixed version-check CI parsing in the dependency-only `package.json` branch by replacing a malformed heredoc with `node -e`, so Dependabot dependency bumps no longer fail unexpectedly. ([#181](https://github.com/dandi/usage-page/pull/181))
#### ๐งช Tests
+- Added unit tests for the new gzipped-JSONL data path (`parse_dandiset_numbers_jsonl`, `decode_maybe_gzipped_response`, `fetch_maybe_gzipped_text`), the `scaled_metric` and `format_ratio` helpers, and the table renderer's handling of missing numeric values. The Chromatic fixtures now serve gzipped asset-count and total-size files, so the snapshot run exercises the client-side decompression too. ([#241](https://github.com/dandi/usage-page/pull/241))
+
- Extracted `escape_html`, `make_cumulative`, `fetchWithRetry`, `apply_view_mode`, `apply_geo_view_mode`, and `render_sortable_table` into a new `src/plot-helpers.ts` module and added 50 unit tests covering all six helpers; raised overall statement coverage from 5% to 13%. ([#175](https://github.com/dandi/usage-page/pull/175))
- Added `stories/PlotSections.stories.js` with Storybook stories for the over-time plot, histogram, geography, and sortable-table components in both dark and light themes. ([#175](https://github.com/dandi/usage-page/pull/175))
- Replaced the live version string in the footer with a fixed mock value (`v0.0.0+test0000`) before taking Chromatic Playwright snapshots so the baseline is not invalidated by version bumps or new commits. ([#178](https://github.com/dandi/usage-page/pull/178))
diff --git a/AGENTS.md b/CLAUDE.md
similarity index 74%
rename from AGENTS.md
rename to CLAUDE.md
index c18ad89b..0c21afab 100644
--- a/AGENTS.md
+++ b/CLAUDE.md
@@ -4,3 +4,4 @@
- Always bump the version in `package.json` appropriately when any file under `src/`, or `package.json` itself, is changed
- Leave a short description of the change or addition in the top `# Upcoming` section of the `CHANGELOG.md`; include the GitHub PR link at the end of each entry in the format `([#N](https://github.com/dandi/usage-page/pull/N))`
- PR titles should be human-readable and in the past tense; they should NOT use conventional commit style
+- Use American English spelling everywhere (code, comments, UI text, changelog entries, and PR descriptions): "normalize" not "normalise", "color" not "colour", "behavior" not "behaviour"
diff --git a/package.json b/package.json
index 5f5a47f6..e9adb4bc 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "access-page",
- "version": "1.8.0",
+ "version": "1.9.0",
"description": "Visualizations of data usage across the archive.",
"private": true,
"type": "module",
diff --git a/src/index.html b/src/index.html
index cb9510db..b04963a0 100644
--- a/src/index.html
+++ b/src/index.html
@@ -248,14 +248,20 @@
-
Plot Settings
+
Settings
+ i
+
+
+ i
+
+
diff --git a/src/plot-helpers.ts b/src/plot-helpers.ts
index 1828b715..e5425cc3 100644
--- a/src/plot-helpers.ts
+++ b/src/plot-helpers.ts
@@ -84,28 +84,107 @@ export async function fetchWithRetry(url: string, options: RequestInit = {}, max
throw new Error("fetchWithRetry: exhausted retries");
}
-// โโ Dandiset ID โ title mapping โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
+// โโ Gzip decoding โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
/**
- * Parses a newline-delimited JSON (JSONL) file where each line is a single
- * `{ "": "" }` object, merging them into one lookup map.
- * Blank lines and lines that fail to parse are skipped so a single malformed
- * entry doesn't prevent the rest of the file from loading.
+ * Decodes a fetched response body that may be gzip-compressed into text.
+ *
+ * The gzipped derivative files are served as raw `application/octet-stream`
+ * bytes (no `Content-Encoding: gzip` header), so the browser hands them over
+ * still compressed and they have to be inflated here. A caching layer that
+ * *does* advertise the encoding will have transparently decompressed the body
+ * already, so the gzip magic number is checked first and an already-plain body
+ * is simply decoded as text.
*/
-export function parse_dandiset_titles_jsonl(text: string): Record<string, string> {
- const titles: Record<string, string> = {};
+export async function decode_maybe_gzipped_response(response: Response): Promise<string> {
+ const bytes = new Uint8Array(await response.arrayBuffer());
+ const is_gzipped = bytes.length > 2 && bytes[0] === 0x1f && bytes[1] === 0x8b;
+ if (!is_gzipped) return new TextDecoder().decode(bytes);
+ if (typeof DecompressionStream === "undefined") {
+ throw new Error("Gzipped data cannot be decoded: DecompressionStream is unavailable.");
+ }
+
+ const stream = new DecompressionStream("gzip");
+ const writer = stream.writable.getWriter();
+ // Feeding the stream is deliberately not awaited here: the writes only
+ // settle once the reader below drains the decompressed output, so awaiting
+ // them first would deadlock. A corrupt body rejects on both ends of the
+ // stream, and it is the reader's rejection that is propagated, so the
+ // writer's duplicate is swallowed rather than left unhandled.
+ writer
+ .write(bytes)
+ .then(() => writer.close())
+ .catch(() => {});
+
+ const reader = stream.readable.getReader();
+ const decoder = new TextDecoder();
+ let text = "";
+ for (;;) {
+ const { done, value } = await reader.read();
+ if (done) break;
+ text += decoder.decode(value, { stream: true });
+ }
+ return text + decoder.decode();
+}
+
+/**
+ * Fetches a (possibly gzipped) file and returns its decoded text, retrying
+ * transient failures via `fetchWithRetry`.
+ */
+export async function fetch_maybe_gzipped_text(url: string): Promise<string> {
+ return decode_maybe_gzipped_response(await fetchWithRetry(url));
+}
+
+// โโ Dandiset ID โ value mappings โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
+
+/**
+ * Walks a newline-delimited JSON (JSONL) file where each line is a single
+ * `{ "<dandiset_id>": <value> }` object, invoking `visit` for every key/value
+ * pair. Blank lines and lines that fail to parse are skipped so a single
+ * malformed entry doesn't prevent the rest of the file from loading.
+ */
+function for_each_jsonl_entry(text: string, visit: (key: string, value: unknown) => void): void {
text.split("\n").forEach((line) => {
const trimmed = line.trim();
if (!trimmed) return;
+ let parsed: unknown;
try {
- Object.assign(titles, JSON.parse(trimmed));
+ parsed = JSON.parse(trimmed);
} catch {
// Skip malformed lines rather than failing the whole page.
+ return;
}
+ if (parsed === null || typeof parsed !== "object") return;
+ Object.entries(parsed as Record<string, unknown>).forEach(([key, value]) => visit(key, value));
+ });
+}
+
+/**
+ * Parses a JSONL file of `{ "<dandiset_id>": "<title>" }` lines into one
+ * lookup map of Dandiset ID โ title.
+ */
+export function parse_dandiset_titles_jsonl(text: string): Record<string, string> {
+ const titles: Record<string, string> = {};
+ for_each_jsonl_entry(text, (key, value) => {
+ titles[key] = value as string;
});
return titles;
}
+/**
+ * Parses a JSONL file of `{ "<dandiset_id>": <number> }` lines (the asset-count
+ * and total-size derivatives) into one lookup map of Dandiset ID โ number.
+ * Entries whose value is not a finite number are dropped, so a malformed value
+ * shows up as missing data rather than as a bogus metric.
+ */
+export function parse_dandiset_numbers_jsonl(text: string): Record<string, number> {
+ const numbers: Record<string, number> = {};
+ for_each_jsonl_entry(text, (key, value) => {
+ if (typeof value === "number" && isFinite(value)) numbers[key] = value;
+ });
+ return numbers;
+}
+
/**
* Formats a Dandiset ID for display, appending " - <title>" when a title is
* known. The "archive" sentinel ID (the whole-archive selection) is always
@@ -215,8 +294,81 @@ export function derive_data_source_urls(raw_url: string): { raw: string; file: s
};
}
+// โโ Table download โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
+
+interface TableColumn {
+ label: string;
+ key: string;
+ numeric: boolean;
+ format_fn?: (val: number) => string;
+ link_fn?: (row: Record<string, unknown>) => string | null;
+ default_sort?: boolean;
+}
+
+/**
+ * Returns the text a table cell displays for one column of one row: numeric
+ * columns go through the column's formatter (or the table's default one),
+ * everything else is stringified as-is.
+ */
+function table_cell_text(col: TableColumn, row: Record<string, unknown>, format_fn: (val: number) => string): string {
+ return col.numeric ? (col.format_fn ?? format_fn)(row[col.key] as number) : String(row[col.key] ?? "");
+}
+
+/**
+ * Serializes a table to tab-separated values exactly as it is displayed: the
+ * same columns in the same order, the same formatted cell text, and `rows` in
+ * whatever order they are passed (the caller passes the current sort order).
+ * This is what the "Download table" menu item hands back, since the on-page
+ * table carries derived columns that no single source file contains.
+ *
+ * Tabs and newlines inside a cell are collapsed to spaces so one stray value
+ * cannot break the column alignment of the whole file.
+ */
+export function build_table_tsv(
+ columns: TableColumn[],
+ rows: Array<Record<string, unknown>>,
+ format_fn: (bytes: number) => string = format_bytes_default
+): string {
+ const sanitize = (text: string) => text.replace(/[\t\r\n]+/g, " ");
+ const lines = [columns.map((col) => sanitize(col.label)).join("\t")];
+ rows.forEach((row) => {
+ lines.push(columns.map((col) => sanitize(table_cell_text(col, row, format_fn))).join("\t"));
+ });
+ return lines.join("\n") + "\n";
+}
+
+/**
+ * Derives a download filename from a table's heading, e.g. "Usage per
+ * Dandiset" โ "usage_per_dandiset.tsv".
+ */
+export function table_download_filename(title: string): string {
+ const slug = title.toLowerCase().replace(/[^a-z0-9]+/g, "_").replace(/^_+|_+$/g, "");
+ return `${slug || "table"}.tsv`;
+}
+
+/**
+ * Hands `text` to the browser as a file download, via a temporary object URL.
+ */
+function download_text_file(text: string, filename: string, mime_type: string): void {
+ const url = URL.createObjectURL(new Blob([text], { type: mime_type }));
+ const link = document.createElement("a");
+ link.href = url;
+ link.download = filename;
+ document.body.appendChild(link);
+ link.click();
+ link.remove();
+ URL.revokeObjectURL(url);
+}
+
// โโ Sortable table renderer โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
+/**
+ * The sort each table is currently under, so that re-rendering it with new
+ * rows can restore it. Keyed by the container element (rather than its ID) so
+ * the state disappears with the element it belongs to.
+ */
+const TABLE_SORT_STATE = new WeakMap<HTMLElement, { key: string; asc: boolean; signature: string }>();
+
/**
* Renders a sortable HTML table inside a container element.
* Clicking a column header re-sorts the table in place and updates the sort
@@ -227,19 +379,22 @@ export function derive_data_source_urls(raw_url: string): { raw: string; file: s
* @param columns - Column definitions. `numeric: true` formats the cell value with
* `format_fn()`; otherwise the raw value is displayed as-is. An optional
* `link_fn(row)` turns a non-numeric cell into a hyperlink: it returns the
- * target URL, or `null` for rows that should stay plain text.
+ * target URL, or `null` for rows that should stay plain text. An optional
+ * `default_sort: true` marks the column the table is sorted by on first
+ * render, for tables whose primary metric is not their leftmost one.
* @param rows - Data rows (plain objects keyed by column.key).
* @param format_fn - Formatter applied to numeric cell values. Defaults to
* `format_bytes` (decimal SI suffixes).
* @param data_url - Optional URL to the source data file; when provided a
- * "Data โพ" menu (GitHub file view / raw download / containing folder)
- * is rendered top-right in the table header. Falls back to a plain
- * "Data" hyperlink when the URL is not a raw.githubusercontent.com URL.
+ * "Data โพ" menu (GitHub file view / download of the table as shown /
+ * containing folder) is rendered top-right in the table header. Falls
+ * back to a plain "Data" hyperlink when the URL is not a
+ * raw.githubusercontent.com URL.
*/
export function render_sortable_table(
container_id: string,
title: string,
- columns: Array<{label: string; key: string; numeric: boolean; format_fn?: (val: number) => string; link_fn?: (row: Record<string, unknown>) => string | null}>,
+ columns: TableColumn[],
rows: Array<Record<string, unknown>>,
format_fn: (bytes: number) => string = format_bytes_default,
data_url?: string
@@ -247,14 +402,29 @@ export function render_sortable_table(
const container = document.getElementById(container_id);
if (!container) return;
- // Default: sort by the first numeric column (the primary "Bytes" metric in
- // every table here) descending, falling back to the last column when no
- // column is numeric. Anchoring to the first numeric column rather than the
- // last keeps the default ordering stable as further metric columns are
- // appended.
+ // Default: sort by the column flagged `default_sort`, else by the first
+ // numeric column (the primary "Bytes" metric in most tables here),
+ // falling back to the last column when no column is numeric. Anchoring to
+ // a named or leading column rather than the last keeps the default ordering
+ // stable as further metric columns are added.
// sort_asc: true = ascending (AโZ / lowโhigh), false = descending (ZโA / highโlow)
- let sort_key = (columns.find((col) => col.numeric) ?? columns[columns.length - 1]).key;
- let sort_asc = false; // start descending so highest values appear first
+ const default_sort_key = (
+ columns.find((col) => col.default_sort) ??
+ columns.find((col) => col.numeric) ??
+ columns[columns.length - 1]
+ ).key;
+
+ // Re-rendering a table with fresh rows (a settings toggle, a reload of the
+ // same view) keeps whatever sort the user last chose for it, instead of
+ // snapping back to the default. The remembered state only applies to a
+ // table with the same columns, so a container that later shows a different
+ // table (the per-asset table replacing the per-Dandiset one, say) starts
+ // from that table's own default.
+ const signature = columns.map((col) => col.key).join(" ");
+ const remembered = TABLE_SORT_STATE.get(container);
+ const restored = remembered?.signature === signature ? remembered : null;
+ let sort_key = restored?.key ?? default_sort_key;
+ let sort_asc = restored?.asc ?? false; // start descending so highest values appear first
function render_table() {
const sorted = [...rows].sort((a, b) => {
@@ -262,7 +432,16 @@ export function render_sortable_table(
const vb = b[sort_key];
const factor = sort_asc ? 1 : -1;
if (typeof va === "number" && typeof vb === "number") {
- return factor * ((va as number) - (vb as number));
+ // Rows without a value for this column (NaN, e.g. a scaled
+ // metric whose denominator is unknown) always sink to the
+ // bottom, in both sort directions.
+ const va_missing = !isFinite(va);
+ const vb_missing = !isFinite(vb);
+ if (va_missing || vb_missing) {
+ if (va_missing && vb_missing) return 0;
+ return va_missing ? 1 : -1;
+ }
+ return factor * (va - vb);
}
// Numeric-aware locale comparison handles Dandiset IDs like "000123"
return factor * String(va).localeCompare(String(vb), undefined, { numeric: true });
@@ -270,13 +449,13 @@ export function render_sortable_table(
let data_link = "";
if (data_url) {
- const { raw, file, folder } = derive_data_source_urls(data_url);
+ const { file, folder } = derive_data_source_urls(data_url);
data_link = file && folder
? `<div class="table-data-menu">` +
`<button type="button" class="table-data-menu-btn" aria-haspopup="true" aria-expanded="false">Data <span class="table-data-caret">โพ</span></button>` +
`<div class="table-data-menu-panel" role="menu">` +
`<a href="${escape_html(file)}" target="_blank" rel="noopener" role="menuitem">View file on GitHub</a>` +
- `<a href="${escape_html(raw)}" target="_blank" rel="noopener" role="menuitem">Download raw file</a>` +
+ `<button type="button" class="table-data-menu-download" role="menuitem">Download table</button>` +
`<a href="${escape_html(folder)}" target="_blank" rel="noopener" role="menuitem">Browse data folder</a>` +
`</div></div>`
: `<a class="table-data-link" href="${escape_html(data_url)}" target="_blank" rel="noopener">Data</a>`;
@@ -340,6 +519,17 @@ export function render_sortable_table(
(menu_btn as HTMLElement).focus();
}
});
+
+ // Downloads the table as it currently stands โ including the sort
+ // order in effect โ rather than the source file behind it.
+ menu.querySelector(".table-data-menu-download")?.addEventListener("click", () => {
+ download_text_file(
+ build_table_tsv(columns, sorted, format_fn),
+ table_download_filename(title),
+ "text/tab-separated-values"
+ );
+ close_menu();
+ });
}
// Attach sort click handlers after innerHTML is set
@@ -352,6 +542,7 @@ export function render_sortable_table(
sort_key = key;
sort_asc = false; // first click on a new column โ descending (highโlow)
}
+ TABLE_SORT_STATE.set(container!, { key: sort_key, asc: sort_asc, signature });
render_table();
});
});
diff --git a/src/plots.ts b/src/plots.ts
index 53df9c23..a599f1e2 100644
--- a/src/plots.ts
+++ b/src/plots.ts
@@ -7,6 +7,9 @@ import {
aggregate_by_timebin,
format_bytes as format_bytes_pure,
bytes_unit,
+ scaled_metric,
+ format_ratio,
+ exclude_testing_dandisets,
} from "./utils.js";
import {
escape_html,
@@ -17,6 +20,8 @@ import {
derive_data_source_urls,
render_sortable_table,
parse_dandiset_titles_jsonl,
+ parse_dandiset_numbers_jsonl,
+ fetch_maybe_gzipped_text,
format_dandiset_label,
} from "./plot-helpers.js";
import { load as loadYaml } from "js-yaml";
@@ -340,6 +345,13 @@ const ALL_DANDISET_TOTALS_URL = `${BASE_URL}/content/totals.json`;
const REGION_CODES_TO_LATITUDE_LONGITUDE_URL = `${BASE_URL}/content/region_codes_to_coordinates.yaml`;
const DANDISET_ID_TO_TITLE_URL =
"https://raw.githubusercontent.com/dandi-cache/dandiset-id-to-title/derivatives/derivatives/dandiset_id_to_title.jsonl";
+// Content of each Dandiset (asset count and stored size), used to scale the raw
+// usage metrics into per-asset and per-stored-byte rates. Both are gzipped
+// JSONL and are decompressed client-side.
+const DANDISET_ID_TO_NUMBER_OF_ASSETS_URL =
+ "https://raw.githubusercontent.com/dandi-cache/dandiset-id-to-number-of-assets/dist/derivatives/dandiset_id_to_number_of_assets.jsonl.gz";
+const DANDISET_ID_TO_TOTAL_SIZE_URL =
+ "https://raw.githubusercontent.com/dandi-cache/dandiset-id-to-total-size/dist/derivatives/dandiset_id_to_total_size.jsonl.gz";
// Landing page for a Dandiset on the DANDI archive.
const DANDI_ARCHIVE_DANDISET_URL = "https://dandiarchive.org/dandiset";
@@ -369,6 +381,11 @@ let REGION_CODES_TO_LATITUDE_LONGITUDE: Record<string, { latitude: number; longi
let ALL_DANDISET_TOTALS: Record<string, DandisetTotals> = {};
// Maps a Dandiset ID to its current title, used to annotate ID displays with a human-readable name.
let DANDISET_TITLES: Record<string, string> = {};
+// Maps a Dandiset ID to the number of assets it contains and to its total
+// stored size in bytes; both are the denominators of the scaled metrics shown
+// in the per-Dandiset table.
+let DANDISET_ASSET_COUNTS: Record<string, number> = {};
+let DANDISET_TOTAL_SIZES: Record<string, number> = {};
let USE_LOG_SCALE = false;
let USE_CUMULATIVE = false;
let USE_OT_LINE_PLOT = false;
@@ -381,6 +398,9 @@ let OVER_TIME_GROUP_BY = "none"; // "none" | "dandisets"
let TOP_N_DANDISETS = 8;
let USE_OVER_TIME_TABLE = false;
let USE_HISTOGRAM_TABLE = false;
+// Defaults on: a first visit should show research usage, not the testing
+// traffic that otherwise leads most of the per-Dandiset metrics.
+let IGNORE_TESTING_DANDISETS = true;
let GEOJSON_DATA: { features: any[] } | null = null;
let NAME_ALIASES: Record<string, Record<string, string>> | null = null;
@@ -475,24 +495,27 @@ function syncFromUrl() {
const histogramRadio = document.querySelector(`input[name="histogram_view"][value="${histogramValue}"]`) as HTMLInputElement | null;
if (histogramRadio) histogramRadio.checked = true;
apply_view_mode("histogram_plot", "histogram_table", USE_HISTOGRAM_TABLE);
- setSettingsBtnDisabled("hist_settings_btn", "hist_settings_panel", USE_HISTOGRAM_TABLE);
+
+ // Ignore testing Dandisets
+ const ignoreTestingCheckbox = document.getElementById("ignore_testing_dandisets") as HTMLInputElement | null;
+ if (ignoreTestingCheckbox) {
+ IGNORE_TESTING_DANDISETS = params.get("ignore_testing") !== "false";
+ ignoreTestingCheckbox.checked = IGNORE_TESTING_DANDISETS;
+ }
+ apply_ignore_testing_visibility();
}
/**
- * Disables or re-enables a settings button. When disabling, also closes the
- * panel if it is currently open.
+ * Shows the "Ignore testing datasets" setting only where it applies: the
+ * per-Dandiset table, which is the archive-wide selection. Any other selection
+ * lists assets rather than Dandisets, so there is nothing to filter.
*/
-function setSettingsBtnDisabled(btnId: string, panelId: string, disabled: boolean): void {
- const btn = document.getElementById(btnId) as HTMLButtonElement | null;
- const panel = document.getElementById(panelId);
- if (btn) {
- btn.disabled = disabled;
- if (disabled && panel) {
- panel.classList.remove("open");
- btn.setAttribute("aria-expanded", "false");
- panel.setAttribute("aria-hidden", "true");
- }
- }
+function apply_ignore_testing_visibility() {
+ const container = document.getElementById("hist_ignore_testing_container");
+ if (!container) return;
+ const selector = document.getElementById("dandiset_selector") as HTMLSelectElement | null;
+ const is_archive = !selector || selector.value === "archive";
+ container.style.display = is_archive ? "" : "none";
}
/**
@@ -695,10 +718,25 @@ window.addEventListener("load", () => {
window.history.pushState({}, "", window.location.pathname + (query ? "?" + query : ""));
apply_view_mode("histogram_plot", "histogram_table", USE_HISTOGRAM_TABLE);
- setSettingsBtnDisabled("hist_settings_btn", "hist_settings_panel", USE_HISTOGRAM_TABLE);
});
});
+ // Add event listener for the "Ignore testing datasets" checkbox
+ const ignoreTestingCheckbox = document.getElementById("ignore_testing_dandisets");
+ if (ignoreTestingCheckbox) {
+ ignoreTestingCheckbox.addEventListener("change", () => {
+ IGNORE_TESTING_DANDISETS = (ignoreTestingCheckbox as HTMLInputElement).checked;
+
+ const params = new URLSearchParams(window.location.search);
+ setUrlParam(params, "ignore_testing", String(IGNORE_TESTING_DANDISETS), "true");
+ const query = params.toString();
+ window.history.pushState({}, "", window.location.pathname + (query ? "?" + query : ""));
+
+ const selected_dandiset = (document.getElementById("dandiset_selector") as HTMLSelectElement | null)?.value ?? "";
+ load_histogram(selected_dandiset);
+ });
+ }
+
// Add event listener for time aggregation radio toggle (Daily / Weekly / Monthly / Yearly)
const timeAggregationRadios = document.querySelectorAll('input[name="time_aggregation"]');
timeAggregationRadios.forEach((radio) => {
@@ -850,8 +888,33 @@ const dandisetTitlesPromise = fetchWithRetry(DANDISET_ID_TO_TITLE_URL)
console.error("Error loading dandiset titles:", error);
});
+// Asset counts and stored sizes are only the denominators of the scaled metrics
+// in the per-Dandiset table, which fall back to "--" when unavailable, so a
+// failure here is logged but does not block the rest of the page.
+const dandisetAssetCountsPromise = fetch_maybe_gzipped_text(DANDISET_ID_TO_NUMBER_OF_ASSETS_URL)
+ .then((asset_counts_text) => {
+ Object.assign(DANDISET_ASSET_COUNTS, parse_dandiset_numbers_jsonl(asset_counts_text));
+ })
+ .catch((error) => {
+ console.error("Error loading dandiset asset counts:", error);
+ });
+
+const dandisetTotalSizesPromise = fetch_maybe_gzipped_text(DANDISET_ID_TO_TOTAL_SIZE_URL)
+ .then((total_sizes_text) => {
+ Object.assign(DANDISET_TOTAL_SIZES, parse_dandiset_numbers_jsonl(total_sizes_text));
+ })
+ .catch((error) => {
+ console.error("Error loading dandiset total sizes:", error);
+ });
+
// Populate the dropdown with IDs and render initial plots only after both fetches complete
-Promise.all([archiveTotalsPromise, allDandisetTotalsPromise, dandisetTitlesPromise])
+Promise.all([
+ archiveTotalsPromise,
+ allDandisetTotalsPromise,
+ dandisetTitlesPromise,
+ dandisetAssetCountsPromise,
+ dandisetTotalSizesPromise,
+])
.then(() => {
// Re-sync from URL here as a safety net: if DOMContentLoaded fired
// before the data was ready, the global state is already correct, but
@@ -895,6 +958,7 @@ Promise.all([archiveTotalsPromise, allDandisetTotalsPromise, dandisetTitlesPromi
const id = validateDandisetId(rawId);
selector.value = id;
apply_over_time_group_by_visibility();
+ apply_ignore_testing_visibility();
update_dandiset_data_link(id);
update_totals(id);
return [
@@ -1754,14 +1818,28 @@ function load_dandiset_histogram(): Promise<void> {
const combined = Object.keys(data)
.map(dandiset_id => {
const raw_id = String(dandiset_id);
+ const bytes = data[dandiset_id].total_bytes_sent as number;
+ const requests = data[dandiset_id].total_number_of_requests as number;
+ const downloads = data[dandiset_id].total_number_of_downloads as number;
+ const views = data[dandiset_id].total_number_of_views as number;
+ // Denominators of the scaled metrics; missing for Dandisets
+ // absent from the content derivatives (such as 'undetermined'),
+ // in which case the ratios come out as NaN and render as "--".
+ const number_of_assets = DANDISET_ASSET_COUNTS[raw_id] ?? NaN;
+ const total_size = DANDISET_TOTAL_SIZES[raw_id] ?? NaN;
return {
raw_id,
dandiset_id: format_dandiset_label(raw_id, DANDISET_TITLES),
title: DANDISET_TITLES[raw_id] ?? "",
- bytes: data[dandiset_id].total_bytes_sent,
- requests: data[dandiset_id].total_number_of_requests as number,
- downloads: data[dandiset_id].total_number_of_downloads as number,
- views: data[dandiset_id].total_number_of_views as number,
+ bytes,
+ requests,
+ downloads,
+ views,
+ bytes_per_size: scaled_metric(bytes, total_size),
+ views_per_asset: scaled_metric(views, number_of_assets),
+ downloads_per_asset: scaled_metric(downloads, number_of_assets),
+ total_size,
+ number_of_assets,
};
})
.sort((a, b) => b.bytes - a.bytes);
@@ -1769,6 +1847,10 @@ function load_dandiset_histogram(): Promise<void> {
// Exclude 'undetermined' from the plot only (table retains all entries)
const plot_combined = combined.filter(item => item.raw_id !== "undetermined");
+ // The table can additionally be told to leave out the Dandisets whose
+ // usage is dominated by automated testing of the archive.
+ const table_rows = exclude_testing_dandisets(combined, IGNORE_TESTING_DANDISETS);
+
const sorted_dandiset_ids = plot_combined.map(item => item.dandiset_id);
const sorted_bytes_sent = plot_combined.map(item => item.bytes);
const sorted_requests = plot_combined.map(item => item.requests);
@@ -1821,14 +1903,31 @@ function load_dandiset_histogram(): Promise<void> {
// Render table view (sortable by column header click; default: bytes descending)
const count_format = (n: number) => n.toLocaleString();
+ // The content columns are unknown for Dandisets missing from the
+ // content derivatives, and show "--" there rather than "NaN".
+ const optional_count_format = (n: number) => (isFinite(n) ? count_format(n) : "--");
+ const optional_bytes_format = (n: number) => (isFinite(n) ? format_bytes(n) : "--");
render_sortable_table("histogram_table", "Usage per Dandiset", [
{ label: "Dandiset ID", key: "raw_id", numeric: false },
{ label: "Name", key: "title", numeric: false, link_fn: (row) => dandiset_archive_url(row.raw_id) },
- { label: "Bytes", key: "bytes", numeric: true },
- { label: "Views", key: "views", numeric: true, format_fn: count_format },
- { label: "Downloads", key: "downloads", numeric: true, format_fn: count_format },
- { label: "Requests", key: "requests", numeric: true, format_fn: count_format },
- ], combined, format_bytes, ALL_DANDISET_TOTALS_URL);
+ // The scaled metrics lead, since they are what makes Dandisets of
+ // very different sizes comparable; the raw totals they are derived
+ // from follow. Each scaled group sits beside its own denominator:
+ // the three per-asset rates with "Total Assets", and "Bytes / Size"
+ // with the "Total Bytes" and "Total Size" pair that ends the table.
+ // "Total Bytes" stays the default sort so the table's initial
+ // ordering still matches the plot beside it.
+ { label: "Views / Asset", key: "views_per_asset", numeric: true, format_fn: format_ratio },
+ { label: "Downloads / Asset", key: "downloads_per_asset", numeric: true, format_fn: format_ratio },
+ { label: "Total Assets", key: "number_of_assets", numeric: true, format_fn: optional_count_format },
+ { label: "Total Views", key: "views", numeric: true, format_fn: count_format },
+ { label: "Total Downloads", key: "downloads", numeric: true, format_fn: count_format },
+ // Bytes sent sits next to bytes stored, the two being directly
+ // comparable, with "Bytes / Size" โ their ratio โ leading them.
+ { label: "Bytes / Size", key: "bytes_per_size", numeric: true, format_fn: format_ratio },
+ { label: "Total Bytes", key: "bytes", numeric: true, default_sort: true },
+ { label: "Total Size", key: "total_size", numeric: true, format_fn: optional_bytes_format },
+ ], table_rows, format_bytes, ALL_DANDISET_TOTALS_URL);
apply_view_mode(plot_element_id, "histogram_table", USE_HISTOGRAM_TABLE);
})
diff --git a/src/styles.css b/src/styles.css
index 284825df..500cd9aa 100644
--- a/src/styles.css
+++ b/src/styles.css
@@ -331,6 +331,16 @@ input[type="number"]:focus {
padding: 0 16px;
}
+/* The per-Dandiset table carries both the scaled metrics and the raw totals
+ they derive from, so it is allowed the full width of the page rather than
+ the 1100px the other tables are capped at, keeping all of its columns
+ visible at once. The table itself is `width: fit-content`, so this is a
+ ceiling and not a stretch: on narrower screens it fills the available width
+ and the container scrolls horizontally. */
+#histogram_table.plot-table {
+ max-width: 100%;
+}
+
.plot-table h3 {
font-weight: 300;
font-size: 1.1em;
@@ -400,17 +410,28 @@ input[type="number"]:focus {
display: flex;
}
-.table-data-menu-panel a {
+.table-data-menu-panel a,
+.table-data-menu-panel button {
display: block;
+ width: 100%;
padding: 6px 9px;
+ border: none;
border-radius: 4px;
+ background: none;
color: var(--color-text);
- text-decoration: none;
+ font: inherit;
font-size: 0.85em;
+ text-align: left;
+ text-decoration: none;
white-space: nowrap;
}
-.table-data-menu-panel a:hover {
+.table-data-menu-panel button {
+ cursor: pointer;
+}
+
+.table-data-menu-panel a:hover,
+.table-data-menu-panel button:hover {
background: rgba(var(--color-accent-rgb), 0.12);
}
diff --git a/src/utils.ts b/src/utils.ts
index 049312bd..58f491e9 100644
--- a/src/utils.ts
+++ b/src/utils.ts
@@ -100,6 +100,67 @@ export function format_bytes(bytes: number, decimals = 2, use_binary = false): s
return `${reduced} ${sizes[i]}`;
}
+/**
+ * Dandisets whose recorded usage comes overwhelmingly from automated testing
+ * of the archive rather than from research use, and which therefore dominate
+ * (and distort) the per-Dandiset metrics. The per-Dandiset table can be told
+ * to leave them out.
+ */
+export const TESTING_DANDISET_IDS = [
+ "000025",
+ "000027",
+ "000029",
+ "000064",
+ "000068",
+ "000126",
+ "000144",
+ "000299",
+ "000411",
+ "000470",
+ "000544",
+ "000717",
+ "000719",
+ "000937",
+ "001083",
+];
+
+/**
+ * Returns `rows` without the testing Dandisets when `exclude` is true, and
+ * unchanged (the same array) when it is false.
+ */
+export function exclude_testing_dandisets<T extends { raw_id: string }>(rows: T[], exclude: boolean): T[] {
+ return exclude ? rows.filter((row) => !TESTING_DANDISET_IDS.includes(row.raw_id)) : rows;
+}
+
+/**
+ * Divides one metric by another to produce a "scaled" (per-unit) metric, for
+ * example bytes sent per byte stored or views per asset. Returns NaN whenever
+ * the ratio would be meaningless โ a missing or non-finite operand, or a
+ * denominator of zero โ so callers can render it as "no data" rather than as
+ * Infinity or a misleading zero.
+ */
+export function scaled_metric(numerator: number | undefined, denominator: number | undefined): number {
+ if (typeof numerator !== "number" || typeof denominator !== "number") return NaN;
+ if (!isFinite(numerator) || !isFinite(denominator) || denominator <= 0) return NaN;
+ return numerator / denominator;
+}
+
+/**
+ * Formats a scaled (per-unit) metric for display in a table cell. Precision
+ * follows magnitude, since these ratios span many orders of magnitude in
+ * practice: values of 100 or more are rounded to whole numbers with thousands
+ * separators, values of at least 1 keep two decimals, and values below 1 keep
+ * two significant digits. Non-finite values (produced by `scaled_metric` when
+ * the ratio is undefined) render as "--".
+ */
+export function format_ratio(value: number): string {
+ if (typeof value !== "number" || !isFinite(value)) return "--";
+ const magnitude = Math.abs(value);
+ if (magnitude >= 100) return value.toLocaleString(undefined, { maximumFractionDigits: 0 });
+ if (magnitude >= 1) return value.toLocaleString(undefined, { maximumFractionDigits: 2 });
+ return value.toLocaleString(undefined, { maximumSignificantDigits: 2 });
+}
+
/**
* Returns just the unit `format_bytes` would use for `bytes` (e.g. "TB", or
* "TiB" in binary mode), without the numeric part. Used to name the unit a
diff --git a/tests/chromatic/visual.test.js b/tests/chromatic/visual.test.js
index c9829e4d..9f2b0298 100644
--- a/tests/chromatic/visual.test.js
+++ b/tests/chromatic/visual.test.js
@@ -1,4 +1,5 @@
import { test, takeSnapshot } from "@chromatic-com/playwright";
+import { gzipSync } from "node:zlib";
// โโ Static fixture data โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
// All external data fetches are intercepted with these fixed values so that
@@ -32,6 +33,19 @@ const DANDISET_TITLES_JSONL = `\
{"000002": "Mock calcium imaging dataset"}
`;
+// Asset counts and stored sizes behind the scaled metrics of the per-Dandiset
+// table. "000003" is deliberately left out of both so the snapshot also covers
+// rows whose scaled metrics are unavailable and render as "--".
+const NUMBER_OF_ASSETS_JSONL = `\
+{"000001": 40}
+{"000002": 12}
+`;
+
+const TOTAL_SIZE_JSONL = `\
+{"000001": 250000000}
+{"000002": 1500000000}
+`;
+
const REGION_COORDS_YAML = `\
US/California:
latitude: 36.7783
@@ -86,6 +100,22 @@ async function setupDataMocks(page) {
await page.route("**/dandiset_id_to_title.jsonl", (route) =>
route.fulfill({ status: 200, contentType: "text/plain", body: DANDISET_TITLES_JSONL }),
);
+ // Served gzipped, exactly as the real derivatives are, so the page's
+ // client-side decompression is exercised by the snapshot run too.
+ await page.route("**/dandiset_id_to_number_of_assets.jsonl.gz", (route) =>
+ route.fulfill({
+ status: 200,
+ contentType: "application/octet-stream",
+ body: gzipSync(Buffer.from(NUMBER_OF_ASSETS_JSONL)),
+ }),
+ );
+ await page.route("**/dandiset_id_to_total_size.jsonl.gz", (route) =>
+ route.fulfill({
+ status: 200,
+ contentType: "application/octet-stream",
+ body: gzipSync(Buffer.from(TOTAL_SIZE_JSONL)),
+ }),
+ );
await page.route(`${BASE_URL}/content/archive_totals.json`, (route) =>
route.fulfill({ status: 200, contentType: "application/json", body: ARCHIVE_TOTALS }),
);
diff --git a/tests/unit/plot-helpers.test.ts b/tests/unit/plot-helpers.test.ts
index e3d39c87..9b9bcb94 100644
--- a/tests/unit/plot-helpers.test.ts
+++ b/tests/unit/plot-helpers.test.ts
@@ -1,4 +1,5 @@
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
+import { gzipSync } from "node:zlib";
import {
escape_html,
make_cumulative,
@@ -8,6 +9,11 @@ import {
derive_data_source_urls,
render_sortable_table,
parse_dandiset_titles_jsonl,
+ parse_dandiset_numbers_jsonl,
+ build_table_tsv,
+ table_download_filename,
+ decode_maybe_gzipped_response,
+ fetch_maybe_gzipped_text,
format_dandiset_label,
} from "../../src/plot-helpers.js";
@@ -576,6 +582,101 @@ describe("parse_dandiset_titles_jsonl", () => {
});
});
+// โโ parse_dandiset_numbers_jsonl โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
+
+describe("parse_dandiset_numbers_jsonl", () => {
+ it("parses one number per line into a single lookup map", () => {
+ const text = '{"000003": 101}\n{"000004": 87}';
+ expect(parse_dandiset_numbers_jsonl(text)).toEqual({ "000003": 101, "000004": 87 });
+ });
+
+ it("keeps zero values", () => {
+ expect(parse_dandiset_numbers_jsonl('{"000003": 0}')).toEqual({ "000003": 0 });
+ });
+
+ it("preserves large byte counts exactly", () => {
+ expect(parse_dandiset_numbers_jsonl('{"001412": 1067232272269263}')).toEqual({
+ "001412": 1067232272269263,
+ });
+ });
+
+ it("skips blank lines and trailing newlines", () => {
+ const text = '{"000003": 101}\n\n\n{"000004": 87}\n';
+ expect(parse_dandiset_numbers_jsonl(text)).toEqual({ "000003": 101, "000004": 87 });
+ });
+
+ it("skips malformed lines without throwing", () => {
+ const text = '{"000003": 101}\nnot json\n{"000004": 87}';
+ expect(parse_dandiset_numbers_jsonl(text)).toEqual({ "000003": 101, "000004": 87 });
+ });
+
+ it("drops entries whose value is not a finite number", () => {
+ const text = '{"000003": 101}\n{"000004": null}\n{"000005": "87"}';
+ expect(parse_dandiset_numbers_jsonl(text)).toEqual({ "000003": 101 });
+ });
+
+ it("returns an empty object for empty input", () => {
+ expect(parse_dandiset_numbers_jsonl("")).toEqual({});
+ });
+});
+
+// โโ decode_maybe_gzipped_response โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
+
+describe("decode_maybe_gzipped_response", () => {
+ const jsonl = '{"000003": 101}\n{"000004": 87}\n';
+
+ it("inflates a gzipped body", async () => {
+ const response = new Response(new Uint8Array(gzipSync(Buffer.from(jsonl))));
+ expect(await decode_maybe_gzipped_response(response)).toBe(jsonl);
+ });
+
+ it("inflates a body large enough to arrive in multiple chunks", async () => {
+ const big = Array.from({ length: 20000 }, (_, i) => `{"${String(i).padStart(6, "0")}": ${i}}`).join("\n");
+ const response = new Response(new Uint8Array(gzipSync(Buffer.from(big))));
+ expect(await decode_maybe_gzipped_response(response)).toBe(big);
+ });
+
+ it("passes through a body that was already decompressed upstream", async () => {
+ const response = new Response(jsonl);
+ expect(await decode_maybe_gzipped_response(response)).toBe(jsonl);
+ });
+
+ it("returns an empty string for an empty body", async () => {
+ const response = new Response("");
+ expect(await decode_maybe_gzipped_response(response)).toBe("");
+ });
+
+ it("rejects when the gzipped body is corrupt", async () => {
+ // Valid gzip magic number followed by garbage
+ const corrupt = new Uint8Array([0x1f, 0x8b, 0x08, 0x00, 0x01, 0x02, 0x03, 0x04]);
+ await expect(decode_maybe_gzipped_response(new Response(corrupt))).rejects.toThrow();
+ });
+});
+
+// โโ fetch_maybe_gzipped_text โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
+
+describe("fetch_maybe_gzipped_text", () => {
+ afterEach(() => {
+ vi.restoreAllMocks();
+ });
+
+ it("fetches and inflates a gzipped file", async () => {
+ const jsonl = '{"000003": 101}\n';
+ vi.stubGlobal(
+ "fetch",
+ vi.fn().mockResolvedValue(new Response(new Uint8Array(gzipSync(Buffer.from(jsonl))), { status: 200 }))
+ );
+ expect(await fetch_maybe_gzipped_text("https://example.com/data.jsonl.gz")).toBe(jsonl);
+ });
+
+ it("propagates a permanent fetch failure", async () => {
+ vi.stubGlobal("fetch", vi.fn().mockResolvedValue({ ok: false, status: 404, statusText: "Not Found" }));
+ await expect(fetch_maybe_gzipped_text("https://example.com/missing.jsonl.gz")).rejects.toThrow(
+ "HTTP error 404: Not Found"
+ );
+ });
+});
+
// โโ format_dandiset_label โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
describe("format_dandiset_label", () => {
@@ -665,18 +766,25 @@ describe("render_sortable_table data menu", () => {
expect(document.querySelector("#my_table a.table-data-link")).toBeNull();
});
- it("renders file, raw, and folder menu items with derived hrefs", () => {
+ it("renders file and folder menu items with derived hrefs", () => {
render_sortable_table("my_table", "Title", columns, rows, fmt, raw_url);
const hrefs = Array.from(
document.querySelectorAll("#my_table .table-data-menu-panel a")
).map((a) => (a as HTMLAnchorElement).href);
expect(hrefs).toEqual([
"https://github.com/dandi/access-summaries/blob/main/content/summaries/000003/by_day.tsv",
- raw_url,
"https://github.com/dandi/access-summaries/tree/main/content/summaries/000003",
]);
});
+ it("offers a table download in place of a link to the raw source file", () => {
+ render_sortable_table("my_table", "Title", columns, rows, fmt, raw_url);
+ const items = Array.from(
+ document.querySelectorAll("#my_table .table-data-menu-panel a, #my_table .table-data-menu-panel button")
+ ).map((el) => el.textContent);
+ expect(items).toEqual(["View file on GitHub", "Download table", "Browse data folder"]);
+ });
+
it("is closed by default and opens on button click", () => {
render_sortable_table("my_table", "Title", columns, rows, fmt, raw_url);
const menu = document.querySelector("#my_table .table-data-menu")!;
@@ -716,3 +824,316 @@ describe("render_sortable_table data menu", () => {
expect(link!.href).toBe("https://example.com/data.tsv");
});
});
+
+// โโ render_sortable_table missing numeric values โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
+
+describe("render_sortable_table with missing numeric values", () => {
+ // NaN is how a scaled metric reports "no ratio available" (unknown or zero
+ // denominator); such rows must never displace rows that do have a value.
+ const columns = [
+ { label: "Name", key: "name", numeric: false },
+ { label: "Ratio", key: "ratio", numeric: true, format_fn: (n: number) => (isFinite(n) ? String(n) : "--") },
+ ];
+ const rows = [
+ { name: "alpha", ratio: 3 },
+ { name: "beta", ratio: NaN },
+ { name: "gamma", ratio: 1 },
+ { name: "delta", ratio: 2 },
+ ];
+
+ const rendered_names = () =>
+ Array.from(document.querySelectorAll("#my_table tbody tr td:first-child")).map((td) => td.textContent);
+
+ beforeEach(() => {
+ document.body.innerHTML = '<div id="my_table"></div>';
+ render_sortable_table("my_table", "Title", columns, rows, (n) => String(n));
+ });
+
+ it("sorts rows without a value last when sorting descending", () => {
+ expect(rendered_names()).toEqual(["alpha", "delta", "gamma", "beta"]);
+ });
+
+ it("keeps rows without a value last when sorting ascending", () => {
+ (document.querySelector('#my_table th[data-key="ratio"]') as HTMLElement).click();
+ expect(rendered_names()).toEqual(["gamma", "delta", "alpha", "beta"]);
+ });
+
+ it("renders the missing value through the column formatter", () => {
+ const last_cell = document.querySelector("#my_table tbody tr:last-child td:last-child")!;
+ expect(last_cell.textContent).toBe("--");
+ });
+});
+
+// โโ render_sortable_table default_sort column flag โโโโโโโโโโโโโโโโโโโโโโโโโโโโ
+
+describe("render_sortable_table default_sort flag", () => {
+ const columns = [
+ { label: "Name", key: "name", numeric: false },
+ { label: "Ratio", key: "ratio", numeric: true },
+ { label: "Total", key: "total", numeric: true, default_sort: true },
+ ];
+ const rows = [
+ { name: "alpha", ratio: 9, total: 100 },
+ { name: "beta", ratio: 5, total: 300 },
+ { name: "gamma", ratio: 7, total: 200 },
+ ];
+
+ beforeEach(() => {
+ document.body.innerHTML = '<div id="my_table"></div>';
+ render_sortable_table("my_table", "Title", columns, rows, (n) => String(n));
+ });
+
+ it("sorts by the flagged column instead of the first numeric one", () => {
+ const sorted_header = document.querySelector("#my_table th.th-sorted");
+ expect((sorted_header as HTMLElement).dataset.key).toBe("total");
+ const first_name = document.querySelector("#my_table tbody tr:first-child td:first-child")!;
+ expect(first_name.textContent).toBe("beta");
+ });
+
+ it("still starts descending on the flagged column", () => {
+ const totals = Array.from(document.querySelectorAll("#my_table tbody tr td:last-child")).map(
+ (td) => td.textContent
+ );
+ expect(totals).toEqual(["300", "200", "100"]);
+ });
+
+ it("lets a click move the sort to another column", () => {
+ (document.querySelector('#my_table th[data-key="ratio"]') as HTMLElement).click();
+ const first_name = document.querySelector("#my_table tbody tr:first-child td:first-child")!;
+ expect(first_name.textContent).toBe("alpha");
+ });
+});
+
+// โโ build_table_tsv โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
+
+describe("build_table_tsv", () => {
+ const columns = [
+ { label: "Name", key: "name", numeric: false },
+ { label: "Total Bytes", key: "bytes", numeric: true },
+ { label: "Views / Asset", key: "ratio", numeric: true, format_fn: (n: number) => (isFinite(n) ? n.toFixed(2) : "--") },
+ ];
+ const rows = [
+ { name: "alpha", bytes: 300, ratio: 1.5 },
+ { name: "beta", bytes: 100, ratio: NaN },
+ ];
+ const fmt = (n: number) => `${n}B`;
+
+ it("writes the column labels as the header row", () => {
+ expect(build_table_tsv(columns, rows, fmt).split("\n")[0]).toBe("Name\tTotal Bytes\tViews / Asset");
+ });
+
+ it("formats cells exactly as the table displays them", () => {
+ const lines = build_table_tsv(columns, rows, fmt).split("\n");
+ expect(lines[1]).toBe("alpha\t300B\t1.50");
+ expect(lines[2]).toBe("beta\t100B\t--");
+ });
+
+ it("preserves the order of the rows it is given", () => {
+ const reversed = build_table_tsv(columns, [...rows].reverse(), fmt).split("\n");
+ expect(reversed[1].startsWith("beta")).toBe(true);
+ expect(reversed[2].startsWith("alpha")).toBe(true);
+ });
+
+ it("ends with a trailing newline", () => {
+ expect(build_table_tsv(columns, rows, fmt).endsWith("\n")).toBe(true);
+ });
+
+ it("renders a missing non-numeric cell as an empty field", () => {
+ const tsv = build_table_tsv(columns, [{ bytes: 1, ratio: 1 }], fmt);
+ expect(tsv.split("\n")[1]).toBe("\t1B\t1.00");
+ });
+
+ it("collapses tabs and newlines inside a cell so the columns stay aligned", () => {
+ const tsv = build_table_tsv(columns, [{ name: "a\tb\nc", bytes: 1, ratio: 1 }], fmt);
+ expect(tsv.split("\n")[1]).toBe("a b c\t1B\t1.00");
+ });
+
+ it("writes only a header row when there are no rows", () => {
+ expect(build_table_tsv(columns, [], fmt)).toBe("Name\tTotal Bytes\tViews / Asset\n");
+ });
+});
+
+// โโ table_download_filename โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
+
+describe("table_download_filename", () => {
+ it("slugifies the table heading", () => {
+ expect(table_download_filename("Usage per Dandiset")).toBe("usage_per_dandiset.tsv");
+ });
+
+ it("collapses punctuation runs into single underscores", () => {
+ expect(table_download_filename("Usage per region (top 10)")).toBe("usage_per_region_top_10.tsv");
+ });
+
+ it("trims leading and trailing separators", () => {
+ expect(table_download_filename(" Usage per asset ")).toBe("usage_per_asset.tsv");
+ });
+
+ it("falls back to a generic name when the title has no usable characters", () => {
+ expect(table_download_filename("โ")).toBe("table.tsv");
+ });
+});
+
+// โโ render_sortable_table download menu item โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
+
+describe("render_sortable_table table download", () => {
+ const columns = [
+ { label: "Name", key: "name", numeric: false },
+ { label: "Total Bytes", key: "bytes", numeric: true },
+ ];
+ const rows = [
+ { name: "alpha", bytes: 100 },
+ { name: "beta", bytes: 300 },
+ ];
+ const fmt = (n: number) => `${n}B`;
+ const raw_url = "https://raw.githubusercontent.com/dandi/access-summaries/main/content/totals.json";
+
+ let created_blobs: Blob[];
+ let revoked: string[];
+
+ beforeEach(() => {
+ document.body.innerHTML = '<div id="my_table"></div>';
+ created_blobs = [];
+ revoked = [];
+ vi.stubGlobal("URL", {
+ ...URL,
+ createObjectURL: vi.fn((blob: Blob) => {
+ created_blobs.push(blob);
+ return "blob:mock-url";
+ }),
+ revokeObjectURL: vi.fn((url: string) => {
+ revoked.push(url);
+ }),
+ });
+ render_sortable_table("my_table", "Usage per Dandiset", columns, rows, fmt, raw_url);
+ });
+
+ afterEach(() => {
+ vi.restoreAllMocks();
+ vi.unstubAllGlobals();
+ });
+
+ const click_download = () =>
+ (document.querySelector("#my_table .table-data-menu-download") as HTMLElement).click();
+
+ it("builds a TSV of the table in its current sort order", async () => {
+ click_download();
+ expect(created_blobs).toHaveLength(1);
+ // Default sort is the first numeric column descending, so beta leads
+ expect(await created_blobs[0].text()).toBe("Name\tTotal Bytes\nbeta\t300B\nalpha\t100B\n");
+ });
+
+ it("follows the sort order chosen by the user", async () => {
+ (document.querySelector('#my_table th[data-key="bytes"]') as HTMLElement).click();
+ click_download();
+ expect(await created_blobs[0].text()).toBe("Name\tTotal Bytes\nalpha\t100B\nbeta\t300B\n");
+ });
+
+ it("names the file after the table heading", () => {
+ // The temporary anchor is removed again after the click, so capture it
+ // as it is created.
+ const anchors: HTMLAnchorElement[] = [];
+ const create_element = document.createElement.bind(document);
+ vi.spyOn(document, "createElement").mockImplementation((tag: string, options?: ElementCreationOptions) => {
+ const element = create_element(tag, options);
+ if (tag === "a") anchors.push(element as HTMLAnchorElement);
+ return element;
+ });
+
+ click_download();
+
+ expect(anchors.at(-1)!.download).toBe("usage_per_dandiset.tsv");
+ });
+
+ it("revokes the object URL once the download has started", () => {
+ click_download();
+ expect(revoked).toEqual(["blob:mock-url"]);
+ });
+
+ it("closes the menu after downloading", () => {
+ (document.querySelector("#my_table .table-data-menu-btn") as HTMLElement).click();
+ expect(document.querySelector("#my_table .table-data-menu")!.classList.contains("open")).toBe(true);
+ click_download();
+ expect(document.querySelector("#my_table .table-data-menu")!.classList.contains("open")).toBe(false);
+ });
+});
+
+// โโ render_sortable_table sort persistence across re-renders โโโโโโโโโโโโโโโโโโ
+
+describe("render_sortable_table sort persistence", () => {
+ const columns = [
+ { label: "Name", key: "name", numeric: false },
+ { label: "Ratio", key: "ratio", numeric: true },
+ { label: "Total", key: "total", numeric: true, default_sort: true },
+ ];
+ const rows = [
+ { name: "alpha", ratio: 9, total: 100 },
+ { name: "beta", ratio: 5, total: 300 },
+ { name: "gamma", ratio: 7, total: 200 },
+ ];
+ const fmt = (n: number) => String(n);
+
+ const sorted_key = () => (document.querySelector("#my_table th.th-sorted") as HTMLElement).dataset.key;
+ const indicator = () => document.querySelector("#my_table th.th-sorted .sort-indicator")!.textContent;
+ const first_name = () => document.querySelector("#my_table tbody tr:first-child td:first-child")!.textContent;
+
+ beforeEach(() => {
+ document.body.innerHTML = '<div id="my_table"></div>';
+ render_sortable_table("my_table", "Title", columns, rows, fmt);
+ });
+
+ it("keeps the user's sort column when the same table is re-rendered", () => {
+ (document.querySelector('#my_table th[data-key="ratio"]') as HTMLElement).click();
+ expect(sorted_key()).toBe("ratio");
+
+ // Same table, fewer rows โ as when a filter setting is toggled
+ render_sortable_table("my_table", "Title", columns, rows.slice(0, 2), fmt);
+
+ expect(sorted_key()).toBe("ratio");
+ expect(first_name()).toBe("alpha");
+ });
+
+ it("keeps the sort direction as well as the column", () => {
+ const th = document.querySelector('#my_table th[data-key="ratio"]') as HTMLElement;
+ th.click();
+ th.click(); // second click flips to ascending
+ expect(indicator()).toBe("โฒ");
+
+ render_sortable_table("my_table", "Title", columns, rows, fmt);
+
+ expect(sorted_key()).toBe("ratio");
+ expect(indicator()).toBe("โฒ");
+ expect(first_name()).toBe("beta");
+ });
+
+ it("starts from the default sort when the columns change", () => {
+ (document.querySelector('#my_table th[data-key="ratio"]') as HTMLElement).click();
+
+ const other_columns = [
+ { label: "Name", key: "name", numeric: false },
+ { label: "Bytes", key: "bytes", numeric: true },
+ ];
+ render_sortable_table("my_table", "Other table", other_columns, [{ name: "alpha", bytes: 1 }], fmt);
+
+ expect(sorted_key()).toBe("bytes");
+ });
+
+ it("does not leak one table's sort into another table", () => {
+ document.body.innerHTML = '<div id="my_table"></div><div id="other_table"></div>';
+ render_sortable_table("my_table", "Title", columns, rows, fmt);
+ (document.querySelector('#my_table th[data-key="ratio"]') as HTMLElement).click();
+
+ render_sortable_table("other_table", "Title", columns, rows, fmt);
+
+ expect((document.querySelector("#other_table th.th-sorted") as HTMLElement).dataset.key).toBe("total");
+ });
+
+ it("uses the default sort for a freshly created container", () => {
+ (document.querySelector('#my_table th[data-key="ratio"]') as HTMLElement).click();
+
+ // Replacing the container element discards the state tied to it
+ document.body.innerHTML = '<div id="my_table"></div>';
+ render_sortable_table("my_table", "Title", columns, rows, fmt);
+
+ expect(sorted_key()).toBe("total");
+ });
+});
diff --git a/tests/unit/utils.test.js b/tests/unit/utils.test.js
index 500f421a..96b4497c 100644
--- a/tests/unit/utils.test.js
+++ b/tests/unit/utils.test.js
@@ -7,6 +7,10 @@ import {
aggregate_by_timebin,
format_bytes,
bytes_unit,
+ scaled_metric,
+ format_ratio,
+ exclude_testing_dandisets,
+ TESTING_DANDISET_IDS,
} from "../../src/utils.ts";
// โโ setUrlParam โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
@@ -290,3 +294,118 @@ describe("bytes_unit", () => {
expect(bytes_unit(1e40)).toBe("YB");
});
});
+
+// โโ scaled_metric โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
+
+describe("scaled_metric", () => {
+ it("divides the numerator by the denominator", () => {
+ expect(scaled_metric(600, 4)).toBe(150);
+ });
+
+ it("returns a fractional ratio when the numerator is smaller", () => {
+ expect(scaled_metric(1, 8)).toBe(0.125);
+ });
+
+ it("returns zero when the numerator is zero", () => {
+ expect(scaled_metric(0, 101)).toBe(0);
+ });
+
+ it("returns NaN for a zero denominator instead of Infinity", () => {
+ expect(scaled_metric(500, 0)).toBeNaN();
+ });
+
+ it("returns NaN for a negative denominator", () => {
+ expect(scaled_metric(500, -1)).toBeNaN();
+ });
+
+ it("returns NaN when either operand is missing", () => {
+ expect(scaled_metric(undefined, 4)).toBeNaN();
+ expect(scaled_metric(600, undefined)).toBeNaN();
+ });
+
+ it("returns NaN when either operand is not finite", () => {
+ expect(scaled_metric(NaN, 4)).toBeNaN();
+ expect(scaled_metric(600, Infinity)).toBeNaN();
+ });
+
+ it("computes bytes sent per stored byte for realistic magnitudes", () => {
+ expect(scaled_metric(164445122622529, 2559248010229)).toBeCloseTo(64.255, 3);
+ });
+});
+
+// โโ format_ratio โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
+
+describe("format_ratio", () => {
+ it("renders values of 100 or more as whole numbers with thousands separators", () => {
+ expect(format_ratio(94966.4)).toBe("94,966");
+ expect(format_ratio(100)).toBe("100");
+ });
+
+ it("keeps two decimals for values between 1 and 100", () => {
+ expect(format_ratio(20.1333)).toBe("20.13");
+ expect(format_ratio(1)).toBe("1");
+ });
+
+ it("keeps two significant digits for values below 1", () => {
+ expect(format_ratio(0.171914)).toBe("0.17");
+ expect(format_ratio(0.00456)).toBe("0.0046");
+ });
+
+ it("renders zero as '0'", () => {
+ expect(format_ratio(0)).toBe("0");
+ });
+
+ it("renders non-finite values as '--'", () => {
+ expect(format_ratio(NaN)).toBe("--");
+ expect(format_ratio(Infinity)).toBe("--");
+ expect(format_ratio(-Infinity)).toBe("--");
+ });
+
+ it("renders a missing value as '--'", () => {
+ expect(format_ratio(undefined)).toBe("--");
+ });
+});
+
+// โโ exclude_testing_dandisets โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
+
+describe("exclude_testing_dandisets", () => {
+ // Derived from the list itself rather than restated, so that adding a
+ // testing Dandiset is a one-line change in src/utils.ts.
+ const rows = [
+ { raw_id: "000003" },
+ ...TESTING_DANDISET_IDS.map((raw_id) => ({ raw_id })),
+ { raw_id: "undetermined" },
+ ];
+
+ it("drops every testing Dandiset when asked to", () => {
+ expect(exclude_testing_dandisets(rows, true)).toEqual([{ raw_id: "000003" }, { raw_id: "undetermined" }]);
+ });
+
+ it("returns the rows unchanged when not asked to", () => {
+ expect(exclude_testing_dandisets(rows, false)).toBe(rows);
+ });
+
+ it("does not mutate the input", () => {
+ const row_count = rows.length;
+ exclude_testing_dandisets(rows, true);
+ expect(rows).toHaveLength(row_count);
+ });
+
+ it("lists well-formed six-digit Dandiset IDs", () => {
+ expect(TESTING_DANDISET_IDS.filter((id) => !/^\d{6}$/.test(id))).toEqual([]);
+ });
+
+ it("lists each testing Dandiset once, in ID order", () => {
+ expect(new Set(TESTING_DANDISET_IDS).size).toBe(TESTING_DANDISET_IDS.length);
+ expect([...TESTING_DANDISET_IDS].sort()).toEqual(TESTING_DANDISET_IDS);
+ });
+
+ it("keeps rows whose ID merely resembles a testing ID", () => {
+ const near_misses = [{ raw_id: "0000027" }, { raw_id: "00027" }, { raw_id: "000027x" }];
+ expect(exclude_testing_dandisets(near_misses, true)).toEqual(near_misses);
+ });
+
+ it("returns an empty array for empty input", () => {
+ expect(exclude_testing_dandisets([], true)).toEqual([]);
+ });
+});