Dev v0.0.2 data preprocess

.gitignore

@@ -1,3 +1,9 @@
+# Big preprocessing output
+/data/processed/processed_depmap_prism_ic50.csv
+
+# Actual config.yml
+config.yml
+
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[codz]

.vscode/settings.json

@@ -0,0 +1,9 @@
+// .vscode/settings.json
+{
+    "python.envFile": "${workspaceFolder}/.env",
+    "python.analysis.extraPaths": [
+      "agentic_system/src",
+      "analysis/src"
+    ],
+    "jupyter.notebookFileRoot": "${workspaceFolder}"
+}

analysis/README.md

@@ -7,12 +7,55 @@ agentic system defined in [`agentic_system/`](../agentic_system/).
 
 ## Experiment/Analysis Overview
 
-**Details TBA**
+### `Configuration`
+
+All notebooks require a `config.yml` file at the project root. 
+This file specifies data paths and API credentials needed for the analysis.
+
+Please refer to `config.yml.template` for guidance on how the file should be formatted.
+
+### `0.data_wrangling` 
+- `0.1.WRANGLE_DEPMAP_PRISM_DATA.ipynb` 
+
+    Preprocesses the raw DepMap PRISM secondary drug repurposing dataset to produce a clean, deduplicated table of drug-cell line-IC50 values. 
+    The script handles deduplication of overlapping entries between the HTS002 and MTS010 screens, prioritizing MTS010 results and highest-quality curve fits (r²).
+
+    **Output:** `data/processed/processed_depmap_prism_ic50.csv` - cleaned dataset with unique (cell line, drug) combinations ready for downstream analysis.
 
 ---
 
 ## Usage
 
-1. Make sure you’ve installed the infrastructure package (from repo root):
 
-    **Details TBA**
+> ⚙️ **Project Setup Note**
+>
+> To run the analysis notebooks or nbconverted scripts correctly, some project setup is required:
+>
+> - **Notebook mode (VS Code / Jupyter)**
+>   - Ensure `.vscode/settings.json` contains a `python.envFile` pointing to a `.env` that sets the `PYTHONPATH` for both partitions:
+>     ```json
+>     {
+>       "python.envFile": "${workspaceFolder}/.env",
+>       "python.analysis.extraPaths": [
+>         "agentic_system/src",
+>         "analysis/src"
+>       ],
+>       "jupyter.notebookFileRoot": "${workspaceFolder}"
+>     }
+>     ```
+>   - Example `.env` (at the repo root):
+>     ```
+>     PYTHONPATH=agentic_system/src:analysis/src:${PYTHONPATH}
+>     ```
+>
+> - **Script mode (running nbconvert-generated `.py`)**
+>   - Perform an **editable install** of the notebook utilities once:
+>     ```bash
+>     pip install -e ./analysis
+>     # or: uv pip install -e ./analysis
+>     ```
+>   - This makes the `nbutils` package importable from the scripts:
+>     ```python
+>     from nbutils.pathing import project_file
+>     config_path = project_file("config.yml")
+>     ```

analysis/nbconvert_all.sh

@@ -0,0 +1,51 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Resolve repo root
+if git rev-parse --show-toplevel &>/dev/null; then
+  REPO_ROOT="$(git rev-parse --show-toplevel)"
+else
+  # Fallback: repo root is parent of this script's directory twice
+  REPO_ROOT="$(realpath "$(dirname "$0")"/..)"
+fi
+
+NB_ROOT="$REPO_ROOT/analysis/notebooks"
+OUT_ROOT="$REPO_ROOT/analysis/scripts"
+
+# Optional --force flag to rebuild everything
+FORCE=0
+if [[ "${1:-}" == "--force" ]]; then
+  FORCE=1
+fi
+
+echo "Repo root: $REPO_ROOT"
+echo "Notebook root: $NB_ROOT"
+echo "Output root: $OUT_ROOT"
+echo
+
+mkdir -p "$OUT_ROOT"
+
+# Find all notebooks (skip checkpoints)
+while IFS= read -r -d '' nb; do
+  rel="${nb#$NB_ROOT/}"                         # path relative to NB_ROOT
+  dest_dir="$OUT_ROOT/$(dirname "$rel")"        # mirror folder structure
+  base="$(basename "$rel" .ipynb)"              # filename without .ipynb
+  out_path="$dest_dir/$base.py"
+
+  mkdir -p "$dest_dir"
+
+  if [[ $FORCE -eq 0 && -f "$out_path" && "$nb" -ot "$out_path" ]]; then
+    echo "Up to date: $rel"
+    continue
+  fi
+
+  echo "Converting: $rel  ->  ${out_path#$REPO_ROOT/}"
+  jupyter nbconvert \
+    --to script \
+    --output-dir "$dest_dir" \
+    "$nb"
+
+done < <(find "$NB_ROOT" -type f -name '*.ipynb' -not -path '*/.ipynb_checkpoints/*' -print0)
+
+echo
+echo "Done."