doc: Big documentation cleanup

doc: Big documentation cleanup

README.md

@@ -36,15 +36,22 @@ In addition to the common features offered by traditional HPO and NAS libraries,
 ## Getting Started
 
 ### 1. Installation
+NePS requires Python 3.8 or higher. You can install it via pip or from source.
 
 Using pip:
-
 ```bash
 pip install neural-pipeline-search
 ```
 
 > Note: As indicated with the `v0.x.x` version number, NePS is early stage code and APIs might change in the future.
 
+You can install from source by cloning the repository and running:
+```bash
+git clone [email protected]:automl/neps.git
+cd neps
+poetry install
+```
+
 ### 2. Basic Usage
 
 Using `neps` always follows the same pattern:
@@ -111,7 +118,7 @@ if __name__ == "__main__":
 ## Examples
 
 Discover how NePS works through these practical examples:
-* **[Pipeline Space via YAML](neps_examples/basic_usage/defining_search_space)**: Explore how to define the `pipeline_space` using a
+* **[Pipeline Space via YAML](neps_examples/basic_usage/hpo_usage_example.py)**: Explore how to define the `pipeline_space` using a
   YAML file instead of a dictionary.
 
 * **[Hyperparameter Optimization (HPO)](neps_examples/basic_usage/hyperparameters.py)**: Learn the essentials of hyperparameter optimization with NePS.

docs/_code/api_generator.py

@@ -0,0 +1,48 @@
+"""Generate the code reference pages and navigation.
+
+# https://mkdocstrings.github.io/recipes/
+"""
+from __future__ import annotations
+
+import logging
+from pathlib import Path
+
+import mkdocs_gen_files
+
+logger = logging.getLogger(__name__)
+
+# Modules whose members should not include inherited attributes or methods
+NO_INHERITS = tuple()
+
+SRCDIR = Path("neps").absolute().resolve()
+ROOT = SRCDIR.parent
+TAB = "    "
+
+
+if not SRCDIR.exists():
+    raise FileNotFoundError(
+        f"{SRCDIR} does not exist, make sure you are running this from the root of the repository."
+    )
+
+for path in sorted(SRCDIR.rglob("*.py")):
+    module_path = path.relative_to(ROOT).with_suffix("")
+    doc_path = path.relative_to(ROOT).with_suffix(".md")
+    full_doc_path = Path("api", doc_path)
+
+    parts = tuple(module_path.parts)
+
+    if parts[-1] in ("__main__", "__version__", "__init__"):
+        continue
+
+    if any(part.startswith("_") for part in parts):
+        continue
+
+    with mkdocs_gen_files.open(full_doc_path, "w") as fd:
+        ident = ".".join(parts)
+        fd.write(f"::: {ident}")
+
+        if ident.endswith(NO_INHERITS):
+            fd.write(f"\n{TAB}options:")
+            fd.write(f"\n{TAB}{TAB}inherited_members: false")
+
+    mkdocs_gen_files.set_edit_path(full_doc_path, path)

docs/_code/example_generator.py

@@ -0,0 +1,76 @@
+"""Generate the code reference pages and navigation.
+
+# https://mkdocstrings.github.io/recipes/
+"""
+from __future__ import annotations
+
+import logging
+from pathlib import Path
+
+import mkdocs_gen_files
+
+logger = logging.getLogger(__name__)
+
+SRCDIR = Path("neps").absolute().resolve()
+ROOT = SRCDIR.parent
+EXAMPLE_FOLDER = ROOT / "neps_examples"
+TAB = "    "
+
+
+if not SRCDIR.exists():
+    raise FileNotFoundError(
+        f"{SRCDIR} does not exist, make sure you are running this from the root of the repository."
+    )
+
+# Write the index page of the examples
+EXAMPLE_INDEX = EXAMPLE_FOLDER / "README.md"
+if not EXAMPLE_INDEX.exists():
+    raise FileNotFoundError(
+        f"{EXAMPLE_INDEX} does not exist, make sure you are running this from the root of the repository."
+    )
+
+with EXAMPLE_INDEX.open() as fd:
+    example_index_contents = fd.read()
+
+DOCS_EXAMPLE_INDEX = Path("examples", "index.md")
+
+with mkdocs_gen_files.open(DOCS_EXAMPLE_INDEX, "w") as fd:
+    fd.write(example_index_contents)
+
+mkdocs_gen_files.set_edit_path(DOCS_EXAMPLE_INDEX, EXAMPLE_INDEX)
+
+# Now Iterate through each example folder
+for example_dir in EXAMPLE_FOLDER.iterdir():
+    if not example_dir.is_dir():
+        continue
+
+    readme = next((p for p in example_dir.iterdir() if p.name == "README.md"), None)
+    doc_example_dir = Path("examples", example_dir.name)
+
+    # Copy the README.md file to the docs/<example_dir>/index.md
+    if readme is not None:
+        doc_example_index = doc_example_dir / "index.md"
+        with readme.open() as fd:
+            contents = fd.read()
+
+        with mkdocs_gen_files.open(doc_example_index, "w") as fd:
+            fd.write(contents)
+
+        mkdocs_gen_files.set_edit_path(doc_example_index, readme)
+
+    # Copy the contents of all of the examples to the docs/<example_dir>/examples/<example_name>.md
+    for path in example_dir.iterdir():
+        if path.suffix != ".py":
+            continue
+
+        with path.open() as fd:
+            contents = fd.readlines()
+
+        # NOTE: We use quad backticks to escape the code blocks that are present in some of the examples
+        escaped_contents = "".join(["````python\n", *contents, "\n````"])
+
+        markdown_example_path = doc_example_dir / f"{path.stem}.md"
+        with mkdocs_gen_files.open(markdown_example_path, "w") as fd:
+            fd.write(escaped_contents)
+
+        mkdocs_gen_files.set_edit_path(markdown_example_path, path)

docs/api/index.md

@@ -0,0 +1,2 @@
+# API
+Use the tree to navigate the API documentation.

docs/getting_started.md

@@ -4,17 +4,23 @@ Getting started with NePS involves a straightforward yet powerful process, cente
 This approach ensures flexibility and efficiency in evaluating different architecture and hyperparameter configurations
 for your problem.
 
+NePS requires Python 3.8 or higher. You can install it via pip or from source.
+
+```bash
+pip install neural-pipeline-search
+```
+
 ## The 3 Main Components
-1. **Define a [`run_pipeline`](https://automl.github.io/neps/latest/run_pipeline) Function**: This function is essential
+1. **Define a [`run_pipeline=`](./reference/run_pipeline.md) Function**: This function is essential
 for evaluating different configurations. You'll implement the specific logic for your problem within this function.
-For detailed instructions on initializing and effectively using `run_pipeline`, refer to the guide.
+For detailed instructions on initializing and effectively using `run_pipeline=`, refer to the guide.
 
-2. **Establish a [`pipeline_space`](https://automl.github.io/neps/latest/pipeline_space)**: Your search space for
+2. **Establish a [`pipeline_space=`](./reference/pipeline_space.md)**: Your search space for
 defining parameters. You can structure this in various formats, including dictionaries, YAML, or ConfigSpace.
 The guide offers insights into defining and configuring your search space.
 
-3. **Execute with [`neps.run`](https://automl.github.io/neps/latest/neps_run)**: Optimize your `run_pipeline` over
-the `pipeline_space` using this function. For a thorough overview of the arguments and their explanations,
+3. **Execute with [`neps.run=`](./reference/neps_run.md)**: Optimize your `run_pipeline=` over
+the `pipeline_space=` using this function. For a thorough overview of the arguments and their explanations,
 check out the detailed documentation.
 
 By following these steps and utilizing the extensive resources provided in the guides, you can tailor NePS to meet
@@ -27,10 +33,10 @@ In code, the usage pattern can look like this:
 import neps
 import logging
 
-
-# 1. Define a function that accepts hyperparameters and computes the validation error
-def run_pipeline(
-    hyperparameter_a: float, hyperparameter_b: int, architecture_parameter: str
+def run_pipeline( # (1)!
+    hyperparameter_a: float,
+    hyperparameter_b: int,
+    architecture_parameter: str,
 ) -> dict:
     # insert here your own model
     model = MyModel(architecture_parameter)
@@ -40,31 +46,22 @@ def run_pipeline(
         model, hyperparameter_a, hyperparameter_b
     )
 
-    return {  # dict or float(validation error)
-        "loss": validation_error,
+    return {
+        "loss": validation_error, #! (2)
         "info_dict": {
             "training_error": training_error
             # + Other metrics
         },
     }
 
 
-# 2. Define a search space of the parameters of interest; ensure that the names are consistent with those defined
-# in the run_pipeline function
-pipeline_space = dict(
-    hyperparameter_b=neps.IntegerParameter(
-        lower=1, upper=42, is_fidelity=True
-    ),  # Mark 'is_fidelity' as true for a multi-fidelity approach.
-    hyperparameter_a=neps.FloatParameter(
-        lower=0.001, upper=0.1, log=True
-    ),  # If True, the search space is sampled in log space.
-    architecture_parameter=neps.CategoricalParameter(
-        ["option_a", "option_b", "option_c"]
-    ),
-)
+pipeline_space = {  # (3)!
+    "hyperparameter_b":neps.IntegerParameter(1, 42, is_fidelity=True), #! (4)
+    "hyperparameter_a":neps.FloatParameter(1e-3, 1e-1, log=True) #! (5)
+    "architecture_parameter": neps.CategoricalParameter(["option_a", "option_b", "option_c"]),
+}
 
 if __name__ == "__main__":
-    # 3. Run the NePS optimization
     logging.basicConfig(level=logging.INFO)
     neps.run(
         run_pipeline=run_pipeline,
@@ -76,26 +73,31 @@ if __name__ == "__main__":
     )
 ```
 
-## Examples
+1.  Define a function that accepts hyperparameters and computes the validation error.
+2.  Return a dictionary with the objective to minimize and any additional information.
+3.  Define a search space of the parameters of interest; ensure that the names are consistent with those defined in the run_pipeline function.
+4.  Use `is_fidelity=True` for a multi-fidelity approach.
+5.  Use `log=True` for a log-spaced hyperparameter.
+
+!!! tip
 
+    Please visit the [full reference](./reference/neps_run.md) for a more comprehensive walkthrough of defining budgets,
+    optimizers, YAML configuration, parallelism, and more.
+
+## Examples
 Discover the features of NePS through these practical examples:
 
-* **[Hyperparameter Optimization (HPO)](
-https://github.com/automl/neps/blob/master/neps_examples/template/basic_template.py)**: Learn the essentials of
-hyperparameter optimization with NePS.
+* **[Hyperparameter Optimization (HPO)](./examples/template/basic_template.md)**:
+Learn the essentials of hyperparameter optimization with NePS.
 
-* **[Architecture Search with Primitives](
-https://github.com/automl/neps/tree/master/neps_examples/basic_usage/architecture.py)**: Dive into architecture search
-using primitives in NePS.
+* **[Architecture Search with Primitives](./examples/basic_usage/architecture.md)**:
+Dive into architecture search using primitives in NePS.
 
-* **[Multi-Fidelity Optimization](
-https://github.com/automl/neps/tree/master/neps_examples/efficiency/multi_fidelity.py)**: Understand how to leverage
-multi-fidelity optimization for efficient model tuning.
+* **[Multi-Fidelity Optimization](./examples/efficiency/multi_fidelity.md)**:
+Understand how to leverage multi-fidelity optimization for efficient model tuning.
 
-* **[Utilizing Expert Priors for Hyperparameters](
-https://github.com/automl/neps/blob/master/neps_examples/template/priorband_template.py)**:
+* **[Utilizing Expert Priors for Hyperparameters](./examples/template/priorband_template.md)**:
 Learn how to incorporate expert priors for more efficient hyperparameter selection.
 
-* **[Additional NePS Examples](
-https://github.com/automl/neps/tree/master/neps_examples/)**: Explore more examples, including various use cases and
-advanced configurations in NePS.
+* **[Additional NePS Examples](./examples/index.md)**:
+Explore more examples, including various use cases and advanced configurations in NePS.