Gird

Gird is a lightweight & general-purpose Make-like build tool & task runner for Python.

Features

• A simple, expressive, and intuitive rule definition and execution scheme very close to that of Make.
• Configuration in Python, allowing straightforward and familiar usage, without the need for a dedicated rule definition syntax.
• Ability to take advantage of Python's flexibility and possibility to easily integrate with Python libraries and tools.
• Emphasis on API simplicity & ease of use.

Example use cases

• Data science & data analytics workflows.
• Portable CI tasks.
• Less rule-heavy application build setups. (Build time overhead may become noticeable with thousands of rules.)
• Any project with tasks that need to be executed automatically when some dependencies are updated.

Installation

Install Gird from PyPI with pip install gird, or from sources with pip install ..

Gird requires Python version 3.9 or newer, and is supported on Linux & macOS.

Usage

Define rules in girdfile.py. Depending on the composition of a rule definition, a rule can, for example,

• define a recipe to run a task, e.g., to update a target file,
• define prerequisites for the target, such as dependency files or other rules, and
• use Python functions for more complex dependency & recipe functionality.

A rule is invoked via the CLI by gird {target}. To list rules, run gird list.

When invoked, a rule will be run if its target is considered outdated. This is the case if the rule

1. has a Phony target,
2. has a Path target that does not exist,
3. has a Path target and a Path dependency that is more recent than the target,
4. has an outdated Rule/target as a dependency, or
5. has a function dependency that returns True.

Rules with outdated targets are run in topological order within the dependency graph, i.e., all outdated dependencies are updated before the respective targets. By default, rules are run in parallel when possible.

Example rules

A rule with files as its target & dependency

A girdfile.py with the following contents defines a single rule that, when gird package.whl is invoked, builds package.whl whenever module.py is modified. If module.py hasn't been modified, the packaging recipe will not be executed.

import pathlib
import gird

RULE_BUILD = gird.rule(
    target=pathlib.Path("package.whl"),
    deps=pathlib.Path("module.py"),
    recipe="python -m build --wheel",
)

A rule with a phony target

Phony targets can be used when there's not any actual target to update. The recipe of a rule with a phony target is always executed.

RULE_TEST = gird.rule(
    target=gird.Phony("test"),
    recipe="pytest",
)

A rule with other rules as dependencies

This is equivalent to using the targets of the rules as dependencies. Here, a phony target is used to give an alias to a group of other rules.

gird.rule(
    target=gird.Phony("all"),
    deps=[
        RULE_TEST,
        RULE_BUILD,
    ],
)

A compound recipe for mixing Python functions with shell commands

FILE1 = pathlib.Path("file1")

gird.rule(
    target=FILE1,
    recipe=[
        FILE1.touch,
        f"echo text >> {FILE1.resolve()}",
    ],
)

A Python function as a recipe with parameters

Use, e.g., functools.partial to turn a function and its arguments into a callable with no arguments.

import functools
import shutil

FILE2 = pathlib.Path("file2")

gird.rule(
    target=FILE2,
    deps=FILE1,
    recipe=functools.partial(shutil.copy, FILE1, FILE2),
)

A Python function as a dependency to arbitrarily trigger rules

Below, have a remote file re-fetched if it has changed.

def has_remote_changed():
    return get_checksum_local() != get_checksum_remote()

gird.rule(
    target=FILE1,
    deps=has_remote_changed,
    recipe=fetch_remote,
)

Implement the TimeTracked protocol for custom targets & dependencies

Such types are treated identically to Path objects, which respectively have a time of modification that is tracked for resolving outdatedness.

For example, define platform-specific logic to apply dependency tracking on a remote file.

class RemoteFile(gird.TimeTracked):
    def __init__(self, url: str):
        self._url = url
    @property
    def id(self):
        return self._url
    @property
    def timestamp(self):
        return get_remote_file_timestamp(self._url)

gird.rule(
    target=FILE1,
    deps=RemoteFile(URL),
    recipe=fetch_remote,
)

Define rules flexibly

All that matter are the rule function calls that are executed when the girdfile.py is imported. The structure of the file and other implementation details are completely up to the user.

RULES = [
    gird.rule(
        target=source.with_suffix(".gz"),
        deps=gird.rule(
            target=source,
            recipe=functools.partial(fetch_remote, source),
        ),
        recipe=f"gzip -k {source.resolve()}",
    )
    for source in [FILE1, FILE2]
]

Example girdfile.py

This is the girdfile.py of the project itself.

from itertools import chain
from pathlib import Path

from gird import Phony, rule
from scripts import assert_readme_updated, get_wheel_path, render_readme

WHEEL_PATH = get_wheel_path()

RULE_PYTEST = rule(
    target=Phony("pytest"),
    recipe="pytest -n auto --cov=gird --cov-report=xml",
    help="Run pytest & get code coverage report.",
)

RULE_MYPY = rule(
    target=Phony("mypy"),
    recipe="mypy --check-untyped-defs -p gird",
    help="Run mypy.",
)

RULE_CHECK_FORMATTING = rule(
    target=Phony("check_formatting"),
    recipe=[
        "black --check gird scripts test girdfile.py",
        "isort --check gird scripts test girdfile.py",
    ],
    help="Check formatting with Black & isort.",
)

RULE_CHECK_README_UPDATED = rule(
    target=Phony("check_readme_updated"),
    recipe=assert_readme_updated,
    help="Check that README.md is updated based on README_template.md.",
)

RULES_TEST = [
    RULE_PYTEST,
    RULE_MYPY,
    RULE_CHECK_FORMATTING,
    RULE_CHECK_README_UPDATED,
]

rule(
    target=Phony("test"),
    deps=RULES_TEST,
    help="\n".join(f"- {rule.help}" for rule in RULES_TEST),
)

rule(
    target=Path("README.md"),
    deps=chain(
        *(Path(path).iterdir() for path in ("scripts", "gird")),
        [Path("girdfile.py"), Path("pyproject.toml")],
    ),
    recipe=render_readme,
    help="Render README.md based on README_template.md.",
)

# Wrap the rule to build WHEEL_PATH in a phony rule for simpler invocation.
# Don't include the inner rule in `gird list`.
rule(
    target=Phony("build"),
    deps=rule(
        target=WHEEL_PATH,
        recipe="poetry build --format wheel",
        listed=False,
    ),
    help="Build distribution packages for the current version.",
)

Respective output from gird list:

pytest
    Run pytest & get code coverage report.
mypy
    Run mypy.
check_formatting
    Check formatting with Black & isort.
check_readme_updated
    Check that README.md is updated based on README_template.md.
test
    - Run pytest & get code coverage report.
    - Run mypy.
    - Check formatting with Black & isort.
    - Check that README.md is updated based on README_template.md.
README.md
    Render README.md based on README_template.md.
build
    Build distribution packages for the current version.