This code is primarily supported on Debian GNU/Linux and other Linux platforms, but most of it also works equivalently on MacOS and Windows.
This project uses Poetry to manage Python packaging and dependencies. Most day-to-day tasks (such as running unit tests from the command line) are orchestrated through Poetry.
A coding standard is enforced using Black, isort and Pylint.
We rely on pre-commit hooks to ensure that the code is properly-formatted,
clean, and type-safe when it's checked in. The run install step described
below installs the project pre-commit hooks into your repository. These hooks
are configured in .pre-commit-config.yaml.
If necessary, you can temporarily disable a hook using Git's --no-verify
switch. However, keep in mind that the CI build on GitHub enforces these
checks, so the build will fail.
The .gitattributes file controls line endings for the files
in this repository. Instead of relying on automatic behavior, the
.gitattributes file forces most files to have UNIX line endings.
Nearly all prerequisites are managed by Poetry. All you need to do is make sure that you have a working Python 3 enviroment and install Poetry itself.
The project is designed to work with Poetry >= 1.8.0. If you already have an older version of Poetry installed on your system, upgrade it first.
On MacOS, it's easiest to use Homebrew to install Python and pipx:
brew install python3 pipx
Once that's done, make sure the python on your $PATH is Python 3 from
Homebrew (in /usr/local), rather than the standard Python 2 that comes with
older versions of MacOS.
Finally, install Poetry itself and then verify your installation:
pipx install poetry
pipx inject poetry poetry-dynamic-versioning
pipx list --include-injected
To upgrade this installation later, use:
pipx upgrade --include-injected poetry
First, install Python 3 and related tools:
sudo apt-get install python3 python-is-python3 pipx
Once that's done, make sure that the python interpreter on your $PATH is
Python 3.
Finally, install Poetry itself and then verify your installation:
pipx install poetry
pipx inject poetry poetry-dynamic-versioning
pipx list --include-injected
To upgrade this installation later, use:
pipx upgrade --include-injected poetry
First, install Python 3 from your preferred source, either a standard
installer or a meta-installer like Chocolatey. Make sure the python
on your $PATH is Python 3.
Next, install pipx:
python -m pip install --user pipx
Finally, install Poetry itself and then verify your installation:
pipx install poetry
pipx inject poetry poetry-dynamic-versioning
pipx list --include-injected
To upgrade this installation later, use:
pipx upgrade --include-injected poetry
Note: The development environment (the
runscript, etc.) expects a bash shell to be available. On Windows, it works fine with the standard Git Bash.
The run script provides shortcuts for common developer tasks:
$ ./run --help
------------------------------------
Shortcuts for common developer tasks
------------------------------------
Basic tasks:
- run install: Setup the virtualenv via Poetry and install pre-commit hooks
- run format: Run the code formatters
- run checks: Run the code checkers
- run build: Build artifacts in the dist/ directory
- run test: Run the unit tests (use --help to see other options)
- run suite: Run the complete test suite, as for the GitHub Actions CI build
Additional tasks:
- run diagnostics: Print CedarBackup runtime diagnostics
- run docs: Build the Sphinx documentation for readthedocs.io
- run docs -o: Build the Sphinx documentation and open in a browser
- run release: Tag and release the code, triggering GHA to publish artifacts
To run scripts, use poetry directly:
- poetry run cback3
- poetry run cback3-amazons3-sync
- poetry run cback3-span
These are the exact scripts published by Poetry as part of the Python package.
Currently, I use PyCharm Community Edition as my day-to-day IDE. By integrating Black and Pylint, most everything important that can be done from a shell environment can also be done right in PyCharm.
PyCharm offers a good developer experience. However, the underlying configuration on disk mixes together project policy (i.e. preferences about which test runner to use) with system-specific settings (such as the name and version of the active Python interpreter). This makes it impossible to commit complete PyCharm configuration to the Git repository. Instead, the repository contains partial configuration, and there are instructions below about how to manually configure the remaining items.
Before going any further, make sure sure that you have installed all of the system prerequisites discussed above. Then, make sure your environment is in working order. In particular, if you do not run the install step, there will be no virtualenv for PyCharm to use:
./run install && ./run suite
Once you have a working shell development environment, Open (do not
Import) the cedar-backup3 directory in PyCharm, then follow the remaining
instructions below. By using Open, the existing .idea directory will be
retained and all of the existing settings will be used.
As a security precaution, PyCharm does not trust any virtual environment
installed within the repository, such as the Poetry .venv directory. In the
status bar on the bottom right, PyCharm will report No interpreter. Click
on this error and select Add Interpreter. In the resulting dialog, click
Ok to accept the selected environment, which should be the Poetry virtual
environment.
Go to the PyCharm settings and find the cedar-backup3 project. Under
Project Structure, mark both src and tests as source folders. In
the Exclude Files box, enter the following:
LICENSE;NOTICE;PyPI.md;.coverage;.coveragerc;.github;.gitignore;.gitattributes;.htmlcov;.idea;.isort.cfg;.pre-commit-config.yaml;.pylintrc;.pytest_cache;.readthedocs.yml;.tabignore;build;dist;docs/_build;out;poetry.lock;poetry.toml;run;.run;.venv;
When you're done, click Ok. Then, go to the gear icon in the project panel and uncheck Show Excluded Files. This will hide the files and directories in the list above.
In the PyCharm settings, go to Editor > Inspections and be sure that the Project Default profile is selected.
Unit tests are written using Unittest, and API documentation is written using Google Style Python Docstring. In the PyCharm settings, go to Tools > Python Integrated Tools. Under Testing > Default test runner, select Unittest. Under Docstrings > Docstring format, select Google.
Right click on the tests folder in the project explorer and choose Run
'Unittests in tests'. Make sure that all of the tests pass. If you see a slightly
different option (i.e. for "pytest" instead of "Unittest") then you probably
skipped the preferences setup discussed above. You may need to remove the
run configuration before PyCharm will find the right test suite.
Optionally, you might want to set up external tools for some of common developer tasks: code reformatting and the PyLint checks. One nice advantage of doing this is that you can configure an output filter, which makes the Pylint errors clickable. To set up external tools, go to PyCharm settings and find Tools > External Tools. Add the tools as described below.
On Linux or MacOS, you can set up the external tools to invoke the run script
directly.
For this to work, it's important that tools like poetry are on the system
path used by PyCharm. On Linux, depending on how you start PyCharm, your
normal shell environment may or may not be inherited. For instance, I had to
adjust the target of my LXDE desktop shortcut to be the script below, which
sources my profile before running the pycharm.sh shell script:
#!/bin/bash
source ~/.bash_profile
/opt/local/lib/pycharm/pycharm-community-2020.3.2/bin/pycharm.sh| Field | Value |
|---|---|
| Name | Format Code |
| Description | Run the Black and isort code formatters |
| Group | Developer Tools |
| Program | $ProjectFileDir$/run |
| Arguments | format |
| Working directory | $ProjectFileDir$ |
| Synchronize files after execution | Checked |
| Open console for tool outout | Checked |
| Make console active on message in stdout | Unchecked |
| Make console active on message in stderr | Unchecked |
| Output filters | Empty |
| Field | Value |
|---|---|
| Name | Run Pylint Checks |
| Description | Run the Pylint code checks |
| Group | Developer Tools |
| Program | $ProjectFileDir$/run |
| Arguments | pylint |
| Working directory | $ProjectFileDir$ |
| Synchronize files after execution | Unchecked |
| Open console for tool outout | Checked |
| Make console active on message in stdout | Checked |
| Make console active on message in stderr | Checked |
| Output filters | $FILE_PATH$:$LINE$:$COLUMN.* |
On Windows, PyCharm has problems invoking the run script. The trick is to
invoke the Bash interpreter and tell it to invoke the run script. The
examples below assume that you have installed Git Bash in its standard location
under C:\Program Files\Git. If it is somewhere else on your system, just
change the path for bash.exe.
| Field | Value |
|---|---|
| Name | Format Code |
| Description | Run the code formatters |
| Group | Developer Tools |
| Program | powershell.exe |
| Arguments | `& 'C:\Program Files\Git\bin\bash.exe' -l "./run" format |
| Working directory | $ProjectFileDir$ |
| Synchronize files after execution | Checked |
| Open console for tool outout | Checked |
| Make console active on message in stdout | Unchecked |
| Make console active on message in stderr | Unchecked |
| Output filters | Empty |
| Field | Value |
|---|---|
| Name | Run Pylint Checks |
| Description | Run the Pylint code checks |
| Group | Developer Tools |
| Program | powershell.exe |
| Arguments | `& 'C:\Program Files\Git\bin\bash.exe' -l "./run" pylint |
| Working directory | $ProjectFileDir$ |
| Synchronize files after execution | Unchecked |
| Open console for tool outout | Checked |
| Make console active on message in stdout | Checked |
| Make console active on message in stderr | Checked |
| Output filters | $FILE_PATH$:$LINE$:$COLUMN.* |
Documentation at Read the Docs is generated via a GitHub hook. So, there is no formal release process for the documentation.
Code is released to PyPI. There is a partially-automated process to publish a new release.
Note: In order to publish code, you must must have push permissions to the GitHub repo.
Ensure that you are on the master branch. Releases must always be done from
master.
Ensure that the Changelog is up-to-date and reflects all of the changes that
will be published. The top line must show your version as unreleased:
Version 3.3.0 unreleased
Run the release command:
./run release 3.3.0
This command updates NOTICE and Changelog to reflect the release version
and release date, commits those changes, tags the code, and pushes to GitHub.
The new tag triggers a GitHub Actions build that runs the test suite, generates
the artifacts, publishes to PyPI, and finally creates a release from the tag.
Note: This process relies on a PyPI API token with upload permissions for the project. This token is stored in a GitHub Actions secret called
PYPI_TOKEN.