Merge pull request #13 from leftfield-geospatial/feature_pytest_click

Feature pytest click
leftfield-geospatial · Oct 15, 2023 · 4420793 · 4420793
2 parents 7fb226c + d8be389
commit 4420793
Show file tree

Hide file tree

Showing 60 changed files with 7,007 additions and 1,827 deletions.
diff --git a/.github/workflows/run-unit-tests_conda-forge.yml b/.github/workflows/run-unit-tests_conda-forge.yml
@@ -0,0 +1,38 @@
+name: Tests (conda-forge)
+
+on:
+  push:
+    branches: [main, feature_pytest_click]
+  pull_request:
+    branches: [main]
+  workflow_dispatch:
+
+jobs:
+  test:
+    runs-on: ${{ matrix.os }}
+    defaults:
+      run:
+        shell: bash -l {0}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ macos-latest, ubuntu-latest, windows-latest ]
+        python-version: ['3.9', '3.10', '3.11']
+    steps:
+    - uses: actions/checkout@v3  # for the test data
+    - uses: conda-incubator/setup-miniconda@v2
+      with:
+        python-version: ${{ matrix.python-version }}
+        channels: conda-forge
+        channel-priority: true
+        activate-environment: test
+        miniforge-variant: Mambaforge
+    - name: Install dependencies
+      run: |
+        mamba info
+        mamba list
+        mamba install -c conda-forge rasterio opencv click tqdm pyyaml pytest pytest-cov
+        mamba list
+    - name: Run unit tests
+      run: |
+        python -m pytest --cov=simple_ortho --cov-report=term-missing --cov-report=xml:coverage.xml ./tests
diff --git a/.github/workflows/run-unit-tests_pypi.yml b/.github/workflows/run-unit-tests_pypi.yml
@@ -0,0 +1,45 @@
+name: Tests (PyPI)
+
+on:
+  push:
+    branches: [main, feature_pytest_click]
+  pull_request:
+    branches: [main]
+  workflow_dispatch:
+
+jobs:
+  test:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ['3.9', '3.10', '3.11']
+        os: [ macos-latest, ubuntu-latest, windows-latest ]
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v4
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        python -m pip install flake8 pytest pytest-cov
+        python -m pip install rasterio opencv-python-headless click tqdm pyyaml
+        python -m pip list
+    - name: Lint with flake8
+      run: |
+        # stop the build if there are Python syntax errors or undefined names
+        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
+        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
+        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
+    - name: Test with pytest
+      timeout-minutes: 5
+      run: |
+        python -m pytest --cov=simple_ortho --cov-report=term-missing --cov-report=xml:coverage.xml ./tests
+    - name: Upload coverage
+      uses: codecov/codecov-action@v3
+      with:
+        fail_ci_if_error: true
+        files: ./coverage.xml
+        token: ${{ secrets.CODECOV_TOKEN }}
diff --git a/.style.yapf b/.style.yapf
@@ -79,7 +79,7 @@ blank_line_before_nested_class_or_def=True
 #        'key1': 'value1',
 #        'key2': 'value2',
 #    })
-coalesce_brackets=False
+coalesce_brackets=True
 
 # The column limit.
 column_limit=120

diff --git a/README.md b/README.md
@@ -1,6 +1,6 @@
 # Simple orthorectification
 
-![banner](./data/outputs/test_example/readme_banner.webp)
+![banner](docs/readme_banner.webp)
 
 Fast and simple orthorectification of images with known DEM and camera model.  Designed and tested on [NGI](http://www.ngi.gov.za/index.php/what-we-do/aerial-photography-and-imagery) aerial imagery.  
 
@@ -10,24 +10,15 @@ Using `conda` is the simplest way to resolve `simple-ortho` binary dependencies.
 ```shell
 conda create -n <environment name> python=3.10 -c conda-forge 
 conda activate <environment name> 
-conda install -c conda-forge rasterio opencv pandas pyyaml shapely
+conda install -c conda-forge rasterio opencv pyyaml click tqdm
 ````
 2) Clone the git repository and link into the conda environment:
 ``` shell
 git clone https://github.com/leftfield-geospatial/simple-ortho.git
-pip install -e simple-ortho
+cd simple-ortho
+pip install -e .
 ```
 
-### Requirements  
-These dependencies are installed in the process above.
-
-  - python >= 3.8
-  - rasterio >= 1.2
-  - opencv >= 4.5
-  - pandas >= 1.2
-  - pyyaml >= 5.4
-  - shapely >= 1.7
-
 ## Usage
 `simple-ortho` functionality can be accessed from the `conda` command line.
 
@@ -52,27 +43,23 @@ Argument | Long form | Description
 `-wc` `<config_path>` | `--write_conf` `<config_path>` | Write current configuration to  `<config_path>` and exit.
 `-v` `{1,2,3,4}` | `--verbosity {1,2,3,4}` | Set the logging level (lower means more logging).  1=debug, 2=info, 3=warning, 4=error (default: 2).
 
-#### Examples
-Orthorectify a single image with a user provided configuration, writing to a specified folder.
-```shell
-simple-ortho -v 2 -rc ./data/inputs/test_example/config.yaml -od ./data/outputs/test_example/ ./data/inputs/test_example/3324c_2015_1004_06_0253_RGB.tif ./data/inputs/test_example/dem.tif ./data/inputs/test_example/camera_pos_ori.txt
-```
+#### Example
 Orthorectify images matching a wildcard, with a user provided configuration, writing to a specified folder.
 ```shell
-simple-ortho -v 2 -rc ./data/inputs/test_example/config.yaml -od ./data/outputs/test_example ./data/inputs/test_example/*_RGB.tif ./data/inputs/test_example/dem.tif ./data/inputs/test_example/camera_pos_ori.txt
+simple-ortho -v 2 -rc config.yaml -od ../outputs *_RGB.tif dem.tif camera_pos_ori.txt
 ```
 
 ### Camera position and orientation
 
-Camera position and orientation for an image is specified in a space-separated text file.  The file format is the same as that used by PCI Geomatica's OrthoEngine i.e. each row specifies the camera position and orientation for an image as follows:    
+Camera position and orientation for an image is specified in a space-separated text file.  Each row specifies the camera position and orientation for an image as follows:    
 ```
-<Image file stem> <Easting (m)> <Northing (m)> <Altitude (m)> <Omega (deg)> <Phi (deg)> <Kappa (deg)> 
+<Image file stem> <Easting> <Northing> <Altitude> <Omega> <Phi> <Kappa> 
 ```
-Where `<Image file stem>` is the source file name excluding extension.  
+Where `<Image file stem>` is the source file name without extension.  
 
 For [`simple-ortho`](#simple-ortho), there should be a row with an `<Image file stem>` corresponding to each image specified by `src_im_file` argument(s).
 
-**Note** that the coordinate reference system (CRS) of camera positions should be a projected, and not geographic CRS.  If the source image(s) aren't projected in this CRS, it should be specified in [``config.yaml``](#configuration-file).  
+**Note** that the coordinate reference system (CRS) of the camera positions should be a projected, and not geographic CRS.  If the source image(s) aren't projected in this CRS, it should be specified in [``config.yaml``](#configuration-file).  Camera (easting, northing, altitude) positions should be given in the units of this CRS (usually meters), and (omega, phi, kappa) orientations in degrees.   
 
 Example file:
 ```
@@ -85,53 +72,54 @@ Example file:
 
 ### Camera type
 
-`simple-ortho` implements common lens distortion models, selectable via the camera type.  The *camera* section of the [configuration file](#configuration-file) contains the camera type and distortion parameter specification.  `simple-ortho` distortion models are compatible with [OpenDroneMap (ODM)](https://opendronemap.org/) / [OpenSFM](https://github.com/mapillary/OpenSfM) and [OpenCV](https://opencv.org/) distortion parameter estimates.  ODM writes parameter values to the *&lt;ODM dataset path&gt;/cameras.json* file, and OpenSFM to the *&lt;OpenSFM dataset path&gt;/camera_models.json* file.  Any parameters not specified will default to zero.  The following camera types are supported.
+`simple-ortho` implements common lens distortion models.  The *camera* section of the [configuration file](#configuration-file) contains the camera type and distortion parameter specification.  `simple-ortho` distortion models are compatible with [OpenDroneMap (ODM)](https://opendronemap.org/) / [OpenSfM](https://github.com/mapillary/OpenSfM) and [OpenCV](https://opencv.org/) distortion parameter estimates.  ODM writes parameter values to the *&lt;ODM dataset path&gt;/cameras.json* file, and OpenSfM to the *&lt;OpenSfM dataset path&gt;/reconstruction.json* file.  Any parameters not specified will default to zero.  The following camera types and distortion parameters are supported.
 
 | Type      | Parameters                                                                          | Description
 |-----------|-------------------------------------------------------------------------------------|------------
-| `pinhole` | None                                                                                | Pinhole camera model with no distortion.
-| `brown`   | `k1`, `k2`, `p1`, `p2`, `k3`, `cx`, `cy`                                            | Brown-Conrady lens distortion compatible with ODM / OpenSFM *brown* parameters, and the 4- and 5- element version of the [generic OpenCV distortion model](https://docs.opencv.org/4.x/d9/d0c/group__calib3d.html). The OpenCV 4- and 5- element models are special cases of the ODM / OpenSFM brown model with `k3, cx, cy = 0` and `cx, cy = 0` respectively.
-| `fisheye` | `k1`, `k2`, `k3`, `k4`                                                              | Fisheye lens distortion compatible ODM / OpenSFM, and [OpenCV](https://docs.opencv.org/4.x/db/d58/group__calib3d__fisheye.html) *fisheye* parameters.  The ODM / OpenSFM model is a special case of the OpenCV version with `k3, k4 = 0`.  
-| `opencv`  | `k1`, `k2`, `p1`, `p2`, `k3`, `k4`, `k5`, `k6`, `s1`, `s2`, `s3`, `s4`, `tx`, `τy` | The full [generic OpenCV distortion model](https://docs.opencv.org/4.x/d9/d0c/group__calib3d.html).  Partial or special cases of the model can be specified by omitting some or all of the parameters; e.g. if no distortion coefficients are specified, this model corresponds to `pinhole`, or if the first 5 distortion coefficients are specified, this model corresponds to `brown` with `cx, cy = 0`.
+| `pinhole` |                                                                                     | Pinhole camera model with no distortion.
+| `brown`   | `k1`, `k2`, `p1`, `p2`, `k3`                                                        | Brown-Conrady lens distortion compatible with ODM / OpenSfM *brown* parameters, and the 4- and 5- element version of the [generic OpenCV distortion model](https://docs.opencv.org/4.x/d9/d0c/group__calib3d.html). 
+| `fisheye` | `k1`, `k2`, `k3`, `k4`                                                              | Fisheye lens distortion compatible ODM / OpenSfM, and [OpenCV](https://docs.opencv.org/4.x/db/d58/group__calib3d__fisheye.html) *fisheye* parameters.  The ODM / OpenSfM model is a special case of the OpenCV version with `k3, k4 = 0`.  
+| `opencv`  | `k1`, `k2`, `p1`, `p2`, `k3`, `k4`, `k5`, `k6`, `s1`, `s2`, `s3`, `s4`, `tx`, `τy`  | The full [generic OpenCV distortion model](https://docs.opencv.org/4.x/d9/d0c/group__calib3d.html).  Partial or special cases of the model can be specified by omitting some or all of the parameters; e.g. if no distortion coefficients are specified, this model corresponds to `pinhole`, or if the first 5 distortion coefficients are specified, this model corresponds to `brown`.
 
 ### Configuration file
 
-Default configuration settings, not passed explicitly on the command line, are read from [config.yaml](config.yaml).  Users can make their own configuration files and pass them to [`simple-ortho`](#simple-ortho) with the `-rc <config_path>` argument.   The configuration file is separated into *camera* and *ortho* sections, with settings for the camera model and orthorectification respectively.  Parameters in each section are described below and commented in [config.yaml](config.yaml).  
+Configuration settings, not passed explicitly on the command line, are read from [config.yaml](config.yaml).  Users can make their own configuration files and pass them to [`simple-ortho`](#simple-ortho) with the `-rc <config_path>` argument.   The configuration file is separated into *camera* and *ortho* sections, with settings for the camera model and orthorectification respectively.  Parameters in each section are described below and commented in [config.yaml](config.yaml).  
 
 | Section  | Parameter       | Description
 |----------|-----------------|------------
-| `camera` | `name`          | Descriptive name
+| `camera` | `name`          | Camera ID.
 |          | `type`          | [Camera type](#camera-type) (`pinhole`, `brown`, `fisheye`, `opencv`).
 |          | `focal_len`     | Focal length in same units/scale as `sensor_size`.  Can be a single value or `[x, y]` pair.
 |          | `sensor_size`   | Optional sensor `[width, height]` in same units/scale as `focal_len`.  If omitted, pixels are assumed square, and`focal_len` should be normalised and unitless:  i.e. `focal_len` = (focal length) / (sensor width).
 |          | `im_size`       | Image `[width, height]` dimensions in pixels.
+|          | `cx`, `cy`      | Principal point offsets in [normalised image coordinates](https://opensfm.readthedocs.io/en/latest/geometry.html#normalized-image-coordinates). Values default to zero if not specified.    
 |          | `k1`, `k2`, ... | Optional distortion coefficients for the `brown`, `fisheye` and `opencv`  [camera types](#camera-type).  Values default to zero if not specified.
-| `ortho`  | `crs`           | CRS of the camera positions and ortho image as an EPSG, proj4 or WKT string.  It should be a projected, and not geographic CRS.  Can be omitted if the source image has this CRS.
-|          | `dem_interp`    | Interpolation method for resampling the DEM (`average`, `bilinear`, `cubic`, `cubic_spline`, `gauss`, `lanczos`).  `cubic_spline` is recommended where the DEM resolution is coarser than the ortho-image resolution.
-|          | `dem_band`      | Index of band in DEM raster to use (1-based).
-|          | `interp`        | Interpolation method to use for warping source to orthorectified image (`nearest`, `average`, `bilinear`, `cubic`, `lanczos`).  `nearest` is recommended where the ortho-image resolution is close to the source image resolution.
-|          | `per_band`      | Remap the source to the ortho-image band-by-band (`True`), or all at once (`False`).  `per_band=False` is generally faster, but requires more memory.   (`True`, `False`).
+| `ortho`  | `crs`           | CRS of the camera positions and ortho image as an EPSG, proj4 or WKT string.  Should be a projected, and not geographic CRS.  Can be omitted if the source image(s) are projected in this CRS.
+|          | `dem_interp`    | Interpolation type for resampling the DEM (`average`, `bilinear`, `cubic`, `lanczos`, `nearest`).  `cubic` or `lanczos` are recommended where the DEM resolution is coarser than the ortho image.
+|          | `dem_band`      | Index of band in DEM image to use (1-based).
+|          | `interp`        | Interpolation type for remapping source to ortho image (`average`, `bilinear`, `cubic`, `lanczos`, `nearest`).  
+|          | `per_band`      | Remap source to ortho image band-by-band (`True`), or all at once (`False`).  `per_band=False` is faster, but requires more memory.  (`True`, `False`).
 |          | `build_ovw`     | Build internal overviews (`True`, `False`).
 |          | `overwrite`     | Overwrite ortho image(s) if they exist (`True`, `False`).
-|          | `write_mask`    | Write an internal mask band - can help remove jpeg noise in nodata area  (`True`, `False`).  (`False` recommended.)
+|          | `write_mask`    | Write an internal mask band - helps remove jpeg noise in nodata area (`True`, `False`).  If omitted, the mask will be written when jpeg compression is used.
 |          | `full_remap`    | Remap source to ortho with full camera model (`True`), or remap undistorted source to ortho with pinhole model (`False`). 
-|          | `dtype`         | Data type of ortho image (`uint8`, `uint16`, `float32` or `float64`).  If no `dtype` is specified the same type as the source image will be used (recommended).
-|          | `resolution`    | Output pixel size `[x, y]` in m.
-|          | `compress`      | Ortho image compression type (`deflate`, `jpeg`, or `auto`). 
+|          | `dtype`         | Data type of ortho image (`uint8`, `uint16`, `float32` or `float64`).  If omitted, the source image `dtype` is used (recommended).
+|          | `resolution`    | Ortho pixel size `[x, y]` in units of the `crs` (usually meters).
+|          | `compress`      | Ortho image compression type (`deflate`, `jpeg`, or `auto`).  `auto` uses jpeg compression for uint8 `dtype`, deflate otherwise.
 
 ## Example Application
-Four [NGI](http://www.ngi.gov.za/index.php/what-we-do/aerial-photography-and-imagery) images before and after orthorectification with simple-ortho.  No radiometric (colour) adjustments have been applied, this can be addressed with [`homonim`](https://github.com/leftfield-geospatial/homonim). 
+Four [NGI](http://www.ngi.gov.za/index.php/what-we-do/aerial-photography-and-imagery) images before and after orthorectification with simple-ortho.  No radiometric (colour) adjustments have been applied, this can be addressed with [`homonim`](https://github.com/leftfield-geospatial/homonim).  
 
-![example](./data/outputs/test_example/readme_eg.webp)
+![example](docs/readme_eg.webp)
 
-Coarse resolution versions of these images, together with supporting data, are included in the [data/inputs/test_example](data/inputs/test_example) directory.  You can orthorectify this data with the following command line (from the simple-ortho directory):
+Coarse resolution versions of these images, together with supporting data, are included in the [tests/data/ngi](tests/data/ngi) directory.  You can orthorectify this data with the following command line (from the simple-ortho directory):
 ```shell
-simple-ortho -v 2 -rc ./data/inputs/test_example/config.yaml -od ./data/outputs/test_example ./data/inputs/test_example/*RGB.tif ./data/inputs/test_example/dem.tif ./data/inputs/test_example/camera_pos_ori.txt
+simple-ortho -v 2 -rc ./tests/data/ngi/config.yaml -od ./tests/data/outputs ./tests/data/ngi/*RGB.tif ./tests/data/ngi/dem.tif ./tests/data/ngi/camera_pos_ori.txt
 ```
 
 ## Known limitations
 
-The `conda` `gdal` package does not support 12bit jpeg compression (the format sometimes used by NGI).  Any tiff compressed in this way would need to be converted using a tool capable of reading these images.  
+The `conda-forge` `rasterio` package does not currently support 12bit jpeg compression (the format sometimes used by NGI).  Any tiff compressed in this way would need to be converted using a tool capable of reading these images.  
 
 ## License
 This project is licensed under the terms of the [Apache-2.0 License](LICENSE).