Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exception: RuntimeError: structure and input must have equal rank #15

Open
simon-tarr opened this issue Aug 10, 2021 · 3 comments
Open

Comments

@simon-tarr
Copy link

simon-tarr commented Aug 10, 2021

  • PyLandStats version: Latest
  • Python version: 3.8.6
  • Operating System: macOS 11.5.1

Description

Some preamble - I'm actually running pylandstats within a conda environment as I'm developing some code to run with Azure Functions. So I'm not sure if the below error is due to a problem with my conda environment, how Azure Functions is executing the code or whether it is actually a problem with pylandstats.

I'm posting this 'bug' here on the off-chance that someone may have seen this error outside of conda.

Please feel free to delete this bug/issue if it's not in any way related to pylandstats!

Trying to run compute_patch_metrics_df() produces the following error:

[2021-08-10T16:24:51.446Z] System.Private.CoreLib: Exception while executing function: Functions.LandcoverQueue. System.Private.CoreLib: Result: Failure
[2021-08-10T16:24:51.446Z] Exception: RuntimeError: structure and input must have equal rank
[2021-08-10T16:24:51.446Z] Stack:   File "/opt/homebrew/Cellar/azure-functions-core-tools@3/3.0.3477/workers/python/3.8/OSX/X64/azure_functions_worker/dispatcher.py", line 399, in _handle__invocation_request
[2021-08-10T16:24:51.446Z]     call_result = await self._loop.run_in_executor(
[2021-08-10T16:24:51.446Z]   File "/Users/simontarr/opt/miniconda3/envs/ecomap/lib/python3.8/concurrent/futures/thread.py", line 57, in run
[2021-08-10T16:24:51.446Z]     result = self.fn(*self.args, **self.kwargs)
[2021-08-10T16:24:51.446Z]   File "/opt/homebrew/Cellar/azure-functions-core-tools@3/3.0.3477/workers/python/3.8/OSX/X64/azure_functions_worker/dispatcher.py", line 603, in _run_sync_func
[2021-08-10T16:24:51.446Z]     return ExtensionManager.get_sync_invocation_wrapper(context,
[2021-08-10T16:24:51.446Z]   File "/opt/homebrew/Cellar/azure-functions-core-tools@3/3.0.3477/workers/python/3.8/OSX/X64/azure_functions_worker/extension.py", line 215, in _raw_invocation_wrapper
[2021-08-10T16:24:51.446Z]     result = function(**args)
[2021-08-10T16:24:51.446Z]   File "/Users/simontarr/Documents/ecomap/LandcoverQueue/__init__.py", line 98, in main
[2021-08-10T16:24:51.446Z]     x = ls.compute_patch_metrics_df()
[2021-08-10T16:24:51.446Z]   File "/Users/simontarr/Documents/ecomap/.venv/lib/python3.8/site-packages/pylandstats/landscape.py", line 2527, in compute_patch_metrics_df
[2021-08-10T16:24:51.446Z]     metrics_dfs = [self._patch_class_ser]
[2021-08-10T16:24:51.446Z]   File "/Users/simontarr/Documents/ecomap/.venv/lib/python3.8/site-packages/pylandstats/landscape.py", line 423, in _patch_class_ser
[2021-08-10T16:24:51.446Z]     np.concatenate([
[2021-08-10T16:24:51.446Z]   File "/Users/simontarr/Documents/ecomap/.venv/lib/python3.8/site-packages/pylandstats/landscape.py", line 424, in <listcomp>
[2021-08-10T16:24:51.446Z]     np.full(self._num_patches_dict[class_val], class_val)
[2021-08-10T16:24:51.446Z]   File "/Users/simontarr/Documents/ecomap/.venv/lib/python3.8/site-packages/pylandstats/landscape.py", line 395, in _num_patches_dict
[2021-08-10T16:24:51.446Z]     self._cached_num_patches_dict = {
[2021-08-10T16:24:51.446Z]   File "/Users/simontarr/Documents/ecomap/.venv/lib/python3.8/site-packages/pylandstats/landscape.py", line 396, in <dictcomp>
[2021-08-10T16:24:51.447Z]     class_val: self.class_label(class_val)[1]
[2021-08-10T16:24:51.447Z]   File "/Users/simontarr/Documents/ecomap/.venv/lib/python3.8/site-packages/pylandstats/landscape.py", line 205, in class_label
[2021-08-10T16:24:51.447Z]     return ndimage.label(self.landscape_arr == class_val,
[2021-08-10T16:24:51.447Z]   File "/Users/simontarr/Documents/ecomap/.venv/lib/python3.8/site-packages/scipy/ndimage/measurements.py", line 183, in label
[2021-08-10T16:24:51.447Z]     raise RuntimeError('structure and input must have equal rank')

My code:

# Load in raster as rioxarray
raster = rxr.open_rasterio(corine_lc['href'])
raster = raster.astype(int)

ls = Landscape(raster.values, res=(len(raster.x), len(raster.y)), nodata=0)

x = ls.compute_patch_metrics_df() # CODE FAILS HERE WITH THE ABOVE ERROR

My conda environment has the following libs/packages installed:

# Name                    Version                   Build  Channel
affine                    2.3.0                      py_0    conda-forge
astunparse                1.6.3              pyhd8ed1ab_0    conda-forge
attrs                     21.2.0             pyhd8ed1ab_0    conda-forge
autopep8                  1.5.7              pyhd8ed1ab_0    conda-forge
beniget                   0.4.1              pyhd8ed1ab_0    conda-forge
boost-cpp                 1.74.0               hff03dee_4    conda-forge
bzip2                     1.0.8                h0d85af4_4    conda-forge
c-ares                    1.17.2               h0d85af4_0    conda-forge
ca-certificates           2021.5.30            h033912b_0    conda-forge
cairo                     1.16.0            he43a7df_1008    conda-forge
cctools                   927.0.2              h5ba7a2e_4    conda-forge
certifi                   2021.5.30        py38h50d1736_0    conda-forge
cfitsio                   3.470                h01dc385_7    conda-forge
clang                     9.0.1           default_hf57f61e_0    conda-forge
clang_osx-64              9.0.1                h05bbb7f_0    conda-forge
clangxx                   9.0.1           default_hf57f61e_0    conda-forge
clangxx_osx-64            9.0.1                h05bbb7f_2    conda-forge
click                     7.1.2              pyh9f0ad1d_0    conda-forge
click-plugins             1.1.1                      py_0    conda-forge
cligj                     0.7.2              pyhd8ed1ab_0    conda-forge
compiler-rt               9.0.1                h6a512c6_3    conda-forge
compiler-rt_osx-64        9.0.1                h99342c6_3    conda-forge
curl                      7.78.0               hb861fe1_0    conda-forge
cycler                    0.10.0                     py_2    conda-forge
decorator                 4.4.2                      py_0    conda-forge
expat                     2.4.1                he49afe7_0    conda-forge
fiona                     1.8.20           py38hc9be10a_1    conda-forge
fontconfig                2.13.1            h10f422b_1005    conda-forge
freetype                  2.10.4               h4cff582_1    conda-forge
freexl                    1.0.6                h0d85af4_0    conda-forge
gast                      0.5.0              pyhd8ed1ab_0    conda-forge
gdal                      3.3.1            py38h140b4a6_1    conda-forge
geopandas                 0.6.1                      py_0  
geos                      3.9.1                he49afe7_2    conda-forge
geotiff                   1.6.0                h26421ea_6    conda-forge
gettext                   0.19.8.1          h7937167_1005    conda-forge
giflib                    5.2.1                hbcb3906_2    conda-forge
gmp                       6.2.1                h2e338ed_0    conda-forge
gmpy2                     2.1.0b5          py38h6052812_0    conda-forge
hdf4                      4.2.15               hefd3b78_3    conda-forge
hdf5                      1.10.6          nompi_hc5d9132_1114    conda-forge
icu                       68.1                 h74dc148_0    conda-forge
jbig                      2.1               h0d85af4_2003    conda-forge
jpeg                      9d                   hbcb3906_0    conda-forge
json-c                    0.15                 hcb556a6_0    conda-forge
kealib                    1.4.14               h31dd65d_2    conda-forge
kiwisolver                1.3.1            py38h12bbefe_1    conda-forge
krb5                      1.19.2               hcfbf3a7_0    conda-forge
lcms2                     2.12                 h577c468_0    conda-forge
ld64                      450.3                h3c32e8a_4    conda-forge
lerc                      2.2.1                h046ec9c_0    conda-forge
libblas                   3.9.0           11_osx64_openblas    conda-forge
libcblas                  3.9.0           11_osx64_openblas    conda-forge
libcurl                   7.78.0               hf45b732_0    conda-forge
libcxx                    12.0.1               habf9029_0    conda-forge
libdap4                   3.20.6               h3e144a0_2    conda-forge
libdeflate                1.7                  h35c211d_5    conda-forge
libedit                   3.1.20191231         h0678c8f_2    conda-forge
libev                     4.33                 haf1e3a3_1    conda-forge
libffi                    3.3                  h046ec9c_2    conda-forge
libgdal                   3.3.1                hd51e85c_1    conda-forge
libgfortran               5.0.0           9_3_0_h6c81a4c_23    conda-forge
libgfortran5              9.3.0               h6c81a4c_23    conda-forge
libglib                   2.68.3               hd556434_0    conda-forge
libiconv                  1.16                 haf1e3a3_0    conda-forge
libkml                    1.3.0             h8fd9edb_1014    conda-forge
liblapack                 3.9.0           11_osx64_openblas    conda-forge
libllvm9                  9.0.1                h223d4b2_3    conda-forge
libnetcdf                 4.8.0           nompi_hb4d10b0_103    conda-forge
libnghttp2                1.43.0               h07e645a_0    conda-forge
libopenblas               0.3.17          openmp_h3351f45_1    conda-forge
libpng                    1.6.37               h7cec526_2    conda-forge
libpq                     13.3                 hea3049e_0    conda-forge
librttopo                 1.1.0                h5413771_6    conda-forge
libspatialindex           1.9.3                he49afe7_4    conda-forge
libspatialite             5.0.1                h035f608_5    conda-forge
libssh2                   1.9.0                h52ee1ee_6    conda-forge
libtiff                   4.3.0                h1167814_1    conda-forge
libwebp-base              1.2.0                h0d85af4_2    conda-forge
libxml2                   2.9.12               h93ec3fd_0    conda-forge
libzip                    1.8.0                h8b0c345_0    conda-forge
llvm-openmp               12.0.1               hda6cdc1_1    conda-forge
lz4-c                     1.9.3                he49afe7_1    conda-forge
matplotlib                3.4.2            py38h50d1736_0    conda-forge
matplotlib-base           3.4.2            py38hc7d2367_0    conda-forge
mpc                       1.1.0             ha57cd0f_1009    conda-forge
mpfr                      4.1.0                h0f52abe_0    conda-forge
mpmath                    1.2.1              pyhd8ed1ab_0    conda-forge
munch                     2.5.0                      py_0    conda-forge
ncurses                   6.2                  h2e338ed_4    conda-forge
networkx                  2.6.2              pyhd8ed1ab_0    conda-forge
numpy                     1.21.1           py38had91d27_0    conda-forge
olefile                   0.46               pyh9f0ad1d_1    conda-forge
openjpeg                  2.4.0                h6e7aa92_1    conda-forge
openssl                   1.1.1k               h0d85af4_0    conda-forge
pandas                    1.3.1            py38h1f261ad_0    conda-forge
pcre                      8.45                 he49afe7_0    conda-forge
pillow                    8.3.1            py38ha4cf6ea_0  
pip                       21.2.3             pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               hbcb3906_0    conda-forge
ply                       3.11                       py_1    conda-forge
poppler                   21.03.0              h640f9a4_0    conda-forge
poppler-data              0.4.10                        0    conda-forge
postgresql                13.3                 he8fe76e_0    conda-forge
proj                      8.0.1                h1512c50_0    conda-forge
pycodestyle               2.7.0              pyhd8ed1ab_0    conda-forge
pylandstats               2.1.1            py38ha0d09dd_0    conda-forge
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
pyproj                    3.1.0            py38hd0b2b97_3    conda-forge
python                    3.8.10          h0e5c897_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.8                      2_cp38    conda-forge
pythran                   0.9.6            py38ha0d09dd_0    conda-forge
pytz                      2021.1             pyhd8ed1ab_0    conda-forge
rasterio                  1.2.6            py38hfca4e73_2    conda-forge
readline                  8.1                  h05e3726_0    conda-forge
rioxarray                 0.6.1              pyhd8ed1ab_0    conda-forge
rtree                     0.9.7            py38hc59ffc2_2    conda-forge
rust                      1.46.0               h1de35cc_0  
scipy                     1.7.1            py38hd329d04_0    conda-forge
setuptools                49.6.0           py38h50d1736_3    conda-forge
shapely                   1.7.1            py38h1830c62_5    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snuggs                    1.4.7                      py_0    conda-forge
sqlite                    3.36.0               h23a322b_0    conda-forge
sympy                     1.8              py38hecd8cb5_0  
tapi                      1000.10.8            h879752b_4    conda-forge
tiledb                    2.3.3                h8370e7a_0    conda-forge
tk                        8.6.10               h0419947_1    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tornado                   6.1              py38h96a0964_1    conda-forge
transonic                 0.4.10           py38h50d1736_0    conda-forge
tzcode                    2021a                h0d85af4_2    conda-forge
tzdata                    2021a                he74cb21_1    conda-forge
wheel                     0.37.0             pyhd8ed1ab_0    conda-forge
xarray                    0.19.0             pyhd8ed1ab_1    conda-forge
xerces-c                  3.2.3                h379762d_2    conda-forge
xsimd                     7.2.2                h770b8ee_0    conda-forge
xz                        5.2.5                haf1e3a3_1    conda-forge
zlib                      1.2.11            h7795811_1010    conda-forge
zstd                      1.5.0                h582d3a0_0    conda-forge

I have checked which dependencies pylandstats requires with:

conda search pylandstats --info 

pylandstats 2.3.0 py38h12bbefe_0
--------------------------------
file name   : pylandstats-2.3.0-py38h12bbefe_0.tar.bz2
name        : pylandstats
version     : 2.3.0
build       : py38h12bbefe_0
build number: 0
size        : 88 KB
license     : GPL-3.0-or-later
subdir      : osx-64
url         : https://conda.anaconda.org/conda-forge/osx-64/pylandstats-2.3.0-py38h12bbefe_0.tar.bz2
md5         : c519c0ad06ec1fdef9c45d87400f3e3d
timestamp   : 2021-04-23 22:38:58 UTC
dependencies: 
  - gast 0.3.3
  - geopandas
  - libcxx >=11.1.0
  - matplotlib-base
  - numpy
  - pandas
  - python >=3.8,<3.9.0a0
  - python_abi 3.8.* *_cp38
  - pythran
  - rasterio
  - scipy
  - transonic >=0.4.0

and can't see anything listed there which isn't included in my list of packages/libraries in my Conda environment.

Thanks!

@simon-tarr
Copy link
Author

simon-tarr commented Aug 11, 2021

Ater some more testing (still within my Conda environment) I've discovered that other methods also don't work e.g. ls.compute_patch_area() and ls.compute_class_metrics_df(). The above runtime error is returned in all instances. However, more 'simple' metrics e.g. ls.total_area() do work.

I will have to do some experimenting to see whether I can replicate this issue outside Conda.

@simon-tarr
Copy link
Author

The more testing I do, the less convinced I am that it's an issue with Conda/Azure.

I created a brand new Conda environment with Python 3.8.6 (I don't think pylandstats is compatible with 3.9.x). With nothing else in the new environment except the default pkg installations (things like pip etc) I installed pylandstats with:

conda install -n geotest pylandstats

It took a long time to solve the environment, but it got there eventually!

I then opened up PyCharm, set the interpreter to my brand new conda environment (called 'geotest'). I ran the following:

import rioxarray as rxr
from pylandstats import Landscape

raster = rxr.open_rasterio('/Users/simontarr/Downloads/_ags_mapb92e5c0e36514569be92dd92aaf02298.tif')
print(dir(raster))
ls = Landscape(raster.values, res=(len(raster.x), len(raster.y)), nodata=0)
x = ls.compute_patch_metrics_df()

But received the error:

/Users/simontarr/opt/miniconda3/envs/geotest/lib/python3.8/site-packages/rioxarray/_io.py:841: NotGeoreferencedWarning: Dataset has no geotransform, gcps, or rpcs. The identity matrix be returned.
  warnings.warn(str(rio_warning.message), type(rio_warning.message))
Traceback (most recent call last):
  File "/Users/simontarr/opt/miniconda3/envs/geotest/lib/python3.8/site-packages/pylandstats/landscape.py", line 420, in _patch_class_ser
    return self._cached_patch_class_ser
AttributeError: 'Landscape' object has no attribute '_cached_patch_class_ser'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/simontarr/opt/miniconda3/envs/geotest/lib/python3.8/site-packages/pylandstats/landscape.py", line 393, in _num_patches_dict
    return self._cached_num_patches_dict
AttributeError: 'Landscape' object has no attribute '_cached_num_patches_dict'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/simontarr/PycharmProjects/pythonProject/main.py", line 10, in <module>
    x = ls.compute_patch_metrics_df()
  File "/Users/simontarr/opt/miniconda3/envs/geotest/lib/python3.8/site-packages/pylandstats/landscape.py", line 2527, in compute_patch_metrics_df
    metrics_dfs = [self._patch_class_ser]
  File "/Users/simontarr/opt/miniconda3/envs/geotest/lib/python3.8/site-packages/pylandstats/landscape.py", line 423, in _patch_class_ser
    np.concatenate([
  File "/Users/simontarr/opt/miniconda3/envs/geotest/lib/python3.8/site-packages/pylandstats/landscape.py", line 424, in <listcomp>
    np.full(self._num_patches_dict[class_val], class_val)
  File "/Users/simontarr/opt/miniconda3/envs/geotest/lib/python3.8/site-packages/pylandstats/landscape.py", line 395, in _num_patches_dict
    self._cached_num_patches_dict = {
  File "/Users/simontarr/opt/miniconda3/envs/geotest/lib/python3.8/site-packages/pylandstats/landscape.py", line 396, in <dictcomp>
    class_val: self.class_label(class_val)[1]
  File "/Users/simontarr/opt/miniconda3/envs/geotest/lib/python3.8/site-packages/pylandstats/landscape.py", line 205, in class_label
    return ndimage.label(self.landscape_arr == class_val,
  File "/Users/simontarr/opt/miniconda3/envs/geotest/lib/python3.8/site-packages/scipy/ndimage/measurements.py", line 183, in label
    raise RuntimeError('structure and input must have equal rank')
RuntimeError: structure and input must have equal rank

This can be replicated with the .tif file attached to this comment.
example_tif.zip

@simonsaysenjoy
Copy link

Hey @simon-tarr ,

I was facing the same issue and looked into the source code a bit.
The issue for me was that my handed raster (as numpy.object) was of higher dimension than the variable structure in _measurements.py allows. To be more specific: Only ndim 2 is allowed, while I handed ndim 3 which was added due to a masking I performed before handing the variable.

Therefore, maybe check the dimension of your object raster.values:

# Check the dimension
print(type(raster.values))
print(raster.values.shape)
print(raster.values.ndim)

if out_image.ndim >= 3:
      raster.values = np.squeeze(raster.values)

Maybe this helps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants