NETCDF / HDF5 Metadata retrieval issue

[pomadchin]

Describe the bug

Getting the metadata of the NETCDF file can not be performed.
So performing a GDALWarp.get_metadata_domain_list(token, datasetType.value, numberOfAttempts, band, arr) of a file that consists of multiple SUBDATASETS seems to be not possible, even with reading the metadata for the entire file (specifying the band = 0))

To Reproduce

1. download a file (rarchv.2008_028_00_2d.nc4) (contact me if you need a copy)
2. retrieve the metadata:

def getMetadataDomainList(token: Long)(datasetType: DatasetType, band: Int): List[GDALMetadataDomain] = {
    val arr = Array.ofDim[Byte](100, 1 << 10)
    val returnValue = GDALWarp.get_metadata_domain_list(token, datasetType.value, numberOfAttempts, band, arr)

    if (returnValue <= 0) {
      val positiveValue = math.abs(returnValue)
      throw new MalformedProjectionException(
        s"Unable to get the metadata domain list. GDAL Error Code: $positiveValue",
        positiveValue
      )
    }

    (arr.map(new String(_, "UTF-8").trim).filter(_.nonEmpty).toList.map(UserDefinedDomain)).distinct
  }

getMetadataDomainList(token)(SOURCE, 0)

Expected behavior

Should behave like usual GDAL JNI bindings:

>>> ds.GetMetadata('SUBDATASETS')
{'SUBDATASET_1_NAME': 'NETCDF:"rarchv.2008_028_00_2d.nc4":qtot', 'SUBDATASET_1_DESC': '[1x2170x4500] surface_downward_heat_flux_in_air (32-bit floating-point)', 'SUBDATASET_2_NAME': 'NETCDF:"rarchv.2008_028_00_2d.nc4":emp', 'SUBDATASET_2_DESC': '[1x2170x4500] water_flux_into_ocean (32-bit floating-point)', 'SUBDATASET_3_NAME': 'NETCDF:"rarchv.2008_028_00_2d.nc4":surface_temperature_trend', 'SUBDATASET_3_DESC': '[1x2170x4500] surface_temperature_trend (32-bit floating-point)', 'SUBDATASET_4_NAME': 'NETCDF:"archv.2008_028_00_2d.nc4":surface_salinity_trend', 'SUBDATASET_4_DESC': '[1x2170x4500] surface_salinity_trend (32-bit floating-point)', 'SUBDATASET_5_NAME': 'NETCDF:"rarchv.2008_028_00_2d.nc4":ssh', 'SUBDATASET_5_DESC': '[1x2170x4500] sea_surface_elevation (32-bit floating-point)', 'SUBDATASET_6_NAME': 'NETCDF:"rarchv.2008_028_00_2d.nc4":surface_boundary_layer_thickness', 'SUBDATASET_6_DESC': '[1x2170x4500] surface_boundary_layer_thickness (32-bit floating-point)', 'SUBDATASET_7_NAME': 'NETCDF:"rarchv.2008_028_00_2d.nc4":mixed_layer_thickness', 'SUBDATASET_7_DESC': '[1x2170x4500] ocean_mixed_layer_thickness (32-bit floating-point)', 'SUBDATASET_8_NAME': 'NETCDF:"rarchv.2008_028_00_2d.nc4":mixed_layer_temperature', 'SUBDATASET_8_DESC': '[1x2170x4500] mixed_layer_temperature (32-bit floating-point)'}

Environment

• OS: Mac OS 10.14.6
• JVM version: 1.8.0_162
• GDAL version: GDAL 2.4.2, released 2019/06/28

[pomadchin]

Assigned myself to look into it deeper and to add more examples / details about the bug.

[pomadchin]

The story here, is that both SOURCE and WARP Datasets are always created here. But for netCDFs (or other files) it can be impossible sometimes to create a Warp object:

# python code 
>>> import gdal
>>> gdal.Warp('', '/tmp/rarchv.2008_028_00_2d.nc4')
ERROR 1: Input file /tmp/rarchv.2008_028_00_2d.nc4 has no raster bands.

[pomadchin]

@jamesmcclain I tried to use this hack / feature to workaround this issue with minimal changes in the code, but it looks like I can't call CPLTurnFailureIntoWarning function for some reason.

[pomadchin]

I think the right solution in this case would be to fail only when the SOURCE can not be created and to hide WARPED exception until the call time. How do you feel about that (making it ~ lazy)?

[pomadchin]

The code that fails when trying to call a CPLTurnFailureIntoWarning function: pomadchin@02309c6

Failure message:

$ .travis/tests.sh 
make: Entering directory '/workdir/src'
make -C main java/com/azavea/gdal/GDALWarp.class
g++ -I/usr/include/gdal -Wall -Werror -O0 -ggdb3 -DSO_FINI -D_GNU_SOURCE -std=c++14 -I/usr/include -fPIC bindings.cpp -c -o bindings.o
make[1]: Entering directory '/workdir/src/main'
javac -h .. -cp java java/com/azavea/gdal/GDALWarp.java
g++ -I/usr/include/gdal -Wall -Werror -O0 -ggdb3 -DSO_FINI -D_GNU_SOURCE -std=c++14 -I/usr/include -fPIC tokens.cpp -c -o tokens.o
g++ -I/usr/include/gdal -Wall -Werror -O0 -ggdb3 -DSO_FINI -D_GNU_SOURCE -std=c++14 -I/usr/include -fPIC errorcodes.cpp -c -o errorcodes.o
make[1]: Leaving directory '/workdir/src/main'
gcc -Wall -Werror -O0 -ggdb3 -DSO_FINI -D_GNU_SOURCE -I/usr/include/gdal -I/usr/lib/jvm/java-8-openjdk-amd64/include -I/usr/lib/jvm/java-8-openjdk-amd64/include/linux -fPIC com_azavea_gdal_GDALWarp.c -c -o com_azavea_gdal_GDALWarp.o
gcc -Wall -Werror -O0 -ggdb3 -DSO_FINI -D_GNU_SOURCE com_azavea_gdal_GDALWarp.o bindings.o tokens.o errorcodes.o -lgdal -l:libstdc++.a -lpthread -shared -o libgdalwarp_bindings.so
make -C unit_tests tests
make[1]: Entering directory '/workdir/src/unit_tests'
g++ -Wall -Werror -O0 -ggdb3 -DSO_FINI -D_GNU_SOURCE -I.. -std=c++14 -I/usr/include token_tests.cpp -lgdal -L.. -lgdalwarp_bindings -lpthread -lm -o token_tests
g++ -Wall -Werror -O0 -ggdb3 -DSO_FINI -D_GNU_SOURCE -I.. -I/usr/include/gdal -std=c++14 -I/usr/include dataset_tests.cpp -lgdal -L.. -lgdalwarp_bindings -lpthread -lm -o dataset_tests
g++ -Wall -Werror -O0 -ggdb3 -DSO_FINI -D_GNU_SOURCE -I.. -I/usr/include/gdal -std=c++14 -I/usr/include cache_tests.cpp -lgdal -L.. -lgdalwarp_bindings -lpthread -lm -o cache_tests
g++ -Wall -Werror -O0 -ggdb3 -DSO_FINI -D_GNU_SOURCE -I.. -I/usr/include/gdal -std=c++14 -I/usr/include bindings_tests.cpp -lgdal -L.. -lgdalwarp_bindings -lpthread -lm -o bindings_tests
../libgdalwarp_bindings.so: undefined reference to `CPLTurnFailureIntoWarning'
collect2: error: ld returned 1 exit status
Makefile:25: recipe for target 'token_tests' failed
make[1]: *** [token_tests] Error 1
make[1]: *** Waiting for unfinished jobs....
../libgdalwarp_bindings.so: undefined reference to `CPLTurnFailureIntoWarning'
collect2: error: ld returned 1 exit status
Makefile:34: recipe for target 'bindings_tests' failed
make[1]: *** [bindings_tests] Error 1
/tmp/ccS0yFsg.o: In function `locked_dataset::open()':
/workdir/src/unit_tests/../locked_dataset.hpp:706: undefined reference to `CPLTurnFailureIntoWarning'
/workdir/src/unit_tests/../locked_dataset.hpp:715: undefined reference to `CPLTurnFailureIntoWarning'
collect2: error: ld returned 1 exit status
Makefile:31: recipe for target 'cache_tests' failed
make[1]: *** [cache_tests] Error 1
/tmp/cco1AL5i.o: In function `locked_dataset::open()':
/workdir/src/unit_tests/../locked_dataset.hpp:706: undefined reference to `CPLTurnFailureIntoWarning'
/workdir/src/unit_tests/../locked_dataset.hpp:715: undefined reference to `CPLTurnFailureIntoWarning'
collect2: error: ld returned 1 exit status
Makefile:28: recipe for target 'dataset_tests' failed
make[1]: *** [dataset_tests] Error 1
make[1]: Leaving directory '/workdir/src/unit_tests'
Makefile:37: recipe for target 'tests' failed
make: *** [tests] Error 2
make: Leaving directory '/workdir/src'

[jamesmcclain]

I confirmed that GDAL 2.2.3 is the version currently in the build container. I will upgrade it to a 2.4 version this evening.

[stjimreal]

The new way to support HDF in GT 3.x is posted here:

1. add sbt dependency:libraryDependencies ++= Seq("org.locationtech.geotrellis" %% "geotrellis-raster" % "3.6.0", "org.locationtech.geotrellis" %% "geotrellis-gdal" % "3.6.0")
2. Here is my example to connect a HDF dataset:

Supposed this is your data:

Driver: HDF4/Hierarchical Data Format Release 4
Files: MCD12Q1.A2019001.h25v09.006.2020212132651.hdf
Size is 512, 512
Metadata:
... ...
Subdatasets:
  SUBDATASET_1_NAME=HDF4_EOS:EOS_GRID:"MCD12Q1.A2019001.h25v09.006.2020212132651.hdf":MCD12Q1:LC_Type1
  SUBDATASET_1_DESC=[2400x2400] LC_Type1 MCD12Q1 (8-bit unsigned integer)
  SUBDATASET_2_NAME=HDF4_EOS:EOS_GRID:"MCD12Q1.A2019001.h25v09.006.2020212132651.hdf":MCD12Q1:LC_Type2
  SUBDATASET_2_DESC=[2400x2400] LC_Type2 MCD12Q1 (8-bit unsigned integer)
  ... ...

Try APIs with SUBDATASET_1_NAME's fullname and replace filename with path to your file.

GDALDataset("HDF4_EOS:EOS_GRID:\"(your path to file/)MCD12Q1.A2019001.h25v09.006.2020212132651.hdf\":MCD12Q1:LC_Type1")

And this is a temporary solution according to @pomadchin.