Skip to content

Commit

Permalink
Fixes non ISTP compliant files axis merging
Browse files Browse the repository at this point in the history 
Some files uses DEPEND_TIME instead of DEPEND_0 to point to time axis.
This was supported by speasy for data variables but not for support variables.

Signed-off-by: Alexis Jeandet <[email protected]>
  • Loading branch information
@jeandet
jeandet committed Nov 5, 2023
1 parent 6730069 commit 37cfaee
Show file tree
Hide file tree
Showing 3 changed files with 18 additions and 7 deletions.
14 changes: 9 additions & 5 deletions speasy/core/cdf/__init__.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,13 +22,17 @@ def _fix_attributes_types(attributes: dict):
return cleaned


def _make_axis(axis, time_axis_name):
def _is_time_dependent(axis, time_axis_name):
if axis.attributes.get('DEPEND_TIME', '') == time_axis_name:
return True
if axis.attributes.get('DEPEND_0', '') == time_axis_name:
is_time_dependent = True
else:
is_time_dependent = False
return True
return False


def _make_axis(axis, time_axis_name):
return VariableAxis(values=axis.values.copy(), meta=_fix_attributes_types(axis.attributes), name=axis.name,
is_time_dependent=is_time_dependent)
is_time_dependent=_is_time_dependent(axis, time_axis_name))


def _build_labels(variable: pyistp.loader.DataVariable):
Expand Down
5 changes: 3 additions & 2 deletions speasy/webservices/generic_archive/__init__.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,9 @@

from speasy.config import SPEASY_CONFIG_DIR
from speasy.config import archive as cfg
from speasy.core import AnyDateTimeType
from speasy.core import AnyDateTimeType, AllowedKwargs
from speasy.core.cdf.inventory_extractor import make_dataset_index
from speasy.core.dataprovider import DataProvider
from speasy.core.dataprovider import DataProvider, GET_DATA_ALLOWED_KWARGS
from speasy.core.direct_archive_downloader import get_product
from speasy.core.inventory.indexes import SpeasyIndex, ParameterIndex
from speasy.products.variable import SpeasyVariable
Expand Down Expand Up @@ -82,6 +82,7 @@ def _parameter_index(self, product: str or ParameterIndex) -> ParameterIndex:
else:
raise ValueError(f"Got unexpected type {type(product)}, expecting str or ParameterIndex")

@AllowedKwargs(GET_DATA_ALLOWED_KWARGS)
def get_data(self, product: str or ParameterIndex, start_time: AnyDateTimeType, stop_time: AnyDateTimeType,
**kwargs) -> Optional[SpeasyVariable]:
var = self._get_data(product=self._parameter_index(product), start_time=start_time, stop_time=stop_time)
Expand Down
6 changes: 6 additions & 0 deletions tests/test_direct_archive_downloader.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,12 @@ def test_get_data(self, product, start, stop):
v = spz.get_data(product, start, stop)
self.assertIsNotNone(v)

def test_axes_merging_across_files(self):
v = spz.get_data(spz.inventories.tree.archive.cdpp.THEMIS.THA.L2.tha_esa.tha_peif_en_eflux, "2018-01-05",
"2018-01-07")
self.assertIsNotNone(v)
self.assertEqual(len(v), len(v.axes[1]))


if __name__ == '__main__':
unittest.main()

3 comments on commit 37cfaee

@RichardHitier
Copy link

@RichardHitier RichardHitier commented on 37cfaee Nov 9, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, I'm running this particular version 1.2.1, but I still get the following message when import speasy without any local cache (~/.cache/speasy/Cache):

Python 3.11.6 (main, Oct  8 2023, 05:06:43) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import speasy as spz
Non compliant ISTP file: Epoch was marked as data variable but it has 0 support variable
Non compliant ISTP file: No data variable found, this is suspicious
Non compliant ISTP file: No data variable found, this is suspicious
Non compliant ISTP file: No data variable found, this is suspicious
Non compliant ISTP file: No data variable found, this is suspicious
Non compliant ISTP file: No data variable found, this is suspicious

A second import, or an existing ~/.cache/speasy/Cache/cache.db wont raise the message.

I'm not sure your commit was supposed to fix it, but in case, I post my request here.
Let me know if you want a new issue instead.

thx for you help.

@jeandet
Copy link
Member Author

@jeandet jeandet commented on 37cfaee Nov 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@RichardHitier these messages are expected if you disabled the speasy proxy usage or this could also be related to new archive module. In both cases this is due to non ISTP compliant files parsed during inventory creation/update.
In both cases, it should only happen when there is no local cache.
You can skip archive module inventory update by disabling it with spz.config.core.disabled_providers.set("archive") or spz.config.core.disabled_providers.set("archive,cda") for example if you also want to disable CDAWeb module.

@RichardHitier
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you !

Please sign in to comment.