Sliceable MetaData Class

[DanRyanIrish]

Description

This PR introduces a sliceable Meta class based on dict, which allows metadata to be associated with data axes and the whole class to be sliced using the normal Python slicing API. Metadata not associated with any axes is not touched by the slicing mechanism.

This PR introduces two types of axis-aware metadata: axis-aligned and grid-aligned. Axis-aligned metadata gives one value for each of its associated axes, and is scalar or a string if only associated with one axis. This type of metadata is only changed if the axis/axes within which it is associated is dropped by slicing. In this case, the values corresponding to the dropped axes are also dropped.

Grid-aligned metadata provides a value for each data array element in its associated axis/axes. This metadata cannot be scalar. Since this metadata mirrors at least part of the data array, it is sliced as if it were data, and its shape is kept consistent with its associated data axes. If all its axes are sliced away, the scalar value at the row/column location at which it was sliced is kept, and its axis-awareness is dropped, i.e. it becomes a piece of normal non-axis-aware scalar metadata.

API Summary

>>> import astropy.units as u
>>> from ndcube.meta import Meta

# Define a Meta object associated with data of shape (3, 4, 5)
>>> header = {"salutation": "hello",
              "name": "world",
              "exposure_time": ([2] * 3) * u.s,  # 1-dimension grid-aligned (pixel-wise) metadata
              "instrument_axes": ["slit step", "slit", "dispersion"],  # axis-aligned (scalar/string-per-axis) metadata
              "pixel_efficiency": np.random.ones(4, 5)  # multi-dimension grid-aligned (pixel-wise) metadata}
>>> comments = {"salutation": "This is polite."}
>>> axes = {"exposure_time": 0,
              "instrument_axes": (0, 1, 2),
              "pixel_efficiency": (1, 2)}
>>> meta = Meta(header, axes=axes, comments=comments, data_shape=(3, 4, 5))

# Use slice item that you would apply to the cube on the Meta object.
>>> sliced_meta = meta[0:2, 2]  # Reduce length of 0th axis and drop 1st axis.
>>> sliced_meta.shape  # Shape has been updated.
(2, 5)
>>> sliced_meta["exposure_time"]  # Exposure time now length of new 0th axis
<Quantity [2., 2.] s >
>>> sliced_meta["instrument_axes"]  # instrument axes has dropped slit entry
["slit step", "dispersion"]
>>> sliced_meta.axes["instrument_axes"]
(0, 1)
>>> sliced_meta["pixel_efficiency"]  # pixel efficiency is now 1-D of length 5
array([1., 1., 1., 1., 1.])
>>> sliced_meta.axes["pixel_efficiency"]
(1,)

# Rebin the Meta object
>>> rebinned_meta = meta.rebin((1, 2, 1))
>>> rebinned_meta["exposure_time"]  # grid-aligned metadata associated with non-rebinned axes remains unchanged...
<Quantity [2., 2., 2.] s >
>>> rebinned_meta.axes["exposure_time"]  # ...and still associated with its axis/axes
(0,)
>>> rebinned_meta["pixel_efficiency"]  # grid-aligned metadata associated with rebinned axes remains...
np.random.ones(4, 5)
>>> rebinned_meta.axes["pixel_efficiency"]  # ...but grid-awareness is removed
None
>>> rebinned_meta["instrument_axes"]  # axis-aligned (scalar/string-per-axis) metadata unchanged...
["slit step", "slit", "dispersion"]
>>> rebinned_meta.axes["instrument_axes"]  # ...and still associated with axes as they haven't been dropped
(0, 1, 2)

To Do

• Implement __delitem__ and remove remove method.
• Implement setter for NDMeta.data_shape which verifies that it's compatible with pre-existing axis- and grid-aligned metadata.
• Improve narrative docs

[pep8speaks]

Hello @DanRyanIrish! Thanks for updating this PR.

• In the file ndcube/meta.py:

Line 71:70: W504 line break after binary operator

• In the file ndcube/tests/helpers.py:

Line 136:38: E721 do not compare types, use 'isinstance()'
Line 118:67: W504 line break after binary operator

Comment last updated at 2021-11-18 14:32:19 UTC

[DanRyanIrish]

@nabobalis Could you give this PR a review? It should be very similar if not the same as what's in your sunraster PR.

[DanRyanIrish]

@Cadair this PR is ready for review. To pre-empt possible concerns regarding DKIST tools, the introduction of this new class does not require that metadata be sliceable. Instead it allows for it if desired.

In order for the Meta object to be sliceable, Meta.shape must be set via the data_shape kwarg during intialization. This must be set to the shape of the data array with which the metadata object is associated. If Meta.shape is None, the object is not sliceable.

Not all entries in the Meta object have to be sliceable. Only those who have a corresponding entry in the Meta.axes dict will be. This gives the axes with which the metadata is associated and is therefore used to work out how to slice it. If Meta.shape is set, but there are no entries in Meta.axes, then slicing has no effect on the Meta object.

Therefore, as far as I understand, this new class should not threaten your assumption in the DKIST tools that .meta never changes.

[DanRyanIrish]

@Cadair, this is ready to merge as far as I can see. Do you still want more time to look at it given your DKIST concerns? As I above, I don't think this causes an issues as the new functionalities are optional and don't have to change previous behaviour.

[DanRyanIrish]

@Cadair, as requested, an explainer on the API/behaviour has been added to the PR description.

[Cadair]

I am in favour of meta rather than header

[Cadair]

This also needs some narrative docs to teach people how and why they should use it.

[DanRyanIrish]

@Cadair, except for narrative docs, I think this PR now addresses all but one minor comment of yours. Unless you'd like to look at it again, I will merge it once I've added narrative docs.

[Cadair]

Looking good, a few more comments. I am happy to give this a final review when you are ready.

[Cadair]

I think you might basically be able to replace all of this (other than the calculations of dropped_axes with this:

new_shape = np.broadcast_to(1, new_meta.data_shape)[item].shape

which is equivalent to np.zeros(shape)[view].shape but without allocating the whole array.

Thanks to @astrofrog for this one.

[DanRyanIrish]

As in:

Suggested change

	new_shape = new_meta.data_shape
	for i, axis_item in enumerate(item):
	if isinstance(axis_item, numbers.Integral):
	dropped_axes[i] = True
	elif isinstance(axis_item, slice):
	start = axis_item.start
	if start is None:
	start = 0
	if start < 0:
	start = data_shape[i] - start
	stop = axis_item.stop
	if stop is None:
	stop = data_shape[i]
	if stop < 0:
	stop = data_shape[i] - stop
	new_shape[i] = stop - start
	else:
	raise TypeError("Unrecognized slice type. "
	"Must be an int, slice and tuple of the same.")
	new_shape = np.broadcast_to(1, new_meta.data_shape)[item].shape
	dropped_axes = np.array([True if isinstance(axis_item, numbers.Integral) else False for axis_item in item], dtype=bool)

[Cadair]

Did we say we were going to nuke this bit?

[DanRyanIrish]

I left a comment about this somewhere after our talk. I realised that without this, NDMeta could not support dropping axes as part of slicing if it contained lists or tuples as axis-aligned metadata. So as ugly as it is, I think it's better to keep it.

[Cadair]

Should this be in the setter for .meta? Do we have a setter for .meta?

[DanRyanIrish]

We don't. We outsource that to NDData.

[DanRyanIrish]

Thanks for the review @Cadair. I've addressed your comments with a couple exceptions where I ask for further clarifications.

WRONG