adding 2D data to hdf5 container

[martinschorb]

Hi,

it seems that with the current 2D implementation something goes wrong when I add multiple 2D datasets into one hdf5. In this case this is tiles of a mosaic.

It looks like it is adding the different images as slices in a single 3D stack container instead of creating individual data cells...

This worked correctly with the previous version.

[martinschorb]

wait....

could be a mistake on my side. Sorry...

But I found another issue with 2D data (now individual image)

  File "c:\users\schorb\pybdv\pybdv\converter.py", line 433, in make_bdv
    factors = [[1, 1, 1]] + downscale_factors

TypeError: can only concatenate list (not "tuple") to list

This used to work before...

[martinschorb]

I provide it as tuple (I think I saw that somewhere in your code...):

downscale_factors = ([1,2,2],[1,2,2],[1,2,2],[1,4,4])

If I provide is as list/array:

downscale_factors = [[1,2,2],[1,2,2],[1,2,2],[1,4,4]]

I get:

  File "c:\users\schorb\pybdv\pybdv\converter.py", line 458, in make_bdv
    enforce_consistency=enforce_consistency)

  File "c:\users\schorb\pybdv\pybdv\metadata.py", line 218, in write_xml_metadata
    _update_attributes(viewsets, attributes, overwrite)

  File "c:\users\schorb\pybdv\pybdv\metadata.py", line 159, in _update_attributes
    for name, val in this_values:

ValueError: too many values to unpack (expected 2)

[martinschorb]

OK, I think I fixed my tile issue from above. But now...

  File "c:\users\schorb\pybdv\pybdv\converter.py", line 409, in make_bdv
    attributes_ = validate_attributes(xml_path, attributes, setup_id, enforce_consistency)

  File "c:\users\schorb\pybdv\pybdv\metadata.py", line 503, in validate_attributes
    attrs_out = _validate_existing_attributes(setups, setup_id, attributes, enforce_consistency)

  File "c:\users\schorb\pybdv\pybdv\metadata.py", line 466, in _validate_existing_attributes
    raise ValueError("Attributes contains unexpected names")

ValueError: Attributes contains unexpected names




thisview['attributes']
Out[18]: 
{'tile': {'id': 0},
 'displaysettings': {'id': 0,
  'color': '255 255 255 255',
  'isset': 'true',
  'Projection_Mode': 'Average',
  'min': '408',
  'max': '10927'}}

Which one creates the problem?

[constantinpape]

This happens if you write a new set-up with an attribute name that has not been added to the xml before.
E.g. if the first call to make_bdv contains attributes={'channel': {...}, tile: {...}, displaysettings: {...}} then all subsequent calls must also contain attributes with channel, tile and displaysettings (and nothing else!).

Otherwise, set-ups would have different attributes, which might leave to undefined behaviour.

[martinschorb]

Is there any way that allows me to create a new XML with pybdv without re-writing the h5?

Assume we have a giant data set like the platy. Now we want to generate a new metadata XML that contains the displaysettingsand some other useful additions. Currently pybdv would not allow this. Would this need another overwrite mode for make_bdv?

[martinschorb]

Something like "force-metadata" or "renew-metadata".... It will completely ignore what already exists and just write the metadata for the given ViewSetup. Obviously it cannot check that the ViewSetups written to the XML will correspond to the data container, but this would be up to the user. To be on the safe side, it could maybe even create a backup of the original XML file before creating the new one.

[martinschorb]

Or it would do that if there is no original xml file (manually removed/renamed)...

[martinschorb]

Back to the original problem:

Now I have a proper h5 and xml and now when I call make_bdv with metadata, I again get:

File "c:\users\schorb\pybdv\pybdv\converter.py", line 433, in make_bdv
    factors = [[1, 1, 1]] + downscale_factors

TypeError: can only concatenate list (not "tuple") to list

[constantinpape]

Something like "force-metadata" or "renew-metadata".... It will completely ignore what already exists and just write the metadata for the given ViewSetup. Obviously it cannot check that the ViewSetups written to the XML will correspond to the data container, but this would be up to the user. To be on the safe side, it could maybe even create a backup of the original XML file before creating the new one.

The problem is that this can bring attributes for different set-ups out of sync.
I don't think it's a good idea to include this in make_bdv. This should either be done by the user or we introduce a different function that only operates on metadata for this in pybdv.

[constantinpape]

File "c:\users\schorb\pybdv\pybdv\converter.py", line 433, in make_bdv
factors = [[1, 1, 1]] + downscale_factors

TypeError: can only concatenate list (not "tuple") to list

I think the problem is that you are passing the downscale_factors as tuple, but right now it only works with a list

I will change the line to

factors = [[1, 1, 1]] + list(downscale_factors)

to allow for both again.

[martinschorb]

OK,
so now whenenver it tries to overwrite an existing XML, even with the very same metadata, Irun into:

 File "c:\users\schorb\pybdv\pybdv\converter.py", line 458, in make_bdv
    enforce_consistency=enforce_consistency)

  File "c:\users\schorb\pybdv\pybdv\metadata.py", line 218, in write_xml_metadata
    _update_attributes(viewsets, attributes, overwrite)

  File "c:\users\schorb\pybdv\pybdv\metadata.py", line 159, in _update_attributes
    for name, val in this_values:

ValueError: too many values to unpack (expected 2)

[constantinpape]

OK,
so now whenenver it tries to overwrite an existing XML, even with the very same metadata, Irun into:

I see, yeah that's an issue in the code (it muse be this_values.items()); looks like there is a code path that is not covered by my tests yet ....

I will fix it and let you know once the fix is there.

[constantinpape]

Ok, both issues should be fixed now.
I haven't checked why the missing .items() wasn't caught by tests yet (it should have ...)
So there might be some other issues coming up.