Merge branch 'main' of https://github.com/sandialabs/pyscan into add-json-converter

rsbrost · rsbrost · commit 648ae612f61c · 2024-07-09T09:41:54.000-06:00
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,34 @@
+## [0.5.4](https://github.com/sandialabs/pyscan/compare/v0.5.3...v0.5.4) (2024-07-09)
+
+
+### Bug Fixes
+
+* **core:** added a json converter class in json_encoder.py to pyscan/general. This is now implemented in the save_metadata method of abstract_experiment.py which enables numpy values to be used as data inputs before saving. The converter changes the numpy values to standard python values and no longer throws the same type error as before. ([c2c2b8f](https://github.com/sandialabs/pyscan/commit/c2c2b8ffb3f8d1860f6dc6612b2c83eddc21a28f))
+* **core:** added working json converter for runinfo and devices metadata now implemented in abstract experiment's save_metadata method. ([11f43a3](https://github.com/sandialabs/pyscan/commit/11f43a3c247dd4294d079ffd20b360572ffc3ed8))
+* **core:** corrected the issue plesiopterys identified with saving numpy data types and consolidated the json converter added in the last commit to the pre existing recursive_to_dict function which can now handle numpy data types for saving. ([fb66ad8](https://github.com/sandialabs/pyscan/commit/fb66ad8727acfd33b6f59feef6b7a68d023d052f))
+* **core:** fixed issue causing multiple experiments run with the same runinfo to fail. These sequential experiments would improperly double runinfo.measured by appending the same values to it more than once. Then when data was tried to be preallocated in abstract expt it would fail. ([cb093a2](https://github.com/sandialabs/pyscan/commit/cb093a25c5e86edb62b2a22ed7eab91fa73a1956))
+* **core:** Fixed the issue with saving if experiments are run 'too fast'. Will now handle ultra fast experiments (up to a micro second or more) which was tested with demo nb 1 using a dt of 0.000000001 and consecutive experiments. ([bba1bca](https://github.com/sandialabs/pyscan/commit/bba1bcac7561a8f9e03083d3b12951223ccba4d4))
+* **core:** updated experiment.py to reinitialize runinfo.measured as an empty list before populating at the beginning of every experiment run. This was causing issues with running subsequent experiments with the same runinfo, and is necessary to prevent runinfo.measured from being reused if runinfo.measure_function changed between runs. ([bbc1be8](https://github.com/sandialabs/pyscan/commit/bbc1be8958081385c4cfe3d23f5ab6eae83b8c82))
+
+
+### Reverts
+
+* Revert "fix(core): added a json converter class in json_encoder.py to pyscan/general. This is now implemented in the save_metadata method of abstract_experiment.py which enables numpy values to be used as data inputs before saving. The converter changes the numpy values to standard python values and no longer throws the same type error as before." ([30dbdd5](https://github.com/sandialabs/pyscan/commit/30dbdd5c25186ee3278894588b9a9121a354d7eb))
+* Revert "fix(core): corrected the issue plesiopterys identified with saving numpy data types and consolidated the json converter added in the last commit to the pre existing recursive_to_dict function which can now handle numpy data types for saving." ([30c601d](https://github.com/sandialabs/pyscan/commit/30c601d6b2db2f3a8aa66debedbcbc5b2a1e7f70))
+* Revert "chore(core): removed old import from abstract_experiment." ([9d7a10b](https://github.com/sandialabs/pyscan/commit/9d7a10bc51932f65bc8bcdba3a35af651f681dbd))
+* Revert "chore(core): fix: fixed misuse of np.array n recursive_to_dict that was causing a type error." ([a56704d](https://github.com/sandialabs/pyscan/commit/a56704d650242ca73d9b3b988e50d64806fbb572))
+* Revert "fix(core): added working json converter for runinfo and devices metadata now implemented in abstract experiment's save_metadata method." ([0d122b0](https://github.com/sandialabs/pyscan/commit/0d122b0d20b999797e136728ca792c092a06adeb))
+* Revert "docs(general): added doc string to json_encoder.py CustomJSONEncoder class." ([1224b06](https://github.com/sandialabs/pyscan/commit/1224b06b573e94347e983ba7fa4156d6b27d97c7))
+* Revert "chore(core): removed no longer used recursive_to_dict import from abstract expt." ([7d5b6e9](https://github.com/sandialabs/pyscan/commit/7d5b6e91487a310b8ef15a8000b9bcabb38cdbae))
+* Revert "refactor(general): replaced recursive_to_item_attribute with json decoder item_attribute_object_hook." ([6dcfdc1](https://github.com/sandialabs/pyscan/commit/6dcfdc1b01811f81fc5e969addce0fb479595521))
+* Revert "chore(general): updated the general __init__.py to account for the file changes from previous commits and import successfully with a new order since itemattribute is used in other pyscan/general modules now." ([27ae91c](https://github.com/sandialabs/pyscan/commit/27ae91c7a1495e05e7887897972ae94ddef886c9))
+* Revert "chore(general): fixed improper spelling in the json_decoder." ([205c33a](https://github.com/sandialabs/pyscan/commit/205c33ac3071adfd3c942c7a65258054baf4d67a))
+* Revert "chore(general): fixed improper spelling in the json_decoder." ([6e5b788](https://github.com/sandialabs/pyscan/commit/6e5b788fe880b43df8fe30ec9e637e6af8b6f3e0))
+* Revert "chore(general): fixed error breaking pytests with decoder referencing np.float_ which was removed in numpy 2.0." ([0886669](https://github.com/sandialabs/pyscan/commit/0886669c34b46721029ed4c810ea9fc2a04c8342))
+* Revert "chore(general): fixed error breaking pytests with decoder referencing np.float_ which was removed in numpy 2.0." ([a32e84d](https://github.com/sandialabs/pyscan/commit/a32e84d3a5c5524d53e7b79aa7d48085e160ada2))
+
+
+
 ## [0.5.3](https://github.com/sandialabs/pyscan/compare/v0.5.2...v0.5.3) (2024-06-18)
 
 
diff --git a/VERSION.json b/VERSION.json
@@ -1,3 +1,3 @@
 {
-  "version": "0.5.3"
+  "version": "0.5.4"
 }
diff --git a/pyscan/general/__init__.py b/pyscan/general/__init__.py
@@ -1,6 +1,6 @@
 # objects
 from .item_attribute import ItemAttribute
-from .json_encoder import PyscanJSONEncoder
+from .pyscan_json_encoder import PyscanJSONEncoder
 
 # methods
 from .d_range import drange
@@ -12,4 +12,4 @@
 from .set_difference import set_difference
 from .stack_or_append import stack_or_append
 from .get_pyscan_version import get_pyscan_version
-from .json_decoder import PyscanJSONDecoder
+from .pyscan_json_decoder import PyscanJSONDecoder
diff --git a/pyscan/general/pyscan_json_decoder.py b/pyscan/general/pyscan_json_decoder.py
diff --git a/pyscan/general/pyscan_json_encoder.py b/pyscan/general/pyscan_json_encoder.py
diff --git a/pyscan/general/recursive_to_dict.py b/pyscan/general/recursive_to_dict.py
@@ -0,0 +1,44 @@
+# -*- coding: utf-8 -*-
+import numpy as np
+
+
+def recursive_to_dict(obj_dict):
+    '''
+    Recursive function that converts metadata in runinfo and devices into
+    serializable object for saving
+
+    Parameters
+    ----------
+    obj_dict : :class:`~pyscan.general.itemattribute.ItemAttribute`
+        runinfo or devices object
+    '''
+    new_dict = {}
+
+    for key, value in obj_dict.items():
+        # print(key, value)
+        # is method/function
+        if key in ['logger', 'expt_thread', 'data_path',
+                   'instrument', 'module_id_string', 'spec']:
+            pass
+        elif hasattr(value, '__call__'):
+            new_dict[key] = value.__name__
+        elif isinstance(value, str):
+            new_dict[key] = value
+        # is a dict
+        elif isinstance(value, dict):
+            new_dict[key] = recursive_to_dict(value)
+        # if it is an np array
+        elif isinstance(value, np.ndarray):
+            new_dict[key] = value.tolist()
+        # is an iterator
+        elif hasattr(value, "__iter__"):
+            new_dict[key] = list(value)
+        # is an object
+        elif hasattr(value, "__dict__"):
+            new_dict[key] = recursive_to_dict(value.__dict__)
+        # anything else
+        else:
+            new_dict[key] = value
+            # maybe pass this, but test first
+
+    return new_dict
diff --git a/pyscan/general/recursive_to_item_attribute.py b/pyscan/general/recursive_to_item_attribute.py
@@ -0,0 +1,28 @@
+# -*- coding: utf-8 -*-
+from .item_attribute import ItemAttribute
+
+
+def recursive_to_itemattribute(data):
+    '''
+    Recursive function that converts serialized metadata into an
+    ItemAttribute object
+
+    Parameters
+    ----------
+    data : dict
+        dictionary
+
+    Returns
+    -------
+    :class:`.ItemAttribute`
+    '''
+
+    new_data = ItemAttribute()
+
+    for key, value in data.items():
+        if isinstance(value, dict):
+            new_data[key] = recursive_to_itemattribute(value)
+        else:
+            new_data[key] = value
+
+    return new_data
diff --git a/pyscan/measurement/abstract_experiment.py b/pyscan/measurement/abstract_experiment.py
@@ -8,7 +8,7 @@
 from pyscan.general import (ItemAttribute,
                             is_list_type)
 from pyscan.measurement.scans import PropertyScan, RepeatScan
-from pyscan.general.json_encoder import PyscanJSONEncoder
+from pyscan.general.pyscan_json_encoder import PyscanJSONEncoder
 
 
 class AbstractExperiment(ItemAttribute):
@@ -160,7 +160,16 @@ def check_runinfo(self):
         if num_repeat_scans > 1:
             assert False, "More than one repeat scan detected. This is not allowed."
 
-        self.runinfo.long_name = strftime("%Y%m%dT%H%M%S")
+        base_name = strftime("%Y%m%dT%H%M%S")
+        save_path = self.runinfo.data_path / '{}.hdf5'.format(base_name)
+        count = 0
+
+        while save_path.exists():
+            count += 1
+            save_path = self.runinfo.data_path / f'{base_name}-{count}.hdf5'
+
+        self.runinfo.long_name = save_path.stem
+
         self.runinfo.short_name = self.runinfo.long_name[8:]
 
         self.runinfo.check()
diff --git a/pyscan/measurement/experiment.py b/pyscan/measurement/experiment.py
@@ -82,6 +82,7 @@ def generic_experiment(self):
                             self.runinfo.t3[indicies] = (datetime.now()).timestamp()
 
                         if np.all(np.array(self.runinfo.indicies) == 0):
+                            self.runinfo.measured = []
                             for key, value in data.items():
                                 self.runinfo.measured.append(key)
                             self.preallocate(data)
@@ -172,6 +173,7 @@ def average_experiment(self):
 
                         # if on the first row of data, log the data names in self.runinfo.measured
                         if np.all(np.array(self.runinfo.indicies) == 0):
+                            self.runinfo.measured = []
                             for key, value in data.items():
                                 self.runinfo.measured.append(key)
                             self.preallocate(data)
diff --git a/pyscan/measurement/load_experiment.py b/pyscan/measurement/load_experiment.py
@@ -3,8 +3,8 @@
 import pickle
 import json
 from pathlib import Path
-from pyscan.general import ItemAttribute
-from pyscan.general.json_decoder import PyscanJSONDecoder
+from pyscan.general.pyscan_json_decoder import PyscanJSONDecoder
+from pyscan.general.item_attribute import ItemAttribute
 
 
 def load_experiment(file_name):
@@ -54,6 +54,8 @@ def load_experiment(file_name):
 
     elif data_version == 0.2:
         expt = ItemAttribute()
+        expt.runinfo = ItemAttribute()
+        expt.devices = ItemAttribute()
 
         f = h5py.File('{}'.format(file_name), 'r')
         expt.runinfo = json.loads(f.attrs['runinfo'], cls=PyscanJSONDecoder)
diff --git a/test/legacy/measurement/test_legacy_abstract_experiment.py b/test/legacy/measurement/test_legacy_abstract_experiment.py
@@ -12,6 +12,7 @@
 from io import StringIO
 import sys
 import shutil
+import re
 
 
 # for testing default trigger function with empty function
@@ -104,7 +105,8 @@ def test_ms_diff_inputs(data_dir=None, measure_function=measure_point, allocate=
 
         assert hasattr(ms.runinfo, 'long_name'), "Meta Sweep runinfo long name is not initialized by check_runinfo()"
         assert isinstance(ms.runinfo.long_name, str), "Meta Sweep runinfo long name is not initialized as a string"
-        assert len(ms.runinfo.long_name) == 15, "Meta Sweep runinfo long name is not 15 characters"
+        # check that the long name is formatted with values for YYYYMMDDTHHMMSS, and optionally a - followed by digits.
+        assert re.match(r'^\d{8}T\d{6}(-\d+)?$', ms.runinfo.long_name), "runinfo long_name is not properly formatted"
 
         assert hasattr(ms.runinfo, 'short_name'), "Meta Sweep runinfo long name is not initialized by check_runinfo()"
         assert isinstance(ms.runinfo.short_name, str), "Meta Sweep runinfo short name is not initialized as a string"
diff --git a/test/legacy/measurement/test_legacy_experiment.py b/test/legacy/measurement/test_legacy_experiment.py
@@ -8,6 +8,8 @@
 import shutil
 import numpy as np
 import pytest
+import re
+import os
 
 
 ##################### FUNCTIONS USED BY TEST CASES #####################
@@ -1098,3 +1100,61 @@ def check_load_expt(temp):
     check_load_expt(temp)
 
     shutil.rmtree('./backup')
+
+
+def test_fast_experiments():
+    devices = ps.ItemAttribute()
+
+    devices.v1 = ps.TestVoltage()  # Device 1
+    devices.v2 = ps.TestVoltage()  # Device 2
+    devices.v3 = ps.TestVoltage()  # Device 3
+
+    def get_voltage_data(ms):
+        """
+        Reads the voltage from v1, v2, and v3 devices. Also adds a calculated value vsum.
+        """
+
+        devices = ms.devices
+
+        d = ps.ItemAttribute()
+
+        d.v1_readout = devices.v1.voltage
+        d.v2_readout = devices.v2.voltage
+        d.v3_readout = devices.v3.voltage
+
+        d.vsum = d.v1_readout + d.v2_readout + d.v3_readout
+
+        return d
+
+    # Create RunInfo instance and set scan0 to PropertyScan
+    runinfo = ps.RunInfo()
+    runinfo.scan0 = ps.RepeatScan(1, dt=0.0000001)
+
+    # Set RunInfo measure_function (remember, it takes a Experiment object as a parameter and
+    # returns an ItemAttribute containing data).
+    runinfo.measure_function = get_voltage_data
+
+    # Create a Experiment class with the RunInfo and Devices just created
+    ms = ps.Sweep(runinfo, devices, time=True)
+
+    long_names = []
+
+    while len(long_names) < 3:
+        ms.run()
+        if len(long_names) == 0:
+            long_names.append(ms.runinfo.long_name)
+        elif (ms.runinfo.long_name[:15] == long_names[0][:15]):
+            long_names.append(ms.runinfo.long_name)
+        else:
+            long_names = [ms.runinfo.long_name]
+
+    err_str = f"First long name '{long_names[0]}' does not match expected date/time format."
+    assert re.match(r'^\d{8}T\d{6}$', long_names[0]), err_str
+    err_str = f"-1 long name '{long_names[1]}' does not match expected increment or format."
+    assert long_names[1] == long_names[0] + '-1', err_str
+    err_str = f"-2 long name '{long_names[1]}' does not match expected increment or format."
+    assert long_names[2] == long_names[0] + '-2', err_str
+
+    for name in long_names:
+        save_path = ms.runinfo.data_path / '{}.hdf5'.format(name)
+        assert os.path.exists(save_path), f"Expected file at path'{save_path}' was not found."
diff --git a/test/measurement/test_abstract_experiment.py b/test/measurement/test_abstract_experiment.py
@@ -12,6 +12,7 @@
 from io import StringIO
 import sys
 import shutil
+import re
 
 
 # for testing default trigger function with empty function
@@ -104,7 +105,8 @@ def test_ms_diff_inputs(data_dir=None, measure_function=measure_point, allocate=
 
         assert hasattr(ms.runinfo, 'long_name'), "Meta Sweep runinfo long name is not initialized by check_runinfo()"
         assert isinstance(ms.runinfo.long_name, str), "Meta Sweep runinfo long name is not initialized as a string"
-        assert len(ms.runinfo.long_name) == 15, "Meta Sweep runinfo long name is not 15 characters"
+        # check that the long name is formatted with values for YYYYMMDDTHHMMSS, and optionally a - followed by digits.
+        assert re.match(r'^\d{8}T\d{6}(-\d+)?$', ms.runinfo.long_name), "runinfo long_name is not properly formatted"
 
         assert hasattr(ms.runinfo, 'short_name'), "Meta Sweep runinfo long name is not initialized by check_runinfo()"
         assert isinstance(ms.runinfo.short_name, str), "Meta Sweep runinfo short name is not initialized as a string"
diff --git a/test/measurement/test_experiment.py b/test/measurement/test_experiment.py
@@ -8,6 +8,8 @@
 import shutil
 import numpy as np
 import pytest
+import re
+import os
 
 
 ##################### FUNCTIONS USED BY TEST CASES #####################
@@ -1093,3 +1095,61 @@ def check_load_expt(temp):
     check_load_expt(temp)
 
     shutil.rmtree('./backup')
+
+
+def test_fast_experiments():
+    devices = ps.ItemAttribute()
+
+    devices.v1 = ps.TestVoltage()  # Device 1
+    devices.v2 = ps.TestVoltage()  # Device 2
+    devices.v3 = ps.TestVoltage()  # Device 3
+
+    def get_voltage_data(expt):
+        """
+        Reads the voltage from v1, v2, and v3 devices. Also adds a calculated value vsum.
+        """
+
+        devices = expt.devices
+
+        d = ps.ItemAttribute()
+
+        d.v1_readout = devices.v1.voltage
+        d.v2_readout = devices.v2.voltage
+        d.v3_readout = devices.v3.voltage
+
+        d.vsum = d.v1_readout + d.v2_readout + d.v3_readout
+
+        return d
+
+    # Create RunInfo instance and set scan0 to PropertyScan
+    runinfo = ps.RunInfo()
+    runinfo.scan0 = ps.RepeatScan(1, dt=0.0000001)
+
+    # Set RunInfo measure_function (remember, it takes a Experiment object as a parameter and
+    # returns an ItemAttribute containing data).
+    runinfo.measure_function = get_voltage_data
+
+    # Create a Experiment class with the RunInfo and Devices just created
+    expt = ps.Experiment(runinfo, devices, time=True)
+
+    long_names = []
+
+    while len(long_names) < 3:
+        expt.run()
+        if len(long_names) == 0:
+            long_names.append(expt.runinfo.long_name)
+        elif (expt.runinfo.long_name[:15] == long_names[0][:15]):
+            long_names.append(expt.runinfo.long_name)
+        else:
+            long_names = [expt.runinfo.long_name]
+
+    err_str = f"First long name '{long_names[0]}' does not match expected date/time format."
+    assert re.match(r'^\d{8}T\d{6}$', long_names[0]), err_str
+    err_str = f"-1 long name '{long_names[1]}' does not match expected increment or format."
+    assert long_names[1] == long_names[0] + '-1', err_str
+    err_str = f"-2 long name '{long_names[1]}' does not match expected increment or format."
+    assert long_names[2] == long_names[0] + '-2', err_str
+
+    for name in long_names:
+        save_path = expt.runinfo.data_path / '{}.hdf5'.format(name)
+        assert os.path.exists(save_path), f"Expected file at path'{save_path}' was not found."

Original file line number	Diff line number	Diff line change
`@@ -1,3 +1,3 @@`
`1`	`1`	`{`
`2`		`- "version": "0.5.3"`
	`2`	`+ "version": "0.5.4"`
`3`	`3`	`}`