Initial commit

Author: Horst Petschenig

LICENSE

@@ -0,0 +1,202 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2024 The Authors of "Rapid learning with phase-change 
+   memory-based in-memory computing through learning-to-learn"
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

README.md

@@ -0,0 +1,43 @@
+# Rapid learning with phase-change memory-based in-memory computing through learning-to-learn
+
+This is the code repository for the paper
+
+Rapid learning with phase-change memory-based in-memory computing through learning-to-learn  
+*Thomas Ortner, Horst Petschenig, Athanasios Vasilopoulos, Roland Renner, Spela Brglez, Thomas Limbacher, Enrique Pinero, Alejandro Linares Barranco, Angeliki Pantazi, Robert Legenstein*  
+[ArXiv Link](https://arxiv.org/abs/)
+
+## Setup
+You need [Tensorflow](https://www.tensorflow.org/) to run this code. We used Python 3.9 and TensorFlow 2.5. See the corresponding [Conda](https://docs.conda.io/en/latest/) `environment.yml` file to install all necessary dependencies: 
+
+    conda env create --file=environment.yml --name RapidLearningInMemoryComputing
+    conda activate RapidLearningInMemoryComputing
+    pip install --no-warn-conflicts -r requirements.txt
+
+## Usage
+
+### Few-shot image classification with PCM-based neuromorphic hardware
+To start training on the few-shot image classification task, run 
+
+    cd few_shot_image_classification
+    python main.py --seed 1234 --batch_size=32 --hidden_channels=56 --noise=False
+
+To load an existing checkpoint, run
+
+    cd few_shot_image_classification
+    python main_omniglot.py --checkpoint checkpoints/pretrained-weights.pickle --seed 42  --batch_size=1 --hidden_channels=56 --noise=False --dataset_seed=128
+
+### Rapid online learning of robot arm trajectories in biologically-inspired neural networks
+
+To start training on the robotic arm online learning task, run 
+
+    cd online_learning_robot
+    python main.py
+
+To load an existing checkpoint, run 
+    
+    cd online_learning_robot
+    python main.py --checkpoint checkpoints/pretrained-weights.pickle
+
+## Acknowledgements
+This work was funded in part by the CHIST-ERA grant CHIST-ERA-18-ACAI-004, by the Austrian Science Fund (FWF) [10.55776/I4670-N], by grant PCI2019-111841-2 funded by MCIN/AEI/ 10.13039/501100011033, by SNSF under the project number 20CH21_186999 / 1 and by the European Union. For the purpose of open access, the author has applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission. This work was supported by NSF EFRI grant #2318152.
+E. P.-F. work was supported by a "Formación de Profesorado Universitario" Scholarship, with reference number FPU19/04597 from the Spanish Ministry of Education, Culture and Sports. Furthermore, we thank the In-Memory Computing team at IBM for their technical support with the PCM-based NMHW as well as the IBM Research AI Hardware Center. Moreover, we thank Joris Gentinetta for his help with the setup for the robotic arm experiments.

environment.yml

@@ -0,0 +1,34 @@
+name: RapidLearningInMemoryComputing
+channels:
+  - conda
+  - anaconda
+  - defaults
+dependencies:
+  - hdf5=1.10.6
+  - python=3.9.5
+  - readline=8.1
+  - scipy=1.7.3
+  - tk=8.6.10
+  - pip:
+    - aim
+    - aim-ui
+    - aimrecords
+    - aimrocks
+    - h5py==3.1.0
+    - ipykernel==6.29.2
+    - keras==2.6.0
+    - matplotlib==3.4.2
+    - matplotlib-inline==0.1.6
+    - pandas==1.3.0
+    - pandocfilters==1.5.1
+    - pip==24.0
+    - python-dateutil==2.8.2
+    - python-json-logger==2.0.7
+    - tensorboard==2.6.0
+    - tensorflow==2.5.0
+    - tensorflow-datasets==4.4.0
+    - tensorflow-estimator==2.5.0
+    - tensorflow-metadata==1.4.0
+    - tensorflow-probability==0.13.0
+    - tqdm==4.62.3
+    - werkzeug==2.0.1

few_shot_image_classification/checkpoint_loader.py

@@ -0,0 +1,30 @@
+import pickle as pkl
+from maml_trainee_omniglot import MAMLTraineeOmniglotConv
+import numpy as np
+
+def load_checkpoint_conv(path, trainee: MAMLTraineeOmniglotConv):
+    with open(path, 'rb') as f:
+        weights = pkl.load(f)
+
+    trainee.kernel_1.load(weights[0])
+    trainee.kernel_2.load(weights[1])
+    trainee.kernel_3.load(weights[2])
+    trainee.kernel_4.load(weights[3])
+    trainee.w_readout.load(weights[4])
+
+    for i, w in enumerate(weights):
+        print(f'Weight {i}', np.abs(w).sum())
+
+
+
+def load_checkpoint_conv_only(path, trainee: MAMLTraineeOmniglotConv):
+    with open(path, 'rb') as f:
+        weights = pkl.load(f)
+
+    trainee.kernel_1.load(weights[0])
+    trainee.kernel_2.load(weights[1])
+    trainee.kernel_3.load(weights[2])
+    trainee.kernel_4.load(weights[3])
+
+    # The readout weights are not loaded!
+    # trainee.w_readout.load(weights[4])

few_shot_image_classification/checkpoints/pretrained-weights.pickle

@@ -0,0 +1,81 @@
+import numpy as np
+import tensorflow as tf
+
+from einsums import tf_bidk_bkf_bdf_noise_factorout
+
+
+@tf.function
+def tf_bikf_bdk_bdf(image_shaped, kernel_flat):
+    return tf.einsum("bkf, bidk->bidf", kernel_flat, image_shaped)
+
+
+class ConvLayer(tf.keras.layers.Layer):
+    """2D convolution layer for MAML and PCM devices.
+
+    2 batch sizes: outer and inner on input, outer on kernel
+    Perform pcm read noise per dot product.
+    tf_bidk_bkf_bdf_noise_factorout factors out the noise part in samples the noise afterwards
+    """
+    def __init__(self, image_h, image_w, in_channels, kernel_dims, out_channels, padding=1, stride=1, noise=True,
+                 **kwargs):
+        super(ConvLayer, self).__init__(**kwargs)
+
+        self.image_h = image_h
+        self.image_w = image_w
+        self.channels = in_channels
+        self.kernel_dims = kernel_dims
+        self.out_channels = out_channels
+        self.padding = padding
+        self.stride = stride
+        self.noise = noise
+
+        self.paddings = tf.constant([[0, 0], [0, 0], [padding, padding], [padding, padding], [0, 0]])
+
+        self.out_height = int((image_h + 2 * padding - kernel_dims) / stride + 1)
+        self.out_width = int((image_w + 2 * padding - kernel_dims) / stride + 1)
+
+        if self.noise:
+            self.mult_function = tf_bidk_bkf_bdf_noise_factorout
+        else:
+            self.mult_function = tf_bikf_bdk_bdf
+
+        # Get the indices of the convolutions
+        all_in = []
+        for center_index_h in range(padding, self.image_h - 1 + padding * 2, self.stride):
+            for center_index_w in range(padding, self.image_w - 1 + padding * 2, self.stride):
+                indices_kernel = []
+                for k_h in range(self.kernel_dims):
+                    for k_w in range(self.kernel_dims):
+                        for k_c in range(self.channels):
+                            indices_kernel.append((center_index_h + k_h - 1, center_index_w + k_w - 1, k_c))
+                all_in.append(indices_kernel)
+
+        self.all_in = tf.constant(np.asarray(all_in))
+
+    def call(self, x, kernel):
+        # Pad image with zeros
+        x_pad = tf.pad(x, self.paddings, "CONSTANT")
+
+        batch_dim = x.shape[0] * x.shape[1]
+
+        # Reshape indices in list
+        indices = tf.reshape(self.all_in, (-1, 3))  # 3 because conv 2d  h,w,c
+
+        batch_inner_outer = x_pad.shape[:2]
+
+        # Extract features in the order they get used by applying the convolution
+        image_as = tf.gather_nd(tf.reshape(x_pad, (-1, * x_pad.shape[2:])),
+                                tf.tile(indices[tf.newaxis, ...], [batch_dim, 1, 1]), batch_dims=1)
+
+        # Reshape image to (batch, flat_features) for dot product, width*height (depends on padding and how many
+        # times the dot product is performed)
+        image_ready_for_conv = tf.reshape(image_as, (*batch_inner_outer, self.all_in.shape[0], self.all_in.shape[1]))
+
+        # Collapse kernel into (height*width*in_channels, out_channels)
+        kernel_reshape = tf.reshape(kernel, (
+            kernel.shape[0], self.kernel_dims * self.kernel_dims * self.channels, self.out_channels))
+
+        conv_output_factor_out = self.mult_function(image_ready_for_conv, kernel_reshape)
+
+        return tf.reshape(conv_output_factor_out,
+                          (*batch_inner_outer, self.out_height, self.out_width, self.out_channels))