data_management_framework

This is a repository for the Data Management Framework - a generic, flexible, and extensible framework to build data-driven prognostics applications off of.

Setup steps

1. clone the repository git clone https://github.com/darrahts/data_management_framework.git && cd data_management_framework
2. make setup_database.sh executable and install_timescale.sh executable as well (if desired). chmod +x setup_database.sh && chmod +x install_timescale.sh
3. Run setup_database.sh to setup the database and core tables ./setup_database.sh
4. [optional] create a conda environment conda create --name=datasci python=3.8 pandas jupyterlab scikit-learn psycopg2 seaborn tqdm pyarrow

The Framework

The framework is centered around abstracting the key features of all components, degradation, or any other physical or non-physical "tangible" that generates data into _assets_ and _processes_. This decouples the implementation of specific component or degradation models from the way they are composed at the system level, as well as how they interact with processes and generate data. The framework was initially conceived to facilitate data generation simulation experiments where the primary challenges consisted of

• Requiring significant code changes when experimenting with different component or degradation models
• Difficulty in linking generated data to a set of specific models and operating conditions
• Difficulty in organizing the generated data in a manner that can be accessed from multiple dimensions (i.e. by component, time, operating conditions, etc)

This framework can be used with

• simulations
• real systems
• existing datasets

Moreover, the framework can be used with any number of systems and system types, components and component types, or processes and process types. This framework is a key driver in implementing data-driven maintenance and operation scheduling, which is a challenge that many organizations continue to face. The framework is also highly extensible, able to meet a wide variety of specific use cases, some of which are discussed in the Examples section.

Assets & Asset Types

Assets and asset-types are at the center of the framework. Every physical "tangible" is an asset, and every asset has an asset-type. There can only be one instance of a given asset-type, however there can be any number of assets of an asset-type. As an example with a UAV system, there is only one type allowed for motor_threephase, but there can be multiple three-phase motors modelled, either with the same or different parameters, or any combination.

The Tarot T18 and DJI S1000 both have 8 three phase motors, but the parameters for both are different. The Tarot T18 and DJI S1000 are systems comprised of multiple components, and they themselves are also different assets with the same asset-type (uav_octorotor). This is where flexibility meets simplicity, and this type of organization in conjunction with key constraints suit a wide variety of applications.

Processes & Process Types

Processes and process-types are analogous to assets and asset-types, however they are for non-physical "tangibles" that affect the system, components, and system operation. These can be internal processes like degradation, or external processes like wind.

Similar to assets and asset-types, there can only be one instance of a process-type, but multiple instances of that process. For example, there is only allowed to be one instance of battery_charge_degradation, but there can be multiple instances of this process, with any number of them having different parameters representing different degradation models for the same degradation type.

Data

All assets and processes generate data, and this data is linked to the assets, processes, and group from which it was generated. The group is not required, but it organizes assets across another dimension (discussed in greater detail in the Examples section.

Every generated data is also associated with a cycle, which takes on different meaning depending on the application. Application-specific data fields must be supplied by the user.

Examples

1. The first example is a generic implementation of a single cyber physical system that is comprised of multiple components. The Entity-Relationship (ER) diagram is shown on the left and depicts the "under the hood" organization of the system and components. This type of diagram is a useful abstraction of the System Composition (SC) diagram, shown on the right. The SC diagram shows us how many systems and components we have and how the individual components relate to the system. In this example, there are two systems, each unique, comprised of several components, also unique. The same component cannot be used on more than one system, but any number of components, component-types, systems, system-types can be modelled.

2. The next example shows the framework implemented with a simulated UAV system for data collection. The UAV is the system and all UAVs are stored in the uav_tb. The UAV is comprised of motors, a battery, and an airframe. Just as above, there can be any number of UAVs, motors, batteries, or airframes, and they can all have different parameters even. Running experiments with different combinations of components only requires changing the asset id of that component on the UAV. Then, any data generated will correctly link to the newly changed component.

Component degradation and wind effects are also modeled and implemented as depicted in the SC diagram below. First, the process types are defined:

• charge_degradation
• capacitance_degradation
• motor_degradation
• wind_gusts

Then the process models are defined and attached to the assets they affect. The degradation models are shown below, and it can be seen that there is a different degradation profile for each motor, however each profile is derived from the same underlying model. This is to correctly capture variations in usage-based degradation.

3. Data organization with the N-CMAPSS dataset
4. Data organization with electronics data

API Usage

References

A Data Management Framework & UAV Simulation Testbed for theStudy of System-level Prognostics Technologies

A Data-Centric Approach to the Study of System-Level Prognostics for Cyber Physical Systems: Application to Safe UAV Operations

Developing Deep Learning Models for System Remaining Useful Life Predictions: Application to Aircraft Engines