Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JSON/TOML backend: introduce abbreviated IO modes #1493

Open
wants to merge 23 commits into
base: dev
Choose a base branch
from

Conversation

franzpoeschel
Copy link
Contributor

@franzpoeschel franzpoeschel commented Aug 4, 2023

Factored out of #1277, based on #1436

Introduces two options:

  1. json.dataset.mode = "template" (default: = "dataset"), write just the dataset extent and not the actual content
  2. json.attribute.mode = "short" (default: = "long"), write an attribute as {"software": "openPMD-api"} instead of {"software": {"value": "openPMD-api", "type": "STRING"}}

TODO:

  • Documentation
  • More testing: openpmd-pipe, erroring IO workflows – fixed, but add test
  • Merge TOML Backend #1436 first
  • In TOML: Enable abbreviated modes by default: Done, but document this
  • Maybe add an option to skip the 'Trying to write data to a template dataset. Will skip.' warning

Diff: https://github.com/franzpoeschel/openPMD-api/compare/parallel-json..topic-json-short-modes

Comment on lines +1258 to +1532
switch (j.type())
{
case nlohmann::json::value_t::null:
throw std::runtime_error(
"[JSON backend] Attribute must not be null: '" +
nameForErrorMessages + "'.");
case nlohmann::json::value_t::object:
throw std::runtime_error(
"[JSON backend] Shorthand-style attribute must not be an "
"object: '" +
nameForErrorMessages + "'.");
case nlohmann::json::value_t::array:
if (j.empty())
{
std::cerr << "Cannot recover datatype of empty vector without "
"explicit type annotation for attribute '"
<< nameForErrorMessages
<< "'. Will continue with VEC_INT datatype."
<< std::endl;
return std::vector<int>{};
}
else
{
auto valueType = j[0].type();
/*
* If the vector is of numeric type, it might happen that the
* first entry is an integer, but a later entry is a float.
* We need to pick the most generic datatype in that case.
*/
if (valueType == nlohmann::json::value_t::number_float ||
valueType == nlohmann::json::value_t::number_unsigned ||
valueType == nlohmann::json::value_t::number_integer)
{
valueType = unifyNumericType(j);
}
switch (valueType)
{
case nlohmann::json::value_t::null:
throw std::runtime_error(
"[JSON backend] Attribute must not be null: '" +
nameForErrorMessages + "'.");
case nlohmann::json::value_t::object:
throw std::runtime_error(
"[JSON backend] Invalid contained datatype (object) "
"inside vector-type attribute: '" +
nameForErrorMessages + "'.");
case nlohmann::json::value_t::array:
throw std::runtime_error(
"[JSON backend] Invalid contained datatype (array) "
"inside vector-type attribute: '" +
nameForErrorMessages + "'.");
case nlohmann::json::value_t::string:
return recoverVectorAttributeFromJson<std::string>(j);
case nlohmann::json::value_t::boolean:
throw std::runtime_error(
"[JSON backend] Attribute must not be vector of bool: "
"'" +
nameForErrorMessages + "'.");
case nlohmann::json::value_t::number_integer:
return recoverVectorAttributeFromJson<
nlohmann::json::number_integer_t>(j);
case nlohmann::json::value_t::number_unsigned:
return recoverVectorAttributeFromJson<
nlohmann::json::number_unsigned_t>(j);
case nlohmann::json::value_t::number_float:
return recoverVectorAttributeFromJson<
nlohmann::json::number_float_t>(j);
case nlohmann::json::value_t::binary:
throw std::runtime_error(
"[JSON backend] Attribute must not have binary type: "
"'" +
nameForErrorMessages + "'.");
case nlohmann::json::value_t::discarded:
throw std::runtime_error(
"Internal JSON parser datatype leaked into JSON "
"value.");
}
throw std::runtime_error("Unreachable!");
}
case nlohmann::json::value_t::string:
return j.get<std::string>();
case nlohmann::json::value_t::boolean:
return j.get<bool>();
case nlohmann::json::value_t::number_integer:
return j.get<nlohmann::json::number_integer_t>();
case nlohmann::json::value_t::number_unsigned:
return j.get<nlohmann::json::number_unsigned_t>();
case nlohmann::json::value_t::number_float:
return j.get<nlohmann::json::number_float_t>();
case nlohmann::json::value_t::binary:
throw std::runtime_error(
"[JSON backend] Attribute must not have binary type: '" +
nameForErrorMessages + "'.");
case nlohmann::json::value_t::discarded:
throw std::runtime_error(
"Internal JSON parser datatype leaked into JSON value.");
}

Check notice

Code scanning / CodeQL

Long switch case Note

Switch has at least one case that is too long:
array (67 lines)
.
test/SerialIOTest.cpp Fixed Show fixed Hide fixed
src/RecordComponent.cpp Fixed Show fixed Hide fixed
Comment on lines +69 to +98
// std::string config = R"(
// {
// "iteration_encoding": "variable_based",
// "toml": {
// "dataset": {"mode": "template"},
// "attribute": {"mode": "short"}
// }
// }
// )";

Check notice

Code scanning / CodeQL

Commented-out code Note

This comment appears to contain commented-out code.
@franzpoeschel
Copy link
Contributor Author

With this, the JSON backend is now sensibly usable from codes such as PIConGPU:

{
  "__openPMD_internal": {
    "attribute_mode": "short",
    "dataset_mode": "template"
  },
  "attributes": {
    "basePath": "/data/%T/",
    "date": "2023-10-11 09:57:53 +0200",
    "iterationEncoding": "fileBased",
    "iterationFormat": "simData_%06T",
    "meshesPath": "fields/",
    "openPMD": "1.1.0",
    "openPMDextension": 0,
    "particlesPath": "particles/",
    "picongpuIOVersionMajor": 2,
    "picongpuIOVersionMinor": 0,
    "software": "PIConGPU",
    "softwareVersion": "0.7.0-dev"
  },
  "data": {
    "100": {
      "attributes": {
        "cell_depth": 4.252342224121094,
        "cell_height": 1.0630855560302734,
        "cell_width": 4.252342224121094,
        "dt": 1,
        "eps0": 169.19711303710938,
        "iteration": 100,
        "mue0": 0.005910266190767288,
        "particleBoundary": [
          "absorbing",
          "absorbing",
          "absorbing",
          "absorbing",
          "absorbing",
          "absorbing"
        ],
        "particleBoundaryParameters": [
          "without field correction",
          "without field correction",
          "without field correction",
          "without field correction",
          "without field correction",
          "without field correction"
        ],
        "sim_slides": 0,
        "time": 100,
        "timeUnitSI": 1.39e-16,
        "unit_bfield": 40903.82224060171,
        "unit_charge": 1.1143237516482563e-15,
        "unit_efield": 12262657411105.05,
        "unit_energy": 5.694183860145225e-10,
        "unit_length": 4.1671151662e-08,
        "unit_mass": 6.335633991170193e-27,
        "unit_speed": 299792458,
        "unit_time": 1.39e-16
      },
      "fields": {
        "B": {
          "attributes": {
            "axisLabels": [
              "z",
              "y",
              "x"
            ],
            "dataOrder": "C",
            "fieldSmoothing": "none",
            "geometry": "cartesian",
            "gridGlobalOffset": [
              0,
              0,
              0
            ],
            "gridSpacing": [
              4.252342224121094,
              1.0630855560302734,
              4.252342224121094
            ],
            "gridUnitSI": 4.1671151662e-08,
            "timeOffset": 0,
            "unitDimension": [
              0,
              1,
              -2,
              -1,
              0,
              0,
              0
            ]
          },
          "x": {
            "attributes": {
              "position": [
                0,
                0.5,
                0.5
              ],
              "unitSI": 40903.82224060171
            },
            "datatype": "FLOAT",
            "extent": [
              32,
              32,
              32
            ]
          },
          "y": {
            "attributes": {
              "position": [
                0.5,
                0,
                0.5
              ],
              "unitSI": 40903.82224060171
            },
            "datatype": "FLOAT",
            "extent": [
              32,
              32,
              32
            ]
          },
          "z": {
            "attributes": {
              "position": [
                0.5,
                0.5,
                0
              ],
              "unitSI": 40903.82224060171
            },
            "datatype": "FLOAT",
            "extent": [
              32,
              32,
              32
            ]
          }
        },
        "E": {
          "attributes": {
            "axisLabels": [
              "z",
              "y",
              "x"
            ],
            "dataOrder": "C",
            "fieldSmoothing": "none",
            "geometry": "cartesian",
            "gridGlobalOffset": [
              0,
              0,
              0
            ],
            "gridSpacing": [
              4.252342224121094,
              1.0630855560302734,
              4.252342224121094
            ],
            "gridUnitSI": 4.1671151662e-08,
            "timeOffset": 0,
            "unitDimension": [
              1,
              1,
              -3,
              -1,
              0,
              0,
              0
            ]
          },
          "x": {
            "attributes": {
              "position": [
                0.5,
                0,
                0
              ],
              "unitSI": 12262657411105.05
            },
            "datatype": "FLOAT",
            "extent": [
              32,
              32,
              32
            ]
          },
          "y": {
            "attributes": {
              "position": [
                0,
                0.5,
                0
              ],
              "unitSI": 12262657411105.05
            },
            "datatype": "FLOAT",
            "extent": [
              32,
              32,
              32
            ]
          },
          "z": {
            "attributes": {
              "position": [
                0,
                0,
                0.5
              ],
              "unitSI": 12262657411105.05
            },
            "datatype": "FLOAT",
            "extent": [
              32,
              32,
              32
            ]
          }
        },
        "attributes": {
          "chargeCorrection": "none",
          "currentSmoothing": "none",
          "fieldBoundary": [
            "open",
            "open",
            "open",
            "open",
            "open",
            "open"
          ],
          "fieldBoundaryParameters": [
            "convolutional PML over 12 cells",
            "convolutional PML over 12 cells",
            "convolutional PML over 12 cells",
            "convolutional PML over 12 cells",
            "convolutional PML over 12 cells",
            "convolutional PML over 12 cells"
          ],
          "fieldSolver": "Yee"
        },
        "e_all_chargeDensity": {
          "attributes": {
            "axisLabels": [
              "z",
              "y",
              "x"
            ],
            "dataOrder": "C",
            "fieldSmoothing": "none",
            "geometry": "cartesian",
            "gridGlobalOffset": [
              0,
              0,
              0
            ],
            "gridSpacing": [
              4.252342224121094,
              1.0630855560302734,
              4.252342224121094
            ],
            "gridUnitSI": 4.1671151662e-08,
            "position": [
              0,
              0,
              0
            ],
            "timeOffset": 0,
            "unitDimension": [
              -3,
              0,
              1,
              1,
              0,
              0,
              0
            ],
            "unitSI": 15399438.226078901
          },
          "datatype": "FLOAT",
          "extent": [
            32,
            32,
            32
          ]
        },
        "e_all_energyDensity": {
          "attributes": {
            "axisLabels": [
              "z",
              "y",
              "x"
            ],
            "dataOrder": "C",
            "fieldSmoothing": "none",
            "geometry": "cartesian",
            "gridGlobalOffset": [
              0,
              0,
              0
            ],
            "gridSpacing": [
              4.252342224121094,
              1.0630855560302734,
              4.252342224121094
            ],
            "gridUnitSI": 4.1671151662e-08,
            "position": [
              0,
              0,
              0
            ],
            "timeOffset": 0,
            "unitDimension": [
              -1,
              1,
              -2,
              0,
              0,
              0,
              0
            ],
            "unitSI": 7869098408118.734
          },
          "datatype": "FLOAT",
          "extent": [
            32,
            32,
            32
          ]
        },
        "picongpu_idProvider": {
          "attributes": {
            "axisLabels": [
              "x"
            ],
            "dataOrder": "C",
            "geometry": "cartesian",
            "gridGlobalOffset": [
              0
            ],
            "gridSpacing": [
              1
            ],
            "gridUnitSI": 1,
            "timeOffset": 0,
            "unitDimension": [
              0,
              0,
              0,
              0,
              0,
              0,
              0
            ]
          },
          "nextId": {
            "attributes": {
              "position": [
                0
              ],
              "unitSI": 1
            },
            "datatype": "ULONG",
            "extent": [
              1,
              1,
              1
            ]
          },
          "startId": {
            "attributes": {
              "maxNumProc": 1,
              "position": [
                0
              ],
              "unitSI": 1
            },
            "datatype": "ULONG",
            "extent": [
              1,
              1,
              1
            ]
          }
        }
      },
      "particles": {
        "e": {
          "attributes": {
            "currentDeposition": "Esirkepov",
            "particleInterpolation": "uniform",
            "particlePush": "Boris",
            "particleShape": 2,
            "particleSmoothing": "none"
          },
          "charge": {
            "attributes": {
              "macroWeighted": 0,
              "shape": [
                0
              ],
              "timeOffset": 0,
              "unitDimension": [
                0,
                0,
                1,
                1,
                0,
                0,
                0
              ],
              "unitSI": 1.1143237516482563e-15,
              "value": -0.0001437801111023873,
              "weightingPower": 1
            }
          },
          "mass": {
            "attributes": {
              "macroWeighted": 0,
              "shape": [
                0
              ],
              "timeOffset": 0,
              "unitDimension": [
                0,
                1,
                0,
                0,
                0,
                0,
                0
              ],
              "unitSI": 6.335633991170193e-27,
              "value": 0.0001437801111023873,
              "weightingPower": 1
            }
          },
          "momentum": {
            "attributes": {
              "macroWeighted": 1,
              "timeOffset": 0,
              "unitDimension": [
                1,
                1,
                -1,
                0,
                0,
                0,
                0
              ],
              "weightingPower": 1
            },
            "x": {
              "attributes": {
                "shape": [
                  0
                ],
                "unitSI": 1.8993752872012626e-18,
                "value": 0
              }
            },
            "y": {
              "attributes": {
                "shape": [
                  0
                ],
                "unitSI": 1.8993752872012626e-18,
                "value": 0
              }
            },
            "z": {
              "attributes": {
                "shape": [
                  0
                ],
                "unitSI": 1.8993752872012626e-18,
                "value": 0
              }
            }
          },
          "particlePatches": {
            "extent": {
              "attributes": {
                "unitDimension": [
                  0,
                  0,
                  0,
                  0,
                  0,
                  0,
                  0
                ]
              },
              "x": {
                "attributes": {
                  "unitSI": 1
                },
                "datatype": "ULONG",
                "extent": [
                  1
                ]
              },
              "y": {
                "attributes": {
                  "unitSI": 1
                },
                "datatype": "ULONG",
                "extent": [
                  1
                ]
              },
              "z": {
                "attributes": {
                  "unitSI": 1
                },
                "datatype": "ULONG",
                "extent": [
                  1
                ]
              }
            },
            "numParticles": {
              "attributes": {
                "unitSI": 1
              },
              "datatype": "ULONG",
              "extent": [
                1
              ]
            },
            "numParticlesOffset": {
              "attributes": {
                "unitSI": 1
              },
              "datatype": "ULONG",
              "extent": [
                1
              ]
            },
            "offset": {
              "attributes": {
                "unitDimension": [
                  0,
                  0,
                  0,
                  0,
                  0,
                  0,
                  0
                ]
              },
              "x": {
                "attributes": {
                  "unitSI": 1
                },
                "datatype": "ULONG",
                "extent": [
                  1
                ]
              },
              "y": {
                "attributes": {
                  "unitSI": 1
                },
                "datatype": "ULONG",
                "extent": [
                  1
                ]
              },
              "z": {
                "attributes": {
                  "unitSI": 1
                },
                "datatype": "ULONG",
                "extent": [
                  1
                ]
              }
            }
          },
          "position": {
            "attributes": {
              "macroWeighted": 0,
              "timeOffset": 0,
              "unitDimension": [
                1,
                0,
                0,
                0,
                0,
                0,
                0
              ],
              "weightingPower": 0
            },
            "x": {
              "attributes": {
                "shape": [
                  0
                ],
                "unitSI": 1.7719999774007647e-07,
                "value": 0
              }
            },
            "y": {
              "attributes": {
                "shape": [
                  0
                ],
                "unitSI": 4.429999943501912e-08,
                "value": 0
              }
            },
            "z": {
              "attributes": {
                "shape": [
                  0
                ],
                "unitSI": 1.7719999774007647e-07,
                "value": 0
              }
            }
          },
          "positionOffset": {
            "attributes": {
              "macroWeighted": 0,
              "timeOffset": 0,
              "unitDimension": [
                1,
                0,
                0,
                0,
                0,
                0,
                0
              ],
              "weightingPower": 0
            },
            "x": {
              "attributes": {
                "shape": [
                  0
                ],
                "unitSI": 1.7719999774007647e-07,
                "value": 0
              }
            },
            "y": {
              "attributes": {
                "shape": [
                  0
                ],
                "unitSI": 4.429999943501912e-08,
                "value": 0
              }
            },
            "z": {
              "attributes": {
                "shape": [
                  0
                ],
                "unitSI": 1.7719999774007647e-07,
                "value": 0
              }
            }
          },
          "weighting": {
            "attributes": {
              "macroWeighted": 1,
              "shape": [
                0
              ],
              "timeOffset": 0,
              "unitDimension": [
                0,
                0,
                0,
                0,
                0,
                0,
                0
              ],
              "unitSI": 1,
              "value": 0,
              "weightingPower": 1
            }
          }
        }
      }
    }
  }
}

@franzpoeschel franzpoeschel force-pushed the topic-json-short-modes branch 3 times, most recently from c511364 to 544923b Compare October 12, 2023 09:39
@franzpoeschel franzpoeschel mentioned this pull request Oct 18, 2023
4 tasks
@franzpoeschel franzpoeschel force-pushed the topic-json-short-modes branch 2 times, most recently from 8ca52da to 6464af8 Compare November 20, 2023 16:36
@@ -1269,40 +2009,165 @@
return (*obtainJsonContents(file))[filePosition->id];
}

void JSONIOHandlerImpl::putJsonContents(
auto JSONIOHandlerImpl::putJsonContents(

Check warning

Code scanning / CodeQL

Poorly documented large function Warning

Poorly documented function: fewer than 2% comments for a function of 161 lines.
@@ -1560,12 +1603,13 @@
}
};

inline void write_test(const std::string &backend)
inline void write_test(

Check warning

Code scanning / CodeQL

Poorly documented large function Warning test

Poorly documented function: fewer than 2% comments for a function of 134 lines.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant