diff --git a/doc/arts/develop.rst b/doc/arts/develop.rst index 92ca2ffb60..8e6c03bca5 100644 --- a/doc/arts/develop.rst +++ b/doc/arts/develop.rst @@ -9,4 +9,4 @@ It contains design descriptions and guidelines for the development of ARTS. develop.classes develop.error - develop.workspace_methods + develop.workspace diff --git a/doc/arts/develop.workspace.agendas.rst b/doc/arts/develop.workspace.agendas.rst new file mode 100644 index 0000000000..8b47667a02 --- /dev/null +++ b/doc/arts/develop.workspace.agendas.rst @@ -0,0 +1,47 @@ +Workspace agendas +################# + +At their simplest, workspace agendas uses a set of :doc:`develop.workspace.methods` that combined requires +a set of :doc:`develop.workspace.variables` to produce another set of workspace variables. + +Workspace agendas are defined in the ``workspace_agendas.cpp`` file. +The workspace agenda definitions are located in the ``wsa_data`` map object. The name of the agenda is the key of the map and the object is a struct with the following fields: + +- ``desc`` - a description of the agenda as a string. +- ``outputs`` - the workspace variables that are produced by the agenda as a list of strings. +- ``inputs`` - the workspace variables that are consumed by the agenda as a list of strings. +- ``array`` - a boolean for whether or not the agenda is described as an :class:`~pyarts.arts.ArrayOfAgenda` (if true) or an :class:`~pyarts.arts.Agenda` (if false, default). + +What quailifies a workspace agenda? +=================================== + +A method or a set of methods must be able to use the workspace agenda to produce a set of desired workspace variables. +Generally, if you can keep the number of inputs limited and the number of outputs fixed, you may consider a workspace agenda. +However, overpopulating the workspace with agendas is not recommended. So please reuse existing ones as appropriate. + +Take :attr:`~pyarts.workspace.Workspace.ray_path_observer_agenda` as an example. The different ways we can define a path +from an observer to a background is practically limitless. However, if we are interested in :attr:`~pyarts.workspace.Workspace.spectral_radiance` +for some observer and frequency, +we know that we just need the background radiation from the background of the observed :attr:`~pyarts.workspace.Workspace.ray_path` and the state of the +atmosphere along the path. So a method like :func:`~pyarts.workspace.Workspace.measurement_vectorFromOperatorPath`, which takes a set of frequencies and a set of observers, +may effectively ignore the complexity of computing the :attr:`~pyarts.workspace.Workspace.ray_path` internally, and leave that to the workspace agenda. + +Generated files +=============== + +The workspace group interface generates ``auto_wsa.cpp`` and ``auto_wsa.h`` files for the C++ interface. +No additional files are generated for the Python binding. + +Workspace agenda naming convention +================================== + +Workspace agendas should be named in ``snake_case`` as they are also :doc:`develop.workspace.groups`. + +Do not hesitate to implement helpers +==================================== + +Several agendas have helper methods to make the user interface easier. +These methods should follow a simple naming convention. + +- There are methods that sets an agenda to a named option, such as :func:`~pyarts.workspace.Workspace.spectral_radiance_space_agendaSet`. +- There are methods that compute an agenda based on existing :doc:`develop.workspace.variables`, such as :func:`~pyarts.workspace.Workspace.propagation_matrix_agendaAuto`. diff --git a/doc/arts/develop.workspace.groups.rst b/doc/arts/develop.workspace.groups.rst new file mode 100644 index 0000000000..b703681b8a --- /dev/null +++ b/doc/arts/develop.workspace.groups.rst @@ -0,0 +1,68 @@ +Workspace groups +################ + +Workspace groups are the types of the :doc:`develop.workspace.variables`. +They define the logic that can be applied to the data. +Workspace groups are all available in the :py:mod:`~pyarts.arts` module. + +Most workspace groups are defined in the ``workspace_groups.cpp`` file. +Additionally, :doc:`develop.workspace.options` are automatically turned into workspace groups using their definitions in the ``arts_options.cpp`` file. + +Defining a workspace group +========================== + +In ``workspace_groups.cpp`` +--------------------------- + +The workspace group definitions are located in the ``workspace_groups.cpp`` +as part of the ``wsg_data`` map object. The name of the group is the key +of the map and the object is a struct with the following fields: + +- ``file`` - the main header file that must be included to use the workspace group. +- ``desc`` - a description of the variable as a string. +- ``array_depth`` - an integer defining the depth of the array. You can recursively access data of this type of array using the ``[]`` operator using an integer or range type. Remember that the ``value_type`` of the array should also be a workspace group. +- ``value_type`` - a boolean that defines whether or not a workspace group instance of this type is copyable in python. You cannot copy python types such as ``int`` and ``str``. +- ``map_type`` - a boolean that defines whether or not the workspace group is a map. Maps can be accessed via the ``[]`` operator using the key type. Remember that both the ``mapped_type`` and the ``key_type`` of the map should also be workspace groups. + +In ``arts_options.cpp`` +----------------------- + +See the page on :doc:`develop.workspace.options` for more information. +The workspace group defined from a workspace option will +simply be the name of the ``enum class`` that defines the option. + +What quailifies as a workspace group? +===================================== + +You need to ensure that the following code is possible for each workspace group ``T``. +Read this assuming that ``a`` is an instance of ``T``, ``x`` is an instance of ``std::formatter``, ``i`` is an instance of an integer, and ``k`` is an instance of a "key". +The following must compile: + +- ``xml_read_from_stream(std::istream &, T&, bifstream *)`` - allow reading the workspace group from an XML file. Failure to comply leads to a linker error. +- ``xml_write_to_stream(std::ostream &, const T&, bofstream *, const String &)`` - allow writing the workspace group to an XML file. Failure to comply leads to a linker error. +- ``T{}`` - allow default construction. Failure to comply leads to a static assertion as the group fails the ``WorkspaceGroupIsDefaultConstructible`` concept. +- ``T{a}`` - allow copy construction of. Failure to comply leads to a static assertion as the group fails the ``WorkspaceGroupIsCopyable`` concept. +- ``std::format("{}", a)``, ``std::format("{:sqNB,}", a)``, ``x.inner_fmt().tags``, and ``static_assert(std::same_as>)`` - allow formatting the group to a string. The exception to this are classes that pass one of these consepts: ``std::integral`` or ``std::floating_point`` or ``std::same_as``. Failure to comply leads to a static assertion as the group fails the ``arts_formattable_or_value_type`` concept. Note that to ensure this, you should read the :doc:`develop.classes.formatter` documentation. +- ``x[i]`` and ``x[k]`` should also implement all of the above if the group is an array or map type, respectively. This also holds true for the group of ``k`` for map types. Failure to compile will lead to difficult errors in the python binding compilation. + +Additionally, you need to ensure that your group passes the ``pytest`` tests when ``pyarts`` is built. There exist a helper method ``workspace_group_interface`` you can send a mutable version +of the ``py::class_`` object to that should ensure this. However, generally they interface via python might require some additional work. + +Generated files +=============== + +The workspace group interface generates ``auto_wsg_init.cpp``, ``auto_wsg_share.h``, +``auto_wsg.h`` and ``auto_workspace.cpp`` files for the C++ interface. +It also generates ``py_auto_wsg.cpp``, ``py_auto_wsg.h``, and ``py_auto_wsgdocs.h`` for the Python binding. + +Workspace group naming convention +================================= + +Workspace groups should be named in ``PascalCase``. The name should be +descriptive of the logic that is handled by the group. + +File naming +=========== + +This is up to the developer. However, try to place the group as early in the compilation as possible. +Preferably, the group should only be part of the ``artscore`` CMake target (or one of its dependencies). diff --git a/doc/arts/develop.workspace_methods.rst b/doc/arts/develop.workspace.methods.rst similarity index 82% rename from doc/arts/develop.workspace_methods.rst rename to doc/arts/develop.workspace.methods.rst index 80f0380332..0fa48b3f13 100644 --- a/doc/arts/develop.workspace_methods.rst +++ b/doc/arts/develop.workspace.methods.rst @@ -33,24 +33,29 @@ method is the key and the value is a struct with the following fields: - ``desc`` - a description of the method as a string. - ``author`` - the author(s) of the method as a list of strings. -- ``out`` - the output of the method as a list of strings. These must be workspace variables. -- ``gout`` - the generic output of the method as a list of strings. These must not be workspace variables. -- ``gout_type`` - the type of the generic output as a list of strings. These must be workspace groups. +- ``out`` - the :doc:`develop.workspace.variables` output of the method as a list of strings. +- ``gout`` - the generic output of the method as a list of strings. These must not be :doc:`develop.workspace.variables`. +- ``gout_type`` - the type of the generic output as a list of strings. These must be :doc:`develop.workspace.groups`. - ``gout_desc`` - the description of the generic output as a list of strings. -- ``in`` - the input of the method as a list of strings. These must be workspace variables. -- ``gin`` - the generic input of the method as a list of strings. These must not be workspace variables. -- ``gin_type`` - the type of the generic input as a list of strings. These must be workspace groups. -- ``gin_value`` - the default value of the generic input as an optional workspace value. +- ``in`` - the :doc:`develop.workspace.variables` input of the method as a list of strings. +- ``gin`` - the generic input of the method as a list of strings. These must not be :doc:`develop.workspace.variables`. +- ``gin_type`` - the type of the generic input as a list of strings. These must be :doc:`develop.workspace.groups`. +- ``gin_value`` - the default value of the generic input as an optional initialized :doc:`develop.workspace.groups`. - ``gin_desc`` - the description of the generic input as a list of strings. -- ``pass_workspace`` - a boolean indicating if the workspace should be passed to the method. If true, the first argument to the method is a ``const Workspace&``. +- ``pass_workspace`` - a boolean indicating if a :class:`~pyarts.workspace.Workspace` instance should be passed to the method. If true, the first argument to the method is a ``const Workspace&``. The expected signature of the method will depend on these fields. A linker error will likely occur if the actual signature does not match the expected signature. -The ``in`` and ``out`` may contain the same variable. If they do, the variable must be -initialized before the method is called because it is assumed the method is intended to -simply modify the existing value. +The ``in`` and ``out`` may contain the same :doc:`develop.workspace.variables`. If they do, the variable must be +initialized before the method is called because it is treated as if the method is intended to +simply modify the existing value. Please indicate strongly in the documentation if you sometimes overwrite the input variable. + +On the other hand, if the :doc:`develop.workspace.variables` is only in ``out`` and not in ``in``, +it is treated as if the workspace variable is created by the method. Note that since the type system +does not account for this, it is important that you clear the current state of the :doc:`develop.workspace.variables` +in a method that is intended to create a new workspace variable. The fields ``gin``, ``gin_type``, ``gin_value``, and ``gin_desc`` must be the same size. The same is true for ``gout``, ``gout_type``, and ``gout_desc``. These are user-generated @@ -99,7 +104,7 @@ treated as normal workspace method. You need to do nothing to define these methods. But please refrain from defining them manually as that may cause undefined naming conflicts. -The expected signature of the method ``propagation_matrix_agendaAuto`` is also +The expected signature of the method :func:`~pyarts.workspace.Workspace.propagation_matrix_agendaAuto` is also generated automatically near the end of ``workspace_methods.cpp``. It takes its input and output from a list of other methods. Feel free to add to this list but make sure that any naming conflicts regarding ``gin`` are resolved diff --git a/doc/arts/develop.workspace.options.rst b/doc/arts/develop.workspace.options.rst new file mode 100644 index 0000000000..1ae9fb021a --- /dev/null +++ b/doc/arts/develop.workspace.options.rst @@ -0,0 +1,38 @@ +Workspace options +################# + +Workspace options are wrappers for creating ``enum class`` types that fully define the set of valid options +that are required for "some" task. They are used instead of strings to ensure that the methods that use them +always cover the full set of solutions required (i.e., by switching over all options in a switch-statement). + +They are mostly intended as input to workspace methods, but they may also be part of variuous workspace groups +to define the logic of the group. The options are defined in the ``arts_options.cc`` file as parts of the +``opts`` list object. Each option is a struct with the following fields: + +- ``name`` - the name of the option as a string. +- ``desc`` - a description of the option as a string. +- ``values_and_desc`` - a list of lists of strings. The outer-most list defines a unique option. The inner lists must all have the same length. The inner list defines at least two strings. The first value of the inner list are used to build the ``enum class`` value and must thus be usable as valid C++ code. The last value of the inner list is the description of the option. All values inbetween are various ways to spell the option. The ``enum class`` will be constructible from strings of any of these values. + +What do you get by defining an option? +====================================== + +Using ``T`` as the option type, ``e`` as an instance of the option type, and ``s`` as an instance of ``std::string``, +you will automatic have access to the following interface for each option type: + +- ``good_enum(e)`` - returns a boolean that checks if the option is valid. +- ``toString(e)`` - returns a string representation of the option. Note that this is a templated type and you can input a templated index to select the name you want from the lists of strings. +- ``to(s)`` - returns the option from a string. It will look through all provided names for the option type. +- ``operator<<(std::ostream &os, const T x)`` - allows streamed output of the option. +- ``operator>>(std::istream &is, T &x)`` - allows streamed input of the option. +- In addition to this, all the requirements for being a :doc:`develop.workspace.group` are also automatically fulfilled. + +Generated files +=============== + +The workspace group interface generates ``enums-common-helper.h``, ``enums.cpp``, ``enums.h``, and ``enumsNAME.h`` files for the C++ interface, here ``NAME`` is the name of the option. +The Python binding is generated as the ``py_auto_options.cpp`` and ``py_auto_options.h`` files. + +Workspace variable naming convention +==================================== + +Workspace options may be named as pleased. The name should be descriptive of the type of options that it offers. diff --git a/doc/arts/develop.workspace.rst b/doc/arts/develop.workspace.rst new file mode 100644 index 0000000000..2bc1edf4a0 --- /dev/null +++ b/doc/arts/develop.workspace.rst @@ -0,0 +1,11 @@ +Workspace +######### + +.. toctree:: + :maxdepth: 2 + + develop.workspace.agendas + develop.workspace.groups + develop.workspace.methods + develop.workspace.options + develop.workspace.variables diff --git a/doc/arts/develop.workspace.variables.rst b/doc/arts/develop.workspace.variables.rst new file mode 100644 index 0000000000..97c190da29 --- /dev/null +++ b/doc/arts/develop.workspace.variables.rst @@ -0,0 +1,91 @@ +Workspace variables +################### + +Workspace variables allow named data to be passed into workspace methods. +They are attributes of the :class:`~pyarts.workspace.Workspace` object. + +Most workspace variables are defined in the ``workspace_variables.cpp`` file. +Workspace agendas are automatically defined as workspace variables from the ``workspace_agendas.cpp``. + +Defining a workspace variable +============================= + +In ``workspace_variables.cpp`` +------------------------------ + +The workspace variable definitions are located in the ``workspace_variables.cpp`` +as part of the ``wsv_data`` map object. The name of the variable is the key +of the map and the object is a struct with the following fields: + +- ``desc`` - a description of the variable as a string. +- ``type`` - the workspace group of the variable as a string. +- ``default_value`` - an optional default value of the workspace variable that is defined when a :class:`~pyarts.workspace.Workspace` object is default-initialized. + +In ``workspace_agendas.cpp`` +---------------------------- + +See the page on :doc:`develop.workspace.agendas` for more information. The workspace variable defined from the workspace agenda will +have no default value. It will be of the type :class:`~pyarts.arts.Agenda` by default but of type :class:`~pyarts.arts.ArrayOfAgenda` +if the field ``array`` is ``true``. + +What quailifies as a workspace variable? +======================================== + +The intent of a workspace variable is to pass information between workspace methods. +Workspace variables must therefore be both input and output of different workspace methods. + +If a method has output that is never used by a different method, it should not be a workspace variable. Instead, it should be a local variable of the method. As defined by workspace method, it should be a ``gout``. + +If a method has input that is not produced by a different method, it should not be a workspace variable. Instead, it should be a local variable of the method. As defined by workspace method, it should be a ``gin``. + +There are only a few exceptions to this rule: + +- Workspace agendas are not required to be output of a workspace method. They are expected to be consumed by a workspace method. The reason for this exception is that they define a special logic that may not be easily expressed as a workspace method or a workspace option. +- Workspace agendas' inputs and outputs are also not required to follow the above rule. The reason is the same as for why workspace agendas are themselves an exception. +- Workspace options are not required to be output of a workspace method. The reason is that workspace methods must fully define their logic, and workspace options are the explicitly named options that a workspace method is expected to understand. + +Generated files +=============== + +The workspace variable interface generates ``auto_wsv.cpp`` and ``auto_wsv.h`` +for the C++ interface to the workspace variables. The Python binding is generated +as the ``py_auto_wsv_N.cpp`` files, where N is an integer. + +Workspace variable naming convention +==================================== + +Names carry meaning. Please follow the naming convention below and +please do not hesitate to fix any naming inconsistencies you find. + +Workspace variables should be named in ``snake_case``. The name should be +explicit. Avoid TLAs [#f1]_ and other abbreviations that are not world-wide exclusive. +A common TLA to avoid is ``lte`` because it means different things in different +contexts (e.g., local thermodynamic equilibrium or long-term evolution). +Using ``lte`` in a name is therefore ambiguous and strongly discouraged. On-the-other hand, ``nonlte`` +is unambiguous - it is not used as anything other than non-local thermodynamic equilibrium. + +To allow searching and connecting different workspace variables to their intended context, +we have reserved a set of terms that should be used in the names of related workspace variables. +This list will connect workspace variables under a rubric ``Related workspace variables`` in the documentation. +Please note that combining more than one of these terms is possible and highly encouraged. +The workspace variable :attr:`~pyarts.workspace.Workspace.propagation_matrix` along the :attr:`~pyarts.workspace.Workspace.ray_path` +is called :attr:`~pyarts.workspace.Workspace.ray_path_propagation_matrix` to connect it to both concepts. +The full list is available in the ``workspace_variables_keywords.cpp`` file. +As of writing this, the list includes the following terms: + +- ``absorption`` - for workspace variables related to absorption. +- ``jacobian`` - for workspace variables related to jacobians. +- ``spectral_radiance`` - for workspace variables related to spectral radiance. +- ``propagation_matrix`` - for workspace variables related to propagation matrices. +- ``source_vector`` - for workspace variables related to source vectors. +- ``nonlte`` - for workspace variables related to non-local thermodynamic equilibrium. +- ``ray_path`` - for workspace variables related to ray paths. +- ``scattering`` - for workspace variables related to scattering. +- ``grid`` - for workspace variables that are also grids. +- ``measurement`` - for workspace variables related to measurements. +- ``model_state`` - for workspace variables related to model states. +- ``disort`` - for workspace variables related to DISORT. + +Check the ``workspace_variables_keywords.cpp`` file for the correct list regardless. If you read this and the list above is no longer correct, please update it. + +.. [#f1] TLA means three-letter abbreviations, and it is commonly used in text and speech to indicate why you should avoid using TLAs - abbreviations feel easy to use and common-know-how once you get used to them but the first time you encounter one you have no idea what it means, which distracts from the content being discussed. diff --git a/python/doc/gen_pyarts_arts_autoclass_rst.py b/python/doc/gen_pyarts_arts_autoclass_rst.py index a3eb1bb941..69ca843c2b 100644 --- a/python/doc/gen_pyarts_arts_autoclass_rst.py +++ b/python/doc/gen_pyarts_arts_autoclass_rst.py @@ -17,7 +17,7 @@ def doc(x): raise Exception(f"Error in doc for {x}:\n{e}") -def short_doc(v): +def short_doc(v, var=None, name=None): try: x = doc(v).split("\n") s = x[0] @@ -28,22 +28,20 @@ def short_doc(v): i += 1 if "->" in s or len(s) == 0: - if isinstance(v, property): - print("NAME:", v.fget.__name__) - attr = v.__name__ if hasattr(v, "__name__") else v + attr = v.__name__ if hasattr(v, "__name__") else name if not is_operator(attr): print( - f"WARNING [ARTS DOC] No short doc found: {attr}" + f"WARNING [ARTS DOC] No short doc found: {var if var is not None else ""}{'.' if var is not None and attr is not None else ""}{attr if attr is not None else ''}" ) return s except Exception as e: - raise Exception(f"Error in short_doc for {x}:\n{e}") + raise Exception(f"Error in short_doc for v={v}, var={var}, name={name}:\n{e}") def func(name, var): try: - return {"name": name, "short": short_doc(getattr(var, name))} + return {"name": name, "short": short_doc(getattr(var, name), var.__name__, name)} except Exception as e: raise Exception(f"Error in func for name={name}, var={var}:\n{e}") diff --git a/python/doc/source/conf.py b/python/doc/source/conf.py index 68d087a6e6..20309725c1 100644 --- a/python/doc/source/conf.py +++ b/python/doc/source/conf.py @@ -310,7 +310,7 @@ autosummary_generate = True -nbsphinx_execute = "always" +nbsphinx_execute = "auto" nbsphinx_execute_arguments = [ "--InlineBackend.figure_formats={'svg'}", diff --git a/src/core/sensor/obsel.cpp b/src/core/sensor/obsel.cpp index 063a6ea9f7..9cae5013cf 100644 --- a/src/core/sensor/obsel.cpp +++ b/src/core/sensor/obsel.cpp @@ -309,7 +309,7 @@ void set_poslos(const SensorObsel& v, for (auto& elem : sensor) { if (elem.poslos_grid_ptr() == ps) { - elem.set_poslos_grid_ptr(ps); + elem.set_poslos_grid_ptr(ps); // may change here } } } diff --git a/src/workspace_variables.h b/src/workspace_variables.h index ddb15918bd..523157d9d2 100644 --- a/src/workspace_variables.h +++ b/src/workspace_variables.h @@ -7,9 +7,9 @@ #include "workspace_wsv.h" struct WorkspaceVariableInternalRecord { - std::string desc{}; - std::string type{}; - std::optional default_value{}; + std::string desc; + std::string type; + std::optional default_value{std::nullopt}; }; std::unordered_map