RECIPE_WORKFLOW.rst

Recipe workflow

Documentation of how data is processed to go from a recipe to a final product.

See GLOSSARY.rst for a detailed explanation of the terms used in arkimet.

Loading recipes

Recipes are loaded by scanning the recipes directory recursively for .yaml files (see arkimapslib.kitchen.Kitchen.load_recipes()).

The inputs part of each recipe is added to a global repository of input definitions.

The flavours part of each recipe is added to a global repository of flavour definitions.

For each recipe which has a recipe section, a product is created with the same name as the name of the recipe file (without .yaml extension).

Filling the pantry

arkimaps is designed to process an unorganized stream of GRIB data.

The stream is processed matching each GRIB data with the filter expressions provided in the input definitions found in recipes, storing each matched GRIB in a directory called pantry inside the working directory.

Inside the pantry, each input found in this way is stored as [$model_]$name+$step.grib, where:

• $model is the model name as found in the input definition, if the input definition defined filtering rules for different models
• $name is the input name
• $step is the step detected for this input: endStep is used when filtering using ecCodes, and a function of the Timerange metadata is used when filtering using Arkimet.

See EccodesPantry.fill() and ArkimetPantry.fill() in arkimapslib.pantry.

Making orders

For each flavour requested for rendering, and for each recipe that can be rendered in that flavour, the pantry is scanned to see which inputs are available and for which steps, and the result is used to build a list of all products that are expected in the output.

During this process, arkimaps also computes those input data that need to be generated as derivations of other input data.

Filtering recipes

A flavour can optionally define a recipes_filter which limits what recipes can be built using that flavour.

As a first step of making orders, each flavour limits the recipes built with it to only those it allows.

See arkimapslib.kitchen.WorkingKitchen.make_orders()

Postprocessing input data

Inputs defined in recipes have a type that defines if they correspond to GRIB data as found in the stream to be processed, if they need to be fetched from other locations, or if they need to be computed from other inputs.

See INPUTS.rst for a detailed description of available input types.

Computing model steps for which an order can be made

For each recipe, arkimaps scans its recipe steps to see what inputs they use, building a list of inputs used by the recipe.

For each input used by the recipe, arkimaps queries the pantry for the set of model steps that are available for it, and intersects all sets found.

The result is the list of model steps that are available for all inputs of the recipe: an order made for those steps will have all needed inputs available for rendering.

See arkimapslib.flavours.Flavour.make_orders()

Generating the orders list

Having computed all the inputs needed for all model step for which a product can be computed, arkimet builds a list of orders for the renderer, one for each product.

For each order, arkimaps also computes the list of recipe steps found in the recipe, plus all the arguments for each step, computed taking the current flavour into account.

Rendering orders

Once an order has been prepared with all its inputs, rendering it only needs executing all recipe steps in order, and writing the resulting image to the working directory. This is done in arkimapslib.orders.Order.prepare().

Since each order is self-contained, it can be rendered in parallel together with all other orders. This is orchestrated by arkimapslib.render.Renderer.renderer, which distributes the orders to be rendered to a to a multiprocessing pool of simple executors.