BeerJSON

[matty0ung]

As noted elsewhere, I've made a start on this. Subject to what people think, I'm going to try to do this differently than some of the previous "big" changes.

The actual reading in of JSON is not that hard (I think!). I already have code that validates a file against the BeerJSON schema for instance. (Two external libraries do most of the heavy lifting for that bit.) But there are likely to be a lot of code changes required to accommodate the richer content available in BeerJSON compared with the BeerXML 1.0 that we know and love/hate/etc. To take just one example, BeerJSON allows you to specify the units for most fields. Some of the units we support (eg diastatic power) and some we don't (eg pressure, which can be kPa, PSI or bar) and some we support partially (eg BeerJSON supports fluid ounces as a volume measurement, which we don't).

So, I'm thinking I'll try, over time, to make a series of PRs that eventually get us to BeerJSON support. Eg, one PR might be cleaning up and extending the unit system. Another might be adding the external (header-only) libraries we use to parse JSON files and do schema validation. Another might bring additional fields or objects (eg in BeerJSON a Recipe can have a pH field representing "The final beer pH at the end of fermentation") - though it may be we decide some of these extras can be left until we have a minimal version if import/export working.

The upside is that each PR is small(er) compared with doing one "everything we need for BeerJSON PR". The downside is that some of them might not be very exciting and I don't yet know how many there will be.

Anyway, would be good to get folks' views.

[matty0ung]

FYI for anyone taking an interest, I now have BeerJSON support at the point where the program can read in (and store) Hop records from a BeerJSON file. This includes all the new attributes of hops that are not present in BeerXML (and need extra properties & DB columns in Brewtarget / Brewken).

I think adding the ability to write Hop records to a BeerJSON file will be next, then other basics (fermentables, miscs, etc) should follow with not too much work.

Recipes will likely be a bit more work, but, overall, we are slowly getting there.

[b-a-p]

Hi @matty0ung,
Many thanks for your efforts! It's very needed change and it is perfectly fine to do it a small step in once instead of trying to eat an elephant at whole :).

[matty0ung]

For anyone following along, here is a bit of an update on the BeerJSON work.

I've been working on quite a number of different requirements for BeerJSON. Some are relatively straightforward and
others less so. The main areas are listed below. Current status is that 1, 2 and 6 are pretty much done; 5 is a no-op;
7 and 8 are in progress; 3 and 4 are done for Hop and in progress for Fermentable. I'm expecting the remaining work
to be relatively straightforward until I get to Recipe which might be a bit complicated.

1. We need to be able to read and write JSON. Boost.JSON takes care of most of the heavy lifting here. We do a bit of
   work to add pretty-printing and to convert between various string representations (Qt strings, C++ standard library
   strings, C-style strings) but the main work was on the build system to make sure that a new enough version of Boost
   is installed on Linux. (In parallel, and as a result of the packaging pain of the 3.0 release, I have been doing a
   lot of investigation on build systems. My current thinking is that Meson + Python scripts could end up being cleaner
   than CMake + CPack + bash scripts, including for simplifying the initial set up your build environment. I have the
   config, build and install working in Python+ Meson on Brewken but have yet to really tackle packaging. The plan is
   to come back to this once BeerJSON is done.)

2. We want to be able to validate BeerJSON files against the BeerJSON schema. This has several benefits:

   • We can give better diagnostics to the user if the file they are trying to read is invalid in any way
   • During development we can have greater certainty that we are writing out valid BeerJSON because we just need to try
     to read it back in again and it will get validated against the schema
   • Our parsing code is simpler because the schema validation eliminates the need to deal with a whole bunch of edge
     cases in code

   The heavy lifting for validation is done by Valijson, which can work with most C++ JSON libraries, including
   Boost.JSON. There's some small build system work to pull in Valijson, but it's also hosted on GitHub so we can do
   it as a git sub-project.

3. BeerJSON introduces a lot of additional fields compared with BeerXML. Adding all these fields to Hop, Fermentable,
   Misc, etc has a bunch of direct work:

   • Adding properties and getters/setters to each of the classes in question
   • Adding fields to the relevant database tables (see database/DatabaseSchemaHelper.cpp) and adding mappings in
     database/ObjectStoreTyped.cpp
   • Adding non-standard fields to our BeerXML serialisation so that values are not thrown away if the user exports to
     XML (though it's unlikely those values will be readable by any program other than Brewtarget/Brewken). This is
     possible because BeerXML 1.0 permits you to add non-standard fields without invalidating the file (something we
     already use for a few fields such as Fermentable::isMashed).
   • Updating the UI so you can see and edit the field values

   However, I have not done anything yet on incorporating new values into recipe calculations etc.

4. Some existing fields need to be modified - eg BeerJSON has more types of Fermentable than BeerXML. In addition to
   extending the class and its DB serialisation to support the new enum values, we also need to extend the XML code so
   that it won't fall over when it encounters one of the new values. New values get mapped to the "closest" old value
   in XML and the "real" value recorded in an XML comment. Some information will be lost when the XML file is
   reimported (even to Brewtarget/Brewken) but things will be approximately right (eg if you used the new Hop form of
   powder which is not supported in BeerXML then it will get exported (and read back in) as pellet if you round trip
   it out to XML. (But storage in the DB and export to/from BeerJSON will, of course, retain the powder value.)

5. In some cases a field in BeerXML is no longer supported in BeerJSON - eg Fermentable::addAfterBoil. For the moment
   I have mostly not worried about this, other than noting it in the comments, on the assumption that the field is
   either unimportant or is represented in some other way in BeerJSON.

6. The new fields in BeerJSON also, by implication, introduce a lot of new measurement scales. In some cases these are
   new things being measured (eg carbonation); in others these are new scales (eg Lovibond) or new points on existing
   scales that we didn't have before (eg pints and fluid ounces did not used to be present on our imperial scales).

   In BeerXML, we didn't have to worry about weights and measures so much because values were always in the same
   units (all weights in kilograms, all volumes in liters, etc). We could display things in metric/imperial/etc but
   all the internal storage, including in XML, was in fixed metric units

   In BeerJSON, the a weight might be recorded in kilograms, grams, pounds or ounces. There was quite a lot of work
   to tidy up how we do weights and measures, and to make it sufficiently flexible and extensible to accommodate all the
   new things used in BeerJSON. This also involved formalising what we mean by "metric/imperial/etc", to avoid having
   lots of special-case handling in code. See long comment in measurement/PhysicalQuantity.h for more detail.

7. The presence of all these new fields also forced us to be more explicit about handling "optional" or "nullable"
   fields. For instance, neither the Fermentable::grainGroup enum field nor the Fermentable::coarseFineDiff_pct
   percentage field has a meaningful value if Fermentable::type is Sugar, Fruit, Juice or Honey. In such
   cases we want to exclude the meaningless fields from export in BeerJSON (or BeerXML) and ensure they are stored as
   NULL in the DB. I have tried to do this "properly" by wrapping optional fields in std::optional (which is more
   type-safe than using QVariant for the same purpose) and introducing a new static isOptional() member function on
   NamedEntity and its descendants. This allows us to handle optional fields relatively generically in the DB, XML
   and JSON serialisation code, with a small compromise about using std::optional<int> to hold an optional enum.

8. There is an extra bit of thought needed for things that can be measured either by weight or by volume. In the past,
   we have relied on explicit amountIsWeight flags on Yeast, Misc and so on, which had corresponding fields in
   BeerXML. BeerJSON doesn't explicitly use such flags because it always gives you the units of measurement (liters,
   pints, kilograms, pounds, etc, etc) and its "obvious" from these whether something is being measured by weight or by
   volume. BeerJSON also introduces the possibility that a Fermentable can be measured by volume (which eg makes a
   lot of sense if it's of type Juice). I have tried to make the handling for "by weight or by volume" measurements
   more generic (including introducing MassOrVolumeAmt in measurement/Amount.h) whilst maintaining as much backwards
   compatibility as possible for BeerXML. (Realistically, if you try and store a Fermentable of type Juice in
   BeerXML, it's not going to come back unscathed on the round trip!)

As ever, happy to answer questions, discuss approach etc.

[mikfire]

A few thoughts and questions.

I am in favor of us always writing in SI. I feel no obligation to consider the user's preferences when creating what is supposed to be a machine interchange file. If the user wants to read our beerJSON, they will need to work in SI. We still need to be able to read files from other sources, so we will still need to deal with stupid systems of measure. I just think we should do our part to kill US Customary.

I am curious how beerJSON distinguishes between an ounce, an ounce, an ounce and an ounce. That is, between a volume measure (aka, "liquid ounce"), a mass measure (aka, "dry ounce" approximately 28g), a US liquid ounce (29.6ml) and an Imperial liquid ounce (28.4ml). I am somewhat concerned that "obvious" is going to take a lot of effort to figure out.

[matty0ung]

Agreed. We'll carry on storing stuff in kilograms, liters and so on internally and will write in those units to BeerJSON. We'll only need to do the conversions when reading in.

For distinguishing things, BeerJSON has, eg "oz" for weight, "floz" for US Customary fluid ounce and "ifloz" for Imperial fluid ounce. If we discover anything they've missed, then they do have a GitHub repository where we can raise issues and propose PRs. (So far, I've found a few typos in the doco and a couple of relatively minor other issues not related to weights & measures - specifically that Hop year is stored as a string rather than an integer, and that there is not obvious place when writing a BeerJSON file to record the name and version of the application that wrote it, which would be useful for debugging.)

[matty0ung]

Another update. I think N° 7 is mostly done, except for UI support, which should be just "blank = null = std::nullopt" and which I'll come back to soon.

I'm currently working on the joys of "mixed" fields. Historically these were things that could be measured by weight or by volume. But, of course, now there are things which could be measured by "mass concentration" (eg milligrams per liter) or "volume concentration" (eg parts per million). So I need to make "mixed" a bit more generic to handle this, which has a silver lining of making PhysicalQuantity a bit cleaner.

[matty0ung]

Things are starting to come together. No 7 and 8 above are finally largely done, including the UI support (where there was a bit more to do than I thought). I can read and write Hop, Fermentable & Misc from/to BeerJSON, and show and edit the new fields that BeerJSON offers.

A few details:

• We can now handle "mass concentration" and "volume concentration" as either-or measurements, just as we already do for "by weight" or "by volume", but we now do it all in a reasonably generic way.
• I've refactored the way we do edit fields to make it easier to add new fields. We retain all the existing clever logic (not written by me!) to allow you to flip measurement systems on individual fields (eg from metric to imperial) and also to eg force a particular scale (eg minutes in preference to hours or seconds). There's less code duplication now, so hopefully things like handling decimal separators and display of units should be more consistent.
• Related to this, we now store more info about pretty much every field, eg we can detect at runtime that a field is a temperature or a pH, etc and we know whether it's stored as a decimal or integer, and we know whether it's optional or required. This makes it easier to do a lot of generic handling in all sorts of different places in the code. Broadly speaking, it means that doubling the number of fields doesn't require us to double the size of the code base.
• I've done a fair bit of work to get the UI to support optional fields. Again, all related to the above. Eg, where a field is a multiple-choice (eg Fermentable type), we now have generic code to construct the drop-down "combo box" which also adds a "blank" option if it's an optional field.

I've made a little tracker page at https://github.com/Brewken/brewken/wiki/BeerJSON for some of the turn-the-handle work. Although it looks like there is "a lot" to do, it should be rather less than it appears because a lot of what has been done to date is reusable.

[pricelessbrewing]

Just wanted to pop in and say nice job so far and sorry for creating all the mixed-field unit systems 😇

[matty0ung]

OK, Styles are done and I can import BJCP Beer Style Guidelines in BeerJson format from https://github.com/beerjson/bjcp-json. (This helped uncover a couple of bugs in handling of optional values.)

Looking at Equipment now. This is a bit more fun, as BeerJSON and BeerXML have a different approach. In BeerXML, there is a single Equipment record type for all the hot-side equipment a recipe might need to refer to. In BeerJSON equipment is modelled as a list of vessels of various forms ("HLT", "Mash Tun", "Lauter Tun", "Brew Kettle", "Fermenter", "Aging Vessel", "Packaging Vessel"). I think the BeerJSON structure is overly flexible, because it allows oddball set-ups like 3 mash tuns and 2 brew kettles. I'm going to assume that 99.9% of the time you only need at most one of each vessel in your equipment description. That way we retain a single overall Equipment record, and use the forms (HLT, Mash Tun, etc) as subrecords. (If it turns out that it's really important to model multiple Fermenters or something then we can make an array of them inside Equipment, but I'll avoid that complexity for now.)

Long story short, I'm now tackling some more general work to do (on how to map BeerJSON arrays to our internal record structure) before I get back to the turn-the-handle stuff.

[matty0ung]

Equipment is pretty much done now. Just tidying up the "show lists of things" code (including to remove a bit of duplication and make things more consistent). Along the way, ended up doing a few neat things (well, they felt neat at the time anyway!):

• We can now refer to named array items with an XPath-like syntax, eg equipment_items[form=\"Aging Vessel\"]/type, and this works both for reading data in and writing it out.

• Quite a lot of work on the Equipment editor to handle all the additional "optional" vessels. You can now hide/show such things as Aging Vessels and Packaging Vessels (which have their own tabs) and we hide vessels by default when they have a zero volume. I've also tried to ensure we "do the right thing" now that lautering deadspace can be on two different vessels (lauter tun if you have one, mash tun if you don't).

• As I was doing the Equipment editor work, it seemed the right time to add the possibility to put an "info button" ⓘ next to certain fields where we want the user to be able to access a bit more explanatory information than will fit in a tooltip. After a bit of fighting with Qt, I got it that these things can be added to the Qt Designer (.ui) files without the need for any additional C++ code.

I think next up are Mash and MashStep.

[matty0ung]

So, now we come to the tricky part, Recipes. This is where BeerJSON is going to force us to look more closely at something relatively fundamental that is (for entirely valid historical reasons) not really how we want it to be in the code.

In the current code, when we have, say, a hop record, it's always the same type of record, whether it's the "master" record (storing everything we know about eg Fuggle hops) or the "use of" record (storing info about the addition of some Fuggle hops to a particular recipe). Back in the day when everything was stored in files and we didn't even have a database, this was the right thing to do to make every recipe completely self-contained. And BeerXML didn't draw a distinction, so we were aligned with that. Since then, things have changed, and everything is stored in a database, so it's now a bit duplicatory (Is that a word?) to make a full copy of the master Fuggle hop record every time we add Fuggle hops to a recipe. But everything works and it looks rather painful to change, so we haven't worried too much about it.

BeerJSON however, being a bit more modern, draws a distinction between a hop and a use of a hop (and the same for fermentables, yeasts, etc, etc). Eg, for a "Fuggle" hop, there are two types of record:

• A freestanding hop record containing lots of information about the Fuggle variety of hop to go in the 'List of all Hops we know about'"
• A hop-addition to recipe record, containing enough information to tell you to add 5 grams of 2022 harvest leaf Fuggle hop with 4.2% alpha acid at 60 minutes before boil end

So, we have to make a decision about how try to proceed. We might be able to get away with forcing the two types of BeerJSON hop record into our single all-purpose all-singing all-dancing hop record, depending on the context. We probably ought to bite the bullet and start splitting our hop record out into the two different things it really is. But this might be a fair bit of work.

I'll have a think about this, and maybe try a couple of different things. I think it's the last substantive issue to solve before we can declare BeerJSON support working. If anyone wants to chime in with their thoughts and suggestions, please do; otherwise, watch this space.

[matty0ung]

I think I have a handle now on the right way to tackle this. It's going to involve a bit of refactoring, but we'll end up in a better place, where the Hop/Fermentable/etc is one thing and its use in a Recipe is another.

(The relevant part of BeerJSON inheritance model was confusing me a bit until I realised we need to think about things differently in our internal data model. In BeerJSON, the hop/fermentable/etc additions contain a lot of fields from the underlying hop/fermentable/etc in case it's not one you already have in your database. We don't need this duplication internally; we just need to create a new Hop if we read in a Recipe that refers to one we don't yet know about.)

[matty0ung]

Things are coming along reasonably well. We now have a new class RecipeAdditionHop, which represents the addition of a Hop to a Recipe. This means we can do away with hop children etc as discussed above. There is still a fair bit of work to do (including to make reading/writing BeerXML compatible with our new internal structures), but I've already got as far as getting everything to compile again(!) so it's an important psychological milestone. 😄 Once everything is working properly for RecipeAdditionHop, it should be relatively straightforward to do the corresponding work for Fermentable, Misc, Yeast.

One other structural change is that BeerJSON supports raw ale - ie beers brewed without a boil. At the moment, my plan is to support such "no-boil" recipes in the data structures, but leave fully enhancing the UI for a separate future project.

[matty0ung]

Of course, once you start thinking about what a "recipe addition" is, you then start asking what the quantity of a recipe addition can be, because different ingredients can be measured in different ways - eg some can be measured only by weight, others by weight or volume, and some (eg yeast) even by "count" (ie number of packets). The same question applies when you think about inventory (of all the same things). Previously, under the influence of BeerXML, there had been case-by-case flags for amountIsWeight etc. But that starts to get a bit painful when we want to write generic code for all ingredients.

So now we have a class IngredientAmount which is used by RecipeAdditionHop, InventoryHop and will be used by future similar classes for fermentable, misc, yeast, etc.

I've also removed the idea of a hop or another ingredient having a default amount. AFAICT it's a hangover (geddit!) from BeerXML which requires the amount field on Hop etc even when it's not inside a recipe. I can't think that it's useful in reality, and we can write a dummy amount to non-recipe BeerXML ingredient records to keep them "correct", so I don't expect anyone to mourn its passing.

I still need to tidy up the UI and finish rewiring the XML / JSON mappings, but we're slowly getting there.

[matty0ung]

In addition to Hop, I've now coded recipe addition classes for Fermentable, Misc and Yeast, plus the corresponding Inventory classes. Now need to do Water (recipe addition but no inventory). As part of that, will bring the way we handle water closer into line with other "ingredients". (At the moment, if you import a recipe with a water profile from BeerXML, I don't think the water profile shows up anywhere.)

[matty0ung]

BeerJSON support is now live in Brewtarget 4.0.