Intent Properties: ordering & references

[NSoiffer]

This issue is meant to pull together some threads on how properties are handled, whether there is an order, and what happens if there is a conflict. Any other conceptual issues related to properties are also appropriate here. Their syntax is part of #448.

I am probably missing some issues raised elsewhere (#446, ???), but here's a starting list:

1. how should a headless property "headless" be interpreted (standalone and when referenced)
2. if multiple properties are given, does the order matter?
3. if a property exists on a referenced item and the referencing item has a property, if order matters, what's the order that should be used?

It would be good to ground some of these examples with real life cases. Hopefully people can generate some.

[davidcarlisle]

some possible answers

1. I think it should apply to the interpretation of the current element so <mtable intent=":system-of-equations"> would process the table "knowing" it's a displayed equation layout not a matrix (whether or not it uses that property)
2. I would say no. The properties can not (currently?) set values you can not have foo:VerbosityLevel=3 (a,b,c) so I think it is best to document them as a set of boolean properties. A system may look for properties in a system specfic way so the order in the document may or may not matter.
3. I think $op .... arg="op" intent="xxxx" should be interpreted as reference(xxxx) (where reference, which could be spelled as _) is a metafunction which just processes its argument. This avoids adding a new grouping construct.
   Then $op:property .... arg="op" intent="xxxx:prop2" is
   reference:property(xxxx:prop2) (or _:property(xxxx:prop2) )
   so the behaviour is still system specific, but not specific to argref, whatever applies to nested functions applies in the same way. Here, xxxx may be an "implied intent" if the referenced element does not have an intent attribute.

[dginev]

A single example that exhibits all questions in the original description appears to be:

<mrow intent="$op:prefix($x,$y)">
  <mi arg="x">x</mi>
  <mo arg="op" intent=":infix:postfix">+</mo>
  <mi arg="y">y</mi>
</mrow>

[davidcarlisle]

so I think that example should act like

<mrow intent="_:prefix (_plus:infix:postfix)(x,y)"/>

after that everything is system specific but possibly it starts on the _:prefix so chooses a prefix reading, then ignores the fixity properties on _plus as it's not looking at a funcall at that point, so ends up

plus x y

[NSoiffer]

@davidcarlisle: I'm confused by your example. Where does the "plus" come from? Are you saying the system should process the reference item and then lift the processed item?

In MathCAT, after a few iterations, I ended up doing the opposite: I converted a literal head into what MathCAT will treat it as. A reference (that isn't a further reference), is then treated as if it is lifted and replaces the reference (this behavior is in the templated string implementation in MathCAT). So `intent='foo:prefix(x)' becomes

<mrow>
  <mi>foo</mi>
  <mi>x</mi>
</mrow>

If we start with

<mrow intent="$op:prefix($x,$y)">
  <mi arg="x">x</mi>
  <mo arg="op" intent=":infix">+</mo>
  <mi arg="y">y</mi>
</mrow>

the arguments just get replaced so the result is

<mrow>
  <mo intent=":infix">+</mo>
  <mi>x</mi>
  <mi>y</mi>
</mrow>

This implementation is making choices about multiple properties: the parent properties are being used and the child ones are ignored at the parent level. It is left on the child to use when processing the child. MathCAT could have lifted the intent to get the property prefix:infix and then had to decide which one to use (a case that happens with Deyan's intent=":infix:postfix". This examples differs from Deyan's example because there is a potential difference due to parent/child relationships.

Note: intent=":infix" on the mo is ignored/is not relevant for a leaf element.

[brucemiller]

@NSoiffer Deep in #446, you had a nice rationalization for why outer properties should override inner ones: #446 (comment); basically that outer intent generally overrides inner intent anyway. This shouldn't mean that inner properties are always ignored, just that outer ones dominate. Presumably the inner intent and properties may make sense on an element out of context, but in the context of its parent, those intents get overridden by the parent's intent. That makes good sense to me.

So, in @dginev example

<mrow intent="$op:prefix($x,$y)">
  <mi arg="x">x</mi>
  <mo arg="op" intent=":infix:postfix">+</mo>
  <mi arg="y">y</mi>
</mrow>

the effective intent on the mrow is intent="+:prefix:infix:postfix(x,y)", and we follow your guidance that the outer or leading properties dominate, so the "+" should be treated as prefix.

If it's the "+" vs "plus" that bothers you, think of the intent as 'intent="plus:prefix:infix:postfix(x,y)", where the "+" has been processed to speech "plus". But that starts to slide into issue #448.

[davidcarlisle]

the effective intent on the mrow is intent="+:prefix:infix:postfix(x,y)",

I am still not seeing why you put the outer property first ,ie in the middle?

<mo arg="op" intent=":infix:postfix">+</mo> is

<mo arg="op" intent="+:infix:postfix">+</mo> or <mo arg="op" intent="plus:infix:postfix">+</mo>

so $op:prefix must surely apply prefix to that so

<mo arg="op" intent="[plus:infix:postfix]:prefix">+</mo> or
<mo arg="op" intent="_:prefix(plus:infix:postfix)">+</mo>

But I suspect it's different ways to say same thing as

and we follow your guidance that the outer or leading properties dominate, so the "+" should be treated as prefix.

I get the same conclusion by a different route.

If it's the "+" vs "plus" that bothers you, think of the intent as 'intent="plus:...where the "+" has been processed to speech "plus".

Yes that's why I used plus in my example (as +:infix is currently invalid)

[brucemiller]

Ah, I see: So the argument "outer dominates inner" seems convincing enough, perhaps. But for listed properties of a single element, does first or last dominate? I'm not sure I have a preference at this point, but you're right: the choice affects how we assemble an "effective" ordered property list.

[NSoiffer]

It seems Bruce feel order matters and that in fact, the order is left-to-right. Is that correct?

I don't think David has indicated an order preference since he is using brackets to group the substitution. However given the statement "it's different ways to say same thing", maybe he is?

Also, for this example, there is an assumption that "plus" is what is said for +. If it is the first argument of an mrow, then it is likely "positive", not "plus". If could be other things also ("limit from above", ...). But you can't know that without understanding that it's position has changed due to the prefix or what the surrounding context is.

I'm seeing a lot of complication from the way at least David thinks this should be used (content being sucked into intent). Maybe I'm missing something. Can one of you write out an algorithm how you think this should work?

[brucemiller]

There are two parts to the property ordering question: (1) Does outer or inner properties take precedence? (2) for multiple properties on a single element, does the first or last take precedence?

To my tastes, either outer+first or inner+last are more consistent with tree traversal, so those combinations feel natural to me and are easiest to think about. At least for me. For either of those cases, when you synthesize the effective properties for the "+", you get :prefix :infix :postfix. And the, for outer+first, :prefix would win; for inner+last, :postfix would win.

As for the second question, the reference $op in the intent expression essentially plucks it out of the mathml tree, so it no longer has any of that context. If it has one at all it would be the context of the intent expression. You might say "positive" if the effective intent ended up being +:prefix, but it's hard to imagine that you'd expect "a positive b" from

<mrow intent="$op:infix($a,$b)"`>
  <mo arg="op">+</mo>
  <mi arg="a">a</mi>
  <mo>,</mo>
  <mi arg="b">b</mi>
</mrow>

(or something equally perverse).

[NSoiffer]

It would definitely help to have a real life example. Here's something closer, but still made up (although probably something similar would be in arXiv)

<msup intent="$op:prefix($a)"`>
  <mi arg="a">a</mi>
  <mo arg="op">+</mo>
</msup>

Here you might expect "positive a". But if I were to write it, I wouldn't reference the + and just use "positive". So this isn't really good example. Maybe someone has one where referencing the op and rearranging its position is sensible and not forced.

Under what I propose should happen, this would get rewritten as

<mrow >
  <mo>+</mo>
  <mi>a</mi>
</mrow>

I'm still not clear on what David is proposing other than you end up with intent and not MathML and then the intent is spoken somehow. Maybe after a hearty English dinner, he'll be ready to tackle this :-)

[davidcarlisle]

@NSoiffer

I'm seeing a lot of complication from the way at least David thinks this should be used (content being sucked into intent).
...
Under what I propose should happen, this would get rewritten as <mrow....

despite seeing it in action in mathcat traces I still find the description of intent as generated mathml somewhat unintuitive.

If we view intent as "generating speech" I see $argref as a reference to a speech string not an element structure
so intent="$op($a)" means

the speech string from op, "of", the speech string from a

If a referenced arg has an intent that is used for its speech string
If it does not have an intent, read its content.

One way to formulate the above is to say the default intent of an mo is its content (which does not mean that default has to be legal as an explicit value) <mo>:</mo> can not literally have intent=":"

---

If we do describe intent in terms of constructing an intent-free mathml element tree, I certainly need to change my mental model. Where would this tree live, would that be the accessibility tree generated for this fragment of the document?

[NSoiffer]

Maybe I'm too stuck with the implementation vision I have, but I see intent as a key for how to speak a string, not what to speak unless it is a literal. So intent="absolute-value($x)" says "speak this the way you speak the concept absolute value. That might be "absolute value of ..." or "valore assoluto di ..." and include some bracketing words. It depends on context. If it were intent="$op($x)" where $op is <mi>absolute value</mi> I would consider the content to be a literal and not translate it. So I do see a significant difference between your vision and my vision of how reference act (since this is a functional form, I'd form a functional template with the mi followed by <mo>&#x2061;</mo>).

Part of the problem we have is that the examples so far for $op are artificial and so not helpful for understanding what to do. In the examples so far (+, =, etc), it doesn't make sense to pull them up into the intent because they speak their symbol. And as discussed during a call, a flat mrow representing a+|2x| = y is better remediated by adding mrows than by adding args and nested functional forms. Or maybe even better with Deyan's hack (yes, I consider it a hack) of using _ for the function head and listing out the arguments with "absolute-value" being the only nested function.

Do we have an example where $op is natural and not artificial? That would really help clarify what should happen.

[davidcarlisle]

Maybe I'm too stuck with the implementation vision I have, but I see intent as a key for how to speak a string, not what to speak unless it is a literal. So intent="absolute-value($x)" says "speak this the way you speak the concept absolute value. That might be "absolute value of ..." or "valore assoluto di ..." and include some bracketing words. It depends on context. If it were intent="$op($x)" where $op is <mi>absolute value</mi> I would consider the content to be a literal and not translate it.

I'd agree with all that, but I (could) phrase it as saying $op references the speech string you generate from the possibly implied intent on the referenced element.

if the referenced element is a token element with no intent such as <mi>absolute value</mi> the implied intent is a non-core literal eqivalent to its content so <mi intent="_absolute_value">absolute value</mi>

So I do see a significant difference between your vision and my vision of how reference act (since this is a functional form, I'd form a functional template with the mi followed by <mo>&#x2061;</mo>).

by "template" here you mean a fragment of mathml?

Part of the problem we have is that the examples so far for $op are artificial and so not helpful for understanding what to do. In the examples so far (+, =, etc), it doesn't make sense to pull them up into the intent because they speak their symbol. And as discussed during a call, a flat mrow representing a+|2x| = y is better remediated by adding mrows than by adding args and nested functional forms. Or maybe even better with Deyan's hack (yes, I consider it a hack) of using _ for the function head and listing out the arguments with "absolute-value" being the only nested function.

Do we have an example where $op is natural and not artificial? That would really help clarify what should happen.

${}^Tx$ for prefix transpose notation? (I'd have used $x^T$ but mathcat needs no intent for that)

<math display='block'>
 <mmultiscripts intent='$op($arg)'>
  <mi arg='arg'>x</mi>
  <mprescripts/>
  <mrow/>
  <mi arg='op'
      intent='transpose'
      mathvariant='normal'>T</mi>
 </mmultiscripts>
</math>

see https://mathml-refresh.github.io/intent-lists/intent4.html#IDxtransposepre-sup

[davidcarlisle]

@NSoiffer mathcat has several ways of reading <mtable which it chooses based on context (I know you know this:-),

by column:
the 2 by 2 determinant;
column 1; eigh;
column 2; b;
column 1; c;
column 2; d;

by line:
2 lines,
line 1; eigh, b;
line 2; c, d;

by row:
the 1 by 2 row matrix;
line 1; eigh, b;
line 2; x, y;

by case:
2 cases,
case 1; x plus y, is equal to, 2;
case 2; x minus y, is equal to, 0;

by equation
2 equations,
equation 1; 2 x, is equal to, 1;
equation 2; y, is greater than, x minus 3;

I don't think it currently works but I'd like to be able to use a table style property hint like the fixity hints so for exmple

<mtable intent=":cases">

would give the "by cases" reading above even if mathcat would not have picked that from the current context.

If I then use intent="foo($m:cases)" ... <mtable arg="m">...

I currently think I mean "slot in the cases-reading of the table at this point"

I think you are saying you are not generating string but a shadow mathml fragment and it should mean drop in some mathml constructed with <mtext>case 1</mtext>... <mtext>case 2</mtext> that gives the desired reading?

[brucemiller]

The whole ordering issue is a bit of a corner case, so it's hard to come up with compelling cases; but @davidcarlisle 's cases example is a good one.

I think of a reference as both a reference to the element it points to, and which will be replaced by the speech generated by formatting that element into speech, in the context of any properties assigned to the reference. Of course, any properties given in the referenced element will also play a role in the generated speech, but @davidcarlisle example shows why the outer properties should dominate the inner ones.

Although it may be perfectly reasonable to implement intent by rewriting the MathML tree (although I'd expect lots of flat mrows from that process), I find it rather confusing as a way to drive the specification.

[dginev]

My implementation intuition for the small example would be:

<mrow intent="$op:prefix($x,$y)">
  <mi arg="x">x</mi>
  <mo arg="op" intent=":infix:postfix">+</mo>
  <mi arg="y">y</mi>
</mrow>

1. process $op:prefix($x,$y), building an AST (abstract syntax tree) realized as some specific data structure favored by implementer in a given programming language.

2. process reference nodes

   • perform 3 descendant lookups obtaining the nodes for $op, $x and $y
   • compute the intents for the obtained nodes (optional: in-context)
     • trivial for <mi> with content x and y
     • <mo arg="op"... has an empty intent with two properties, so the legacy computation of the intent for a mo node with content + needs to be performed, while still keeping the provided properties.
       • this may depend on local context (e.g. atom/prefix/infix/n-ary infix/postfix/other position of the mo in the mrow argument list - it is infix in this example). Possibly it infers the Core concept plus here.
       • for AT that does not perform inference (or has no applicable rules), the self-voicing Unicode approach remains as baseline, i.e. _plus-sign (for U+002B) - or better, if the AT has its own "self-voicing" customization.
   • fill in the argument holes with the assembled intents, then apply current level metadata
     • for the sake of example, let's assume an AT that recognizes the infix plus notation, infers plus from Core
     • This yields plus:infix:postfix:prefix(x,y)
     • I mean "fill" in the abstract pseudo-code sense. A concrete implementation should be welcome to continue to both build a new AST, or modify in place the same AST it constructed initially. It can also decide if references are replaced by their expansion, or if the slot representing each reference will also contain its expanded form (and/or a pointer to a subordinate intent-AST for the descendant node being referenced and/or...)
     • similarly, ATs that want to use global context may pause after all referenced intent-ASTs are connected, and perform a global scan step, inferring additional metadata for nodes in legacy mode (or wherever it is useful, really).

3. process property metadata

   • After references are dealt with (and only then), it seems appropriate to deal with conflicting property information
   • I agree with Bruce that the outer context should dominate the inner, as the higher-level annotations tend to be "better informed" in a MathML-generation context. (properties closer to the leaf nodes may be leftover due to reusing the same TeX macro in an overridden context).
   • So AT here would first add :infix fixity to plus, then see :postfix, and override to :postfix fixity. Then see :prefix and override to :prefix fixity. Then stop.

Leaving us with the final "fully expanded" intent: plus:prefix(x,y)

5. Next steps: localization, and generating the ultimate narration string, can follow from here.

To try and summarize this intuition, it would be:
"Outer properties trump inner properties.
In horizontal property lists, the last (rightmost) conflicting value wins."

[brucemiller]

I pretty much agree with @dginev outline with one or two exceptions, provided you clarify that the AST represents the expanded intent (not the speech, which confused me on the first couple of readings).

I think that if outer properties win (which they should), then the first horizontal property should win. This is not from any compelling usecase perspective, but to have a conceptually simple principle that "first property wins". If you do a tree traversal, when you get to the content of $op, you will have seen the properties :prefix, :infix, :postfix in that order. So, :prefix wins. If there hadn't been a :prefix on the outer mrow, then the list would have been :infix, :postfix and so I'd expect that :infix wins.

The "may depend on local context" seems counter-intuitive, since a reference effectively plucks the element out of it's existing context and puts it into a new one (Hence @NSoiffer perspective that this is a rearrangement of the MathML tree).
I'm a bit confused about how you've described the processing of $op. @davidcarlisle cases example shows that you should process $op in the context of having :cases property

[dginev]

I pretty much agree with @dginev outline with one or two exceptions, provided you clarify that the AST represents the expanded intent (not the speech, which confused me on the first couple of readings).

Right, apologies if that was left unclear.

I think that if outer properties win (which they should), then the first horizontal property should win. This is not from any compelling usecase perspective, but to have a conceptually simple principle that "first property wins". If you do a tree traversal, when you get to the content of $op, you will have seen the properties :prefix, :infix, :postfix in that order. So, :prefix wins. If there hadn't been a :prefix on the outer mrow, then the list would have been :infix, :postfix and so I'd expect that :infix wins.

I see. I was viewing them as "metadata assignments" with implementer eyes. And using the intuition of "the last assignment wins". In the steps above the outer property is attached after the inner one is assembled, so it is "last", hence it wins. If the horizontal properties follow the left-to-right convention, the rightmost is last. For plus:infix:postfix I'd expect :postfix to win.

But I can get used to either. I can even get used to that being undefined behavior / an error.

The "may depend on local context" seems counter-intuitive, since a reference effectively plucks the element out of it's existing context and puts it into a new one (Hence @NSoiffer perspective that this is a rearrangement of the MathML tree).

Sure, but there is no "plucking out", there is a partial parallel AST for intent. The Presentation tree is still there, and still completely sufficient for a sighted reader to understand the notations. Hence its local context is also meaningful. Note that the example is in legacy mode, since <mo intent="plus">+</mo> has certainty this is Core plus, while <mo>+</mo> does not, and could be anything (particularly in documents from corpora such as arXiv).

I'm a bit confused about how you've described the processing of $op. @davidcarlisle cases example shows that you should process $op in the context of having :cases property

Right, the :cases will get carried along until the speech generation starts, which happens after the steps outlined in my comment, and beyond the scope of what my comment covers. David C's comment assumes that speech generation happens before references are filled in, which I think is way too early - and precludes any use of ancestor context.

[davidcarlisle]

David C's comment assumes that speech generation happens before references are filled in, which I think is way too early - and precludes any use of ancestor context.

that wasn't my intention: I think you should be able to expand first either expanding as a literal (possiby extended) intent attribute or,as you say an AST of the function tree from the intent.

What I don't still really understand despite watching mathcat in action is viewing intent as a rewrite of the mathml tree.

[brucemiller]

Re: I was viewing them as "metadata assignments"

That's exactly where I started out! Until various examples from @NSoiffer and @davidcarlisle convinced me that outer should dominate. And then, looking at the tree made me prefer the first on horizontal lists, otherwise it's too back-and-forthy.

[dginev]

Re: I was viewing them as "metadata assignments"

That's exactly where I started out!

A shared intuition! Too rare to miss out on :>

Until various examples from @NSoiffer and @davidcarlisle convinced me that outer should dominate.

Depending on how the reference expansion is implemented, the outer dominating overlaps the "last assignment" justification - I tried to walk through one way of realizing that.

And then, looking at the tree made me prefer the first on horizontal lists, otherwise it's too back-and-forthy.

No back-and-forthiness in my walk-through at least. Just keep assigning, whatever gets done last, sticks.

[brucemiller]

There's "virtual" plucking out, and the implementation could be in terms of a "virtual" AST, so the conversion to speech of could already be done at the time $op is looked for (in the context of :prefix. (I'd avoid building in too many implementation assumptions).

You say "local context is also meaningful", but that seems wrong to me. Can you think of a use case where that would be so?

[brucemiller]

No back-and-forthiness in my walk-through at least. Just keep assigning, whatever gets done last, sticks.

But you've built in the assumption of the order of traversal, which seems an implementation detail. My intuition was from looking at the tree. Or, since the topic is accessibility, if the tree was being read out literally. And in those cases it's very back-and-forthy.

[dginev]

But you've built in the assumption of the order of traversal, which seems an implementation detail

Ah, well, the current issue is actually raising a valid point - once we allow multiple properties on the same intent value, the order of traversal and the exact steps in which references get expanded and filled in within the outer intent expression - are no longer an implementation detail, but need to be specified. Even if we specify them as "conflicting properties are an error, don't do that"

[brucemiller]

Right, we do have to specify the precedence of properties, although I'm not so sure we have to specify the exact steps. I'm just saying we should make that choice based on what will be less confusing to users, rather than implementation assumptions.

[dginev]

Treating "conflicting properties" as "errors" would be least confusing to users, since it will reduce the cognitive load of the spec.

[brucemiller]

Ah, you mean that "It didn't do what I expected" is better than "It didn't do what I expected"? Yeah.

("That was a Joke, son! I say, That was a Joke!" Foghorn Leghorn)

[dginev]

Re: an example for "local context is meaningful"

From examples I have looked at, this comes into play where we mix-and-match nodes with intent, and nodes without (i.e. legacy mode nodes). I am sure all of us know plenty of examples where local context is meaningful in the "full grammatical parsing" task over math syntax. The legacy mode allows for that task to be posed, and as Neil has mentioned - applications such as MathPlayer and SRE have done a lot of heuristic guessing to infer the underlying concepts for MathML 3. I expect MathCAT and SRE will continue to do so in legacy nodes for MathML 4.

Having said that, and observing that <mo arg="op">+</mo> needs to be processed in legacy mode, it's just a matter of thinking of the various notations where the plus character is used, which do not denote the addition operation.

Say positive charge postfix and superfix notations:

<mrow intent="$op:prefix($arg)">
  <mn arg="arg">1</mn>
  <mo arg="op">+</mo>
</mrow>

<msup intent="$op:prefix($arg)">
  <mi arg="arg">K</mi>
  <mo arg="op">+</mo>
</msup>

If a hypothetical ChemistCAT AT was designed to expect chemistry, it is easy to imagine it can infer intent="positive-charge" in a similar way that MathCat will infer intent="plus" for legacy node processing.

Edit: There is an unrelated curveball hidden in my examples here, which touches on language generation. Some (but not all) concepts have adverbial forms, and a :prefix may be best resolved by switching to them. Also, this is in reaction to the annotator requesting :prefix, it is almost certainly not the "usual"/"preferred" reading. Here that leads to positvely charged one or positively charged potassium. Of course this may end up too roundabout in practice, where "hacks" become more convenient with intent="_(_positively_charged, $arg)". I added :prefix to stay loosely compatible with the previous example I was working on in the issue, apologies if this added confusion.

[brucemiller]

So, legacy here means MathML without intent? In which case, sure the AT should consider the context. But is it a matter of analyzing the mrow and the contexts of the pieces within it, or is it a matter of ending up at the mo and then backing up to look at its context? I would have thought the former, but...

I don't quite see what's expected in this mix-and-match situation. You have an apparently postfix "+", but you say to treat it as prefix. And then you have a hypothetical ChemistCAT which presumably has a rule for postfix "+", which says positive charge $something? If that's the case, I think you should have left off the mrows intent altogether, rather than trying to make the "+" both prefix and postfix at the same time. Unconvinced :>