Source position for attributes

[jrfaller]

Hi all,

First, thanks a lot for this excellent library!

I am working on an open-source tool that diffs tree-based languages such as HTML or XML (http://github.com/GumTreeDiff/gumtree), and I wanted to use JSoup to parse HTML and XML files since it is one of the very few parsers I know that has precise source position for the XML/HTML elements. However, I have remarked that this information is not present for the attributes, and it prevents me from fully diffing these files, as attribute name or value modifications are frequent.

Would it be possible to add source location information for attribute keys and values? Or do you have an idea for a workaround?

Thanks in advance.

Cheers!

[jurgenvinju]

I have a different use case for the same functionality. Also a fan of JSoup!

We're generating LSP servers for DSLs and some of these have an XML syntax. Check out usethesource/rascal-language-servers .

With accurate locations for each attribute and value we can more easily generate useful features like reference resolution, also inside string attributes that are parsed further.

[jhy]

Thanks for the feedback and info on your use cases. And sorry for the very late reply!

I think this is a great idea and look forward to adding it.

Question for you both (and for any other folks using the source ranges). Currently the implementation only provides the source range for explicitly inserted tags. Implicitly created tags (like e.g. head or body) will have a -1 start / end range. Are you using value for a specific reason? I am thinking that it would be better to give that the range of whichever tag triggered that node's creation.

[jrfaller]

Hi @jhy and thanks for your feedback!

I was not aware of the behavior you described, thanks for pointing it out.

In our use case, we use Jsoup as a black box to parse XML files, with all the tags already present (and where we absolutely do not want to have nodes not in the document). Ideally, we would need precise positions for all the syntactic elements inside the file (elements, attribute names, attribute values). From what I understand there are some cases where nodes are dynamically inserted by Jsoup that are not present in the text file being parsed? I think this is beyond the scope of my use case or the one of @jurgenvinju if this is the case.

I am not sure how to handle these elements though since they are not in the text file it could be nice to have a source range that conveys this information. However, for debugging purposes, it could be also useful to pinpoint the places responsible for the creation of the element. So I have mixed feelings about this one (IIUC)!

Cheers!

[jhy]

Thanks for the (fast!) reply. Definitely makes sense. Using the XML Parser, you won't have any implicitly created elements -- that's only for the HTML parser. So it's not directly applicable to your use case.

I think if there's a clean way to have the SourceRange be able to tell if it was explicit or implicit, we'll be in a good spot. And similarly for closing nodes.

[jrfaller]

Ha yes right we have this problem for closing nodes when someone uses the <foo/> syntax. So this would be nice to know whenever we hit an element that is not materialized in the textual syntax! Maybe this could be a property on the nodes like isImplicit(), avoiding from the beginning to search for source information for those nodes?

[jurgenvinju]

Hi! cool that we are making progress on this topic. Our use case is that of "high fidelity" parsing, so we want to introduce as little "noise" in the process as possible. The image of what we have in the AST model should be as close as possible to that what the user is looking at in their editor.

• additional nodes or attributes that are inserted into the trees to canonicalize the model are "noise" to us. So it's important they are marked as such, like @jrfaller indicated, or that we can chose not to include them at all.
• closing nodes that are not there: that's great! As long as the closing range (offset+length of the entire node) does not extend over into the range of the next node. Especially if final spaces or newlines of the contents of the node with a missing closing tag are attributed to the "right node" using the offset/length, we are very very happy.

[jhy]

Thanks, that aligns to what I was thinking too. I have refactored and improved how source ranges are tracked in #2056. It would be great if you could install a snapshot version of 1.17.1 and have a try and see if there are any gaps. (Not including attribute ranges yet though).

someone uses the <foo/> syntax

That will now include a source range and an end range that is the same and are equal; pos 0 through 6.

Maybe this could be a property on the nodes like isImplicit()

Yes, added that. The source range and/or the end source will be tracked() and isImplicit(). And the pos/endPos will be the same. I.e. it's a zero-width range for the start or for the end. In HTML that might be an implicitly inserted <head>. In XML, that might be implicitly closed tags. E.g. <foo><bar></foo>. Bar will be implicitly closed with the pos before </foo>.

Especially if final spaces or newlines of the contents of the node with a missing closing tag are attributed to the "right node" using the offset/length

<foo>\n - the \n is a TextNode and will have a range coming directly after and non-overlapping with the foo.

[jhy]

I have implemented a first draft of attribute tracking in master...attr-track

It's a bit of a different approach than the node tracking because attributes are built differently. They aren't tokens, but rather sub tokens of tag tokens. So the tracking has to be done internally.