Node Metadata is not preserved by Yarn Editor

[alakoring]

What is the current behavior?

Yarn Editor doesn’t preserve arbitrary metadata

Please provide the steps to reproduce, and if possible a minimal demo of the problem:

I edited a .yarn file to add my own metadata.
I opened this in the Yarn Editor (on macOS).
Saved the file.
The custom metadata was no longer in the file.

What is the expected behavior?

The editor should only change the parts it cares about. Additional lines of metadata should be saved.

Please tell us about your environment:

• Yarn Editor Version: Version 0.4.44 (0.4.44)
• Unity Version: N/A

Other information

It’s possible this is a feature request, but either way I would like to be able to use the tool.

[desplesda]

Thank you for the issue report! Transferring this to the repo for the editor.

[blurymind]

@alakoring what did you use to add metadata to the file? Also can you attach an example file with metadata on it?

[blurymind]

I am also curious why do you expect that yarn is supposed to preserve metadata you attach with external software? To yarn, this is just a text file. You save to it - you completely overwrite the file. I don't think any other software out there preserves the metadata. If however you put your metadata in another file with the same name, but different extension - yarn will not overwrite that.

If you edit the file with the metadata on it with something like notepad - will that keep your metadata?

This so far looks to me like an unreasonable expectation to have to any software.

There might be a way to preserve it, but remember we need to also be able to read and write it in the first place.

Can you find me a single library here that can do that?
https://www.npmjs.com/

I am sort of curious about this instead of outright dismissing and closing the issue, because I think the ability to store metadata can be beneficial to yarn itself too.
It could let us store more data about the file without changing the format's spec

[alakoring]

Thanks for the quick response, I didn’t realize there was a separate repo.

I used a text editor (TextEdit on macOS), exactly because I thought this was a text file, and that the Yarn format explicitly allowed metadata.

The header contains metadata for the node, and can have as many pieces of node metadata as you wish. Metadata is created as key-value pairs. Each metadata piece has the same layout:

So I added a line of metadata, in key-value format like the ones I saw already.

As to the use case: I am debating using Yarn for a procedural narrative. So I want to be able to add additional data (not just tags) that would let me pick appropriate start nodes. Yarn by itself probably lets me do that, but my narrative designer values the ability to see the nodes visually (i.e. using Yarn Editor and not TextEdit). I don’t want metadata (which is legal Yarn as I read it) to be stripped if someone uses Yarn Editor and saves a change.

My expectation is that Yarn Editor already reads in all the metadata from the header. It uses some of it to draw nodes. It lets you edit tags. It could easily simply show all other metadata and not allow it to be edited, keep it (to be written later) and not show it, or even add a line (like Title or Tag) to edit the text value.

So basically, I’d like you to keep using the library you already use to read the key-value pairs, and just retain all the ones you must currently ignore as not special.

Archive.zip

[desplesda]

Currently the Yarn Compiler doesn’t do anything with any headers besides title and tags, and doesn’t provide a way to access them, so I can see why the Yarn Editor’s current behaviour is what it is.

That said, I agree that Yarn Editor should not discard user data that’s represented in a valid way but that it isn’t expecting to see.

[alakoring]

Sounds like I should add a feature request to Yarn then—file format document implied you could access it (otherwise why say you can have more). I suspect that’s easier for me to add if necessary, but that would be a ways down the road.

[blurymind]

So when you say metadata, I did not expect you to mean actual text in the file. Metadata implies extra data that is attached to the file (such as author, modification date, etc etc). Media files tend to have that and require extra libraries to author it.

When you edit a yarn file with a text editor to add extra data - it is true - the parser will ignore your extra data, but so will yarn when you do the loading/saving.
You load the file and since yarn doesnt know about the extra data you added, it doesnt load it in memory. So later on when you save, since its missing from memory - it doesnt get written.

In order to keep it, we need to make it a part of the file spec. The file itself could have an entry field called metadata or something, where it is expected to add extra info and yarn will consistently be finding it there.

If you want such a new feature supported, you need to at least share an example file and explain its benefits :)

I noticed you already shared a file, so next step is to agree on the file spec :)

[blurymind]

So looking at the example file- we cant just call the data "extra". Also you have it attached on a per node basis - which is not just a header.

Do you need to be able to attach extra data to each node or just the file header ?

Currently Yarn file loading is absolutely specific about the spec and expects the same fields
https://github.com/YarnSpinnerTool/YarnEditor/blob/master/src/js/classes/data.js#L103
File writing is simple - just writes what it has loaded in memory to a text file

[desplesda]

I think that the best behaviour would be to read in all headers for all nodes when the file is loaded, keep them in memory, and write them back out in modified when the file is saved?

[blurymind]

It's a bit risky to change that code, but we could give it a try and make it less specific :)
It would be amazing if yarn itself detects the extra fileds and even lets you add extra fields.

I am a bit curious though. Why not just use the tags field to add extra metadata?
@alakoring what made the available fields unusable for you and drove you to manually add new ones with external editors?
The tags field has no limitation on characters

Do you need to have some sort of a field where you can just dump any text and the available ones are somehow not equiped for that?
I would really like our users to stay inside yarn and not have to edit their files with external text editors

[alakoring]

Yeah, “metadata” can be used pretty loosely. I was going by my own informal usage in game design, and also “header contains metadata for the node” from the docs. Tags are metadata, but I use also other forms (conditions when a node could be selected as part of procedural generation, or how much of a delay before a node can occur again randomly).

My own needs match the documentation: “The header contains metadata for the node.”

I know there was discussion about adding per-file (text-based) metadata, and this could perhaps list any keys that should have editable fields at the top of a node. (This would let a user create the per-node metadata.) An interim version might be to require the user to use a text editor once, and any keys that were on any node would be implied as available as editable fields in Yarn Editor.)

Glancing at the C# code, it looks like a Node does have a _unknownFields it tracks, but doesn’t make public. (And this may be a protobuf artifact.) So “The header … can have as many pieces of node metadata as you wish” may not really be implemented. Obviously I think it should be. (Sadly one fix would be to correct the misleading documentation…)

In my example

===
title: ChoiceA
tags: 
colorID: 0
position: 1300,374
extra: Metadata
---
Empty Text

the key extra was just for testing. I would like to have the possibility of different keys.

[blurymind]

Thats one way I thought of sorting this out on yarn editor - adding an unknown fields object to to each node - to store in memory.
What I am kind of wondering is - can we make the interface of Yarn editor nicer and let you handle adding your own metadata without the need to use and external text editor.
So for example if you had a metadata editor inside yarn editor.

When you say metadata- what is that to your game engine? Is that a string, or an object that contains a bunch of variables?

With yarn editor being a javascript app, one thing that is absolutely possible is even adding
https://github.com/josdejong/jsoneditor
http://jsoneditoronline.org/
to it and allow editing the metadata object directly in yarn. That would however restrict the metadata to be a json structure.
Not sure if thats worth investing too much time in. Do many people add extra metadata?

[alakoring]

I had been viewing Yarn as a text format, and the key:value of the documentation had suggested a string value. Which would be fine for my potential use. Much as the Yarn Editor uses the node metadata to store coordinates as strings (as key:value).

(I can’t imagine using JSON without ending up with multiple lines of text, and that seems much more complicated. You’re no longer dealing with a line that starts with a key and has a value.)