Allow custom GDScript annotations which can be read at runtime

[Wavesonics]

Describe the project you are working on:
Multiplayer game

Describe the problem or limitation you are having in your project:
Automaticly serializing custom data classes

Describe the feature / enhancement and how it helps to overcome the problem or limitation:
In 4.0 GDScript now has annotations. What I propose is two fold:

1. User's can declare their own custom annotations much like how Java allows you to (override a built in Annotation class)
2. User's can query a classes properties and methods for their associated annotations at runtime

Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams:

extends Reference
class_name MyDataClass

@Serialize(name="json_foo")
var foo: int

@Serialize
var bar: String

var x: Boolean

Then I can wrote some code to introspect MyDataClass and find it's properties that it wants serialized, and produce something like:

{
    "json_foo": 0,
    "bar": "test"
}

If this enhancement will not be used often, can it be worked around with a few lines of script?:
It can be worked around, it will just be less flexible and elegant.

Is there a reason why this should be core and not an add-on in the asset library?:
At least this particular approach would not be possible without support from GDScript it's self. But other approaches are possible, such as just listing all of the properties using the current inspection methods, and serializing all of them.

[KoBeWi]

You could also make your variables exported and save the data as .tres. It's a text format too.

[jonbonazza]

You could also make your variables exported and save the data as .tres. It's a text format too.

AFAIK, this doesn't work for embedded classes.

[jordo]

+1 for this. So I'm basically working on something similar. exposing an annotation property would be nice. My work around is to use some naming prefix convention and yes iterate the object properties. Example:

@export net_serialized_counter : int   // gets serialized and synced
@export net_serialized_state : int  // gets serialized and synced
@export particles : Node // not serialized or synced

Then a net synchronizer just iterates the properties and checks for that prefix. But it's not ideal. I'd prefer not to rely on a naming convention here, as a custom annotation would be much less hacky and easier to use.

[vnen]

My main question about this is how do you expect to gather this information? Can you mock up a sample code with the functions you would call to retrieve the annotations?

I also want to make annotations explicitly registered, so a typo won't break the functionality without the user noticing.

[AndreaCatania]

Define

To register it would be nice an API like this:

@tool

func _ready():
    Engine.add_annotation("custom_annotation")

C++ version:

void register_custom_module() {
    Engine::get_singleton()->add_annotation("custom_annotation")
}

Fetch

I think that a good way to retrieve the annotations would the following:

GDscript version:

# Returns the scripts path 
var annotated_scripts = Engine.get_annotated_scripts("custom_annotation")

for script in annotated_scripts:
    print("Script: ", script)

    var annotated_functions = Engine.get_annotated_functions("custom_annotation", script)
    var annotated_variables = Engine.get_annotated_variables("custom_annotation", script)

    func f in annotated_functions:
        print("-- Function: ", f)
        var arguments = Engine.get_annotated_functions_arguments("custom_annotation", script, f)
        for key in arguments.key():
                print("---- ", key, " = ", arguments[key])

    func v in annotated_variables:
        print("-- Variable: ", v)
        var arguments = Engine.get_annotated_variables_arguments("custom_annotation", script, v)
        for key in arguments.key():
                print("---- Annotation: ", key, " = ", arguments[key])

C++ version:

struct ScriptAnnotationData {
    OHashMap<StringName, Dictionary> function_data;
    OHashMap<StringName, Dictionary> variable_data;
};

const OAHashMap<String, ScriptAnnotationData> &annotated_scripts = Engine::get_singleton()->get_annotated_scripts("custom_annotation")
// Returns the stored annotations for each script. The key is the script path.
[.....]

Usage

This kind of syntax allows the annotation to contains any kind of data. So that it's possible to define dynamic annotations; allowing:

@Serialize(name="json_foo")
var foo: int

@Serialize
var bar: String

@Serialize(max_length=20)
var long_string: String

@Serialize(networked=true)
var long_string_2: String

@Serialize(networked=false, max_length=20, name="json_foo")
var long_string_3: String

[AndreaCatania]

We also need a way to take the script level annotations like @tool. This function var annotated_scripts = Engine.get_annotated_scripts("tool_like") still seems valid. Though, we need a way to take the arguments.

[jordo]

+1 for this. Like @AndreaCatania we have a client-predicted, server-auth, (rollback re-sim) network module.

Right now our net 'state' is explicity defined under list of exported properties of our "NetworkGameObject"s. GameData, Player1, and Player2 in below screenshot:

We inspect this list at runtime to serialize the game's state, compare, and do any rollback and resims.. The downside to using just this list we can't add any other properties to these objects without them getting serialized along with whats currently exported.

Likewise some properties we will want to predict client-side, and some properties we will want to render lag and show/draw in different timestream. And even like the above comment by @AndreaCatania perhaps serialize in different ways, etc.

From the c++/native side, it would be great to access any custom annotations at runtime say in here somewhere:

Right now we rely and default storage attribute flags for processing, but access to custom annotations here in the object runtime would be quite powerful and flexible.

[AndreaCatania]

OFFTOPIC: @jordo this may interest you: godotengine/godot#37200

[jordo]

But ya, for sure +1 for what @AndreaCatania proposing below, this is great:

@Serialize(networked=false, max_length=20, name="json_foo")

OFFTOPIC: @AndreaCatania yes I've followed what you've been doing here, godotengine/godot#37200 It's quite an effort! But we needed more explicit control and a more data-orientated approached with our module. Simpler and not tied into as many different godot systems.

[Killfrra]

+1 for this proposal. Because that's how I solve the simular problem now. Perhaps this suggestion will allow me to get rid of the initialization of var_attrs, and in combination with the #2837, and the explicit call to sync_set and setters with emit_stat.
P.S. I'm sorry for the large amount of code. It's here to demonstrate what we can get rid of

# CHILD CLASS ON SERVER

var mana
var mana_max: float = 418.0
var mana_regen: float = 8.0
var ability_power := 9.0
var ability_haste := 0.0
var level_int := 1
var level_frac
var exp_to_next_level := 280
var gold := 500.0
var creep_score := 0
var kills := 0
var deaths := 0
var assists := 0

func generate_variable_attributes():
    .generate_variable_attributes()
    var_attrs["mana"] = Attrs.UNRELIABLE | Attrs.VISIBLE_TO_EVERYONE
    var_attrs["mana_max"] = Attrs.VISIBLE_TO_EVERYONE
    var_attrs["mana_regen"] = Attrs.VISIBLE_TO_OWNER_AND_SPEC
    var_attrs["ability_power"] = Attrs.VISIBLE_TO_EVERYONE
    var_attrs["ability_haste"] = Attrs.VISIBLE_TO_EVERYONE
    var_attrs["strike_chance"] = Attrs.VISIBLE_TO_EVERYONE
    var_attrs["level_int"] = Attrs.VISIBLE_TO_EVERYONE
    var_attrs["level_frac"] = Attrs.VISIBLE_TO_OWNER_AND_SPEC
    var_attrs["gold"] = Attrs.VISIBLE_TO_OWNER_AND_SPEC
    var_attrs["kills"] = Attrs.VISIBLE_TO_EVERYONE
    var_attrs["deaths"] = Attrs.VISIBLE_TO_EVERYONE
    var_attrs["assists"] = Attrs.VISIBLE_TO_EVERYONE
    var_attrs["creep_score"] = Attrs.VISIBLE_TO_EVERYONE

func level_up_stats():
    # damagable
    var health_change = 92 * (0.65 + 0.035 * level_int)
    sync_set("health_max", health_max + health_change)
    sync_set("health_regen", health_regen + 0.06 * (0.65 + 0.035 * level_int))
    sync_set("health", min(health + health_change, health_max))
    
    sync_set("armor", armor + 3.5 * (0.65 + 0.035 * level_int))
    sync_set("mr", mr + 0.5 * (0.65 + 0.035 * level_int))
    # autoattacking
    sync_set("attack_damage", attack_damage + 3 * (0.65 + 0.035 * level_int))
    # champion
    var mana_change = 92 * (0.65 + 0.035 * level_int)
    sync_set("mana_max", mana_max + 25 * (0.65 + 0.035 * level_int))
    sync_set("mana_regen", mana_regen + 0.08 * (0.65 + 0.035 * level_int))
    sync_set("mana", min(mana + mana_change, mana_max))

# BASE CLASS ON SERVER

enum Attrs {
    RELIABLE = 0, # default
    UNRELIABLE = 1,
    CONST = 2,
    VISIBLE_TO_EVERYONE = 4,
    VISIBLE_TO_TEAM_AND_SPEC = 8,
    VISIBLE_TO_OWNER_AND_SPEC = 16,
}

var var_attrs := {}
func generate_variable_attributes():
    pass

func _init():
    generate_variable_attributes()

func sync_set(key, value = null):
    if value == null:
        value = get(key)
    else:
        set(key, value)

    var attrs = var_attrs.get(key) if var_attrs else null
    if attrs:
        
        var func_name = "rpc_unreliable_id" if attrs | Attrs.UNRELIABLE else "rpc_id"
        
        if attrs & Attrs.VISIBLE_TO_EVERYONE:
            for client in Game.clients.values():
                if should_sync_to_client(client):
                    avatar.call(func_name, client.id, "set_remote", key, value)
        
        else:
            
            if attrs & Attrs.VISIBLE_TO_TEAM_AND_SPEC:
                for player in Game.lists[team].values():
                    if should_sync_to_client(player):
                        avatar.call(func_name, player.id, "set_remote", key, value)
            elif attrs & Attrs.VISIBLE_TO_OWNER_AND_SPEC:
                avatar.call(func_name, id, "set_remote", key, value)
            
            for spectator in Game.spectators.values():
                if should_sync_to_client(spectator):
                    avatar.call(func_name, spectator.id, "set_remote", key, value)

        return true
    return false

func should_sync_to_client(client):
    return client.team == team || client.team == Types.Team.Spectators || seen_by_teams[client.team]

# BASE CLASS ON CLIENT (avatar)

puppetsync func set_remote(key, value):
    self[key] = value
    
# CHILD CLASS ON CLIENT

var health := 0.0 setget set_health
var health_max := 0.0 setget set_health_max
var health_regen := 0.0 setget set_health_regen
var armor := 0.0 setget set_armor
var mr := 0.0 setget set_mr

func set_health(to):
    emit_stat("health", to)
    health = to

func set_health_max(to):
    emit_stat("health_max", to)
    health_max = to

func set_health_regen(to):
    emit_stat("health_regen", to)
    health_regen = to
    
func set_armor(to):
    emit_stat("armor", to)
    armor = to

func set_mr(to):
    emit_stat("mr", to)
    mr = to

[sairam4123]

Define

To register it would be nice an API like this:

@tool

func _ready():
    Engine.add_annotation("custom_annotation")

C++ version:

void register_custom_module() {
    Engine::get_singleton()->add_annotation("custom_annotation")
}

Define

We could use annotations for defining annotations:

# Console.gd
var commands = {}


@annotation
func register_command(function: Callable):
    commands[function.name] = function

This is similar to decorators in Python, a annotation could possibly call back the function it got annotated with. The function will have one argument, function, extra arguments could be added, extra arguments must be passed using the decorator itself.
Then we can split up annotations to functions and variables, because who wants function annotations to be used in variables or vice versa.

So the above code could be written this way:

var commands = {}


@function_annotation
func register_command(function: Callable):
    commands[function.name] = function

I have a mockup for variables too.

# Settings.gd
var settings = {}

@variable_annotation
func setting(name, type, value):
    settings[name] = {name, type, value}

As we don't have a class for variables we'd be passing name, type and the value, if type is not defined, we'd pass inferred type.
And I think, instead of having to pass 3 arguments, we could compile them into one class named Property, this could be discussed further in a separate proposal.

Fetch

As for fetching functions, we can possibly callback the function that has been annotated as annotation, with the function or variable that got annotated in the first argument and the arguments that got passed with it.
We can also use Engine.get_annotated_scripts to get annotated script as mentioned in #1316 (comment)

@function_annotation and @variable_annotation

As said previously, this comment introduces two new annotations, function and variable annotations to declare custom annotations.

These are built in annotations and will work like other built-in annotation.

Usage

Variable/Property Annotation

@Settings.setting()
var speed: int = 50

@Settings.setting()
var start_money := 10.0

@Settings.setting() # I'm yet to decide how we should override variable names that get passed.
var this_name_will_be_overriden := "Hi!"

@Settings.setting("range", 10, 11, 1.0) # Extra arguments
var extra_args = 10 # will be inferred at runtime

Function Annotation

@Console.command()
func this_is_a_console_command(arg: int, arg_2: int) -> bool:
    return arg > arg_2

@Console.command("test")
func set_start_money(money: int) -> bool:
    start_money = money
    return true

This comment is kinda of a new proposal but still I'm commenting it here.

[JuerGenie]

I like @sairam4123 's proposal, decorators yes!

[Calinou]

@JuerGenie Please don't bump issues without contributing significant new information. Use the 👍 reaction button on the first post instead.