feat(unmarshaler): add interface so that children nodes can define custom unmarshal behavior

[pkarakal]

This adds the Unmarshaler interface where structs have to implement the UnmarshalMapStructure method and then on decoder.Decode, this will trigger the custom unmarshal method. This will make sure that mapstructure is more in line with the rest of the marshaling libraries (like json and yaml)

[sagikazarmark]

@pkarakal thanks for the PR.

Before accepting it, I'd like to run through a couple questions (see comments) and a potential alternative solution:

Do you think this feature could be implemented as a decode hook?

I think it could be, in which case, it may be a better approach and may reduce complexity.

I've implemented something similar here a while ago: https://github.com/sagikazarmark/mapstructurex/blob/main/decode_hook_map_decoder.go

[sagikazarmark]

Nit-picking: can we brainstorm on this name a bit? It doesn't sit right with me right off the bat.

A couple ideas:

• UnmarshalMap (might be too generic)
• UnmarshalMapstructure (feels closer to the library name)

[Azmisov]

Let's go with UnmarshalMapstructure. It matches the naming in some other places in the code

[sagikazarmark]

If this PR gets accepted as it is, I think it might make sense to document the behavior on the interface.

(Technically, the interface doesn't have to be exported, but it's a good place to document the feature)

[sagikazarmark]

I wonder if we should add more sophisticated examples. Particularly for maps, named types, pointers, pointers of pointers, etc.

[pkarakal]

It can be turned into a decode hook for sure. What I mostly had in mind is having an interface that structs could implement to control the way they are decoded into a struct. This can definitely be controlled by a feature flag in the DecoderConfig.

I believe that it would a good feature to have for anyone trying to manipulate a parsed config without having to write a custom decode hook. JSON and YAML packages already have a custom Unmarshaler interfaces for such cases but say for example you want to support multiple config types (json, yaml, toml for example using viper), you'd have to basically implement the same function for each of those configs. Having mapstructure offer such a functionality would mean that one would have to do that just once, drastically simplifying the code.

Take for example the following use case: I have a program that can have multiple data stores (e.g. a database, a message broker) that implement a common interface. To create those data stores, I have a Config interface which has a method that instantiate them and can have different fields. What I want to have is a global config that takes an array of those configs and based on the key I provide parse the decode the element as the correct config type to then instantiate the store. (Basically turn the YAML into the following struct, where the Config name becomes the key)

scheduleStores:
  - storeName: rdb
    postgres:
      - host: "127.0.0.1"
        port: 5432
        user: "root"
        password: "pass"
        name: "postgres"
  - storeName: amqp
    rabbitmq:
      - host: "127.0.0.1"
        port: 5672
        protocol: "amqp"
        vhost: "/
        username: "admin"
        password: "admin"
        exchange: "default
        queue: "default"

type ScheduleStoreConfig struct {
	StoreName      string        `mapstructure:"storeName"`
	ScheduleConfig store.Configs `mapstructure:"-,omitempty"`
}

type Config interface {
	// Name returns the name of the store backend.
	Name() string
        // other methods
}

type PGConfig struct {
	Port         int    `mapstructure:"port"`
	DatabaseHost string `mapstructure:"host"`
	User         string `mapstructure:"user"`
	Password     string `mapstructure:"password"`
	DatabaseName string `mapstructure:"name"`
}

func (c *PGConfig) Name() string {
	return "postgres"
}

type RMQConfig struct {
	Port     uint64 `mapstructure:"port"`
	Protocol string `mapstructure:"protocol"`
	Host     string `mapstructure:"host"`
	VHost    string `mapstructure:"vhost"`
	Username string `mapstructure:"username"`
	Password string `mapstructure:"password"`
	Exchange string `mapstructure:"exchange"`
	Queue    string `mapstructure:"queue"`
}

func (c *RMQConfig) Name() string {
	return "rabbitmq"
}

On the PR comments, no strong feelings on the method name and I can definitely add documentation and more complex test cases. Just trying to see if what I'm describing is something you'd like in the library.

[Azmisov]

I'm in favor of how it is implemented in this PR, rather than as decode hook:

• It matches how JSON unmarshaling behaves, which is standard Go. Unmarshaling JSON is automatic without needing to configure hooks
• Old code which uses decode hooks can start using the new UnmarshalMapstructure interface immediately without needing to modify their hook code.
• Less complexity for end-users: No need to configure an extra hook. If we require a hook to be configured, there is a lot of hook documentation users have to read through and understand to know that's what they need. Everyone is already familiar with JSON unmarshaling, and I imagine almost all new users will prefer implementing the UnmarshalMapstructure interface and not even look at the hook interface.
• The code is less complex, essentially a 3 line change.
• This skips the decode hook execution path, which will be more performant; no reflect calls

Note also the discussion in #22. Seems 11 people were in favor of the original PR in mitchellh's repo, and 3 from this repo (@pkarakal, @pcfreak30, myself).

@sagikazarmark As its been 2.5 years since that original PR, I'd prefer an executive decision be made one way or the other, rather than let this project go stale. Either:

1. go forward with this PR
2. new PR with UnmarshallMapstructure, UnmarshallMapstructureDecoderHookFunc builtin, with associated documentation on how to use and/or modify existing hooks

[pcfreak30]

Yes please, I really want to not need to be maintaining a fork b/c this/#22? hasn't been merged yet, its an unneeded maintenance burden for my project.

[pkarakal]

I can surely update the PR and make any changes required, since maintaining a fork for this is also not something I fancy, just want the ok to proceed from the maintainers, so that the effort is not in vain

[sagikazarmark]

Given the high demand for this feature, I'm willing to accept this change.

Honestly, I still don't see any scenario where a decode hook couldn't be used to achieve the same (decode structs implementing an interface), but as others pointed out, it's a relatively simple change with a low risk of breaking anything.

That being said, in order for this change to be accepted:

• The interface needs more documentation
• We need to settle on a name for the function (UnmarshalMapstructure feels like a good compromise: it's not too generic to overlap with anything)
• I'd like to see a couple more test scenarios/examples that better showcase how this feature works/protects against future regressions (mainly: make sure it works not just on the top-level struct, make sure that the order of precedence is kept (stop decoding, even if there is a decode hook for that type)

Lastly, I wonder if we should add a builtin error that signals mapstructure that the struct was not able to decode itself and the decoder should proceed with any decode hooks or builtin decoders. WDYT? (We can always add this later, but I just had this idea now)

[pkarakal]

Cool thanks for that. Imma go ahead and make the required changes and will definitely add more test cases for that.

I think it's also fine to have the custom error in case the unmarshaling fails so that it may proceed with the rest. This could also be a decoder configuration: if there is an error in one of the steps, proceed to the next step and join errors along the way. Having those separated I think would also be wise. WDYT? I'm fine either having both of these in this PR or splitting into two PRs.

[pcfreak30]

@pkarakal how is this pr different from #22?

[pcfreak30]

@pkarakal btw while I haven't looked please check

LumeWeb@c63fee0 to see if what your doing is equiv to what I am b/c i had to modify the older pr due to a quirk and having the interface check run before outputKind := getKind(outVal) to get what my requirements are.

[pkarakal]

We need to settle on a name for the function (UnmarshalMapstructure feels like a good compromise: it's not too generic to overlap with anything)

I renamed the function name to UnmarshalMapstructure and I also added a couple of more complex examples based on configs that I have been using in some of my production apps. If you need a couple more, please let me know so I can add them. Also regarding documentation feel to give any feedback you deem necessary.

@pkarakal how is this pr different from #22?

@pcfreak30 I think just the name of the interface and the method are different. Other than that the implementation looks the same to me.

[pcfreak30]

We need to settle on a name for the function (UnmarshalMapstructure feels like a good compromise: it's not too generic to overlap with anything)

I renamed the function name to UnmarshalMapstructure and I also added a couple of more complex examples based on configs that I have been using in some of my production apps. If you need a couple more, please let me know so I can add them. Also regarding documentation feel to give any feedback you deem necessary.

@pkarakal how is this pr different from #22?

@pcfreak30 I think just the name of the interface and the method are different. Other than that the implementation looks the same to me.

I just compared your latest push to what my fork has. LGTM. Don't think I will have issues with that.