Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(database/gdb): json type field incorrect scanning to slice attribute #4107

Open
wants to merge 54 commits into
base: master
Choose a base branch
from

Conversation

@gqcn gqcn requested review from houseme, hailaz, oldme-git and wln32 January 9, 2025 06:18
@gqcn gqcn marked this pull request as ready for review January 10, 2025 02:17
@wln32
Copy link
Member

wln32 commented Jan 13, 2025

@gqcn 后续对于此类问题可以使用 pr #4114 ,不用每次遇到类型不支持时,就修改gf,开放自定义类型转换的接口出去即可,orm默认只支持一些常见类型即可

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


@gqcn You can use pr #4114 for such problems in the future. You don’t need to modify gf every time you encounter a type that is not supported. Just open the interface for custom type conversion. ORM only supports some common types by default.

Base automatically changed from feat/v2.9.0 to master February 27, 2025 07:53
@gogf gogf deleted a comment from Issues-translate-bot Feb 28, 2025
@cyjaysong
Copy link
Contributor

这次改动挺大的啊,使用方法有啥调整吗

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


This time the change is quite big. Are there any adjustments to the usage method?

@gqcn
Copy link
Member Author

gqcn commented Mar 3, 2025

@gqcn 既然都把转换相关的已经做成了可配置的,那么移出internal目录是否更合适一点?本来就是提供给其他模块做不同的配置,现在放在internal目录下,还需要从gconv导出internal里面的一些类型,有些啰嗦,建议把structcache,converter都从internal目录移出,直接放到gconv目录下即可

主要是为了解耦设计,之前这个converter的实现是放外层的,但是容易出现循环依赖。比如在converter的实现中调用了String方法,String方法内部又调用了defaultConverter。放独立的包容易解耦。

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


@gqcn Since all the conversion-related items have been made configurable, is it more appropriate to move out of the internal directory? It was originally provided to other modules to make different configurations. Now it is placed in the internal directory. You also need to export some types in the internal from gconv. It is a bit verbose. It is recommended to move structcache and converter from the internal directory and directly put it in the gconv directory.

It is mainly designed for decoupling. The previous implementation of this converter was placed on the outer layer, but it is prone to circular dependencies. For example, the String method is called in the converter implementation, and the defaultConverter is called inside the String method. It is easy to decouple if placed in an independent package.

@wln32
Copy link
Member

wln32 commented Mar 3, 2025

@gqcn 既然都把转换相关的已经做成了可配置的,那么移出internal目录是否更合适一点?本来就是提供给其他模块做不同的配置,现在放在internal目录下,还需要从gconv导出internal里面的一些类型,有些啰嗦,建议把structcache,converter都从internal目录移出,直接放到gconv目录下即可

主要是为了解耦设计,之前这个converter的实现是放外层的,但是容易出现循环依赖。比如在converter的实现中调用了String方法,String方法内部又调用了defaultConverter。放独立的包容易解耦。

那不应该啊,现在把所有的逻辑都移到converter类型上,直接从这个类型调用就行了,为什么还要去调用外部的String方法,
目前你实现的也是converter.Bool内部调用的是converter.String,而不是gconv.String

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


@gqcn Since all the conversion-related items have been made configurable, is it more appropriate to move out of the internal directory? It was originally provided to other modules to make different configurations. Now it is placed in the internal directory. You also need to export some types in the internal from gconv. It is a bit verbose. It is recommended to move structcache and converter from the internal directory and directly put it in the gconv directory.

It is mainly designed for decoupling. The previous implementation of this converter was placed on the outer layer, but it is prone to circular dependencies. For example, the String method is called in the converter implementation, and the defaultConverter is called inside the String method. It is easy to decouple if placed in an independent package.

That shouldn't be. Now move all the logic to the converter type and just call it from this type. Why do you still need to call the external String method?
Currently, what you implement is also converter.Bool called converter.String instead of gconv.String

@gqcn
Copy link
Member Author

gqcn commented Mar 3, 2025

@gqcn 既然都把转换相关的已经做成了可配置的,那么移出internal目录是否更合适一点?本来就是提供给其他模块做不同的配置,现在放在internal目录下,还需要从gconv导出internal里面的一些类型,有些啰嗦,建议把structcache,converter都从internal目录移出,直接放到gconv目录下即可

主要是为了解耦设计,之前这个converter的实现是放外层的,但是容易出现循环依赖。比如在converter的实现中调用了String方法,String方法内部又调用了defaultConverter。放独立的包容易解耦。

那不应该啊,现在把所有的逻辑都移到converter类型上,直接从这个类型调用就行了,为什么还要去调用外部的String方法, 目前你实现的也是converter.Bool内部调用的是converter.String,而不是gconv.String

是得,作为独立包就可以完全避免这样的问题了。放一起的话,很难避免,容易出错。

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


@gqcn Since all the conversion-related items have been made configurable, is it more appropriate to move out of the internal directory? It was originally provided to other modules to make different configurations. Now it is placed in the internal directory. You also need to export some types in the internal from gconv. It is a bit verbose. It is recommended to move structcache and converter from the internal directory and directly put it in the gconv directory.

Mainly designed for decoupling. The previous implementation of this converter was placed on the outer layer, but it is prone to circular dependencies. For example, the String method is called in the converter implementation, and the defaultConverter is called inside the String method. It is easy to decouple if placed in an independent package.

That shouldn't be. Now move all the logic to the converter type and just call it from this type. Why do you still need to call the external String method? What you are implementing is also converter.String internally, not gconv.String

Yes, as an independent package, you can completely avoid such problems. If put together, it is difficult to avoid and makes mistakes easily.

@wln32
Copy link
Member

wln32 commented Mar 4, 2025

@gqcn 目前还存在一些问题

func anyToMyInt(from any, to reflect.Value) error {
	switch x := from.(type) {
	case int:
		println("x=", x)
		to.SetInt(123456)
	default:
		return fmt.Errorf("unsupported type %T(%v)", from, from)
	}
	return nil
}
// 1. 注册为指针类型时,此时自定义转换不生效
conv1 := gconv.NewConverter()
conv1.RegisterAnyConverterFunc(anyToMyInt, reflect.TypeOf((*myInt)(nil)))
var dst struct {
	A myInt
	B myInt
}
err := conv1.Struct(map[string]any{
	"a": 1200,
	"b": 2400,
}, &dst, gconv.StructOption{})

// dst = {1200, 2400}

//2. 注册为值类型,自定义转换会生效
conv2 := gconv.NewConverter()
conv2 .RegisterAnyConverterFunc(anyToMyInt, reflect.TypeOf((myInt)(0)))
var dst struct {
	A *myInt
	B *myInt
}
err := conv2 .Struct(map[string]any{
	"a": 1200,
	"b": 2400,
}, &dst, gconv.StructOption{})

// dst = {123456, 123456}
  1. 不支持接口类型,例如sql库里面的sql.Null类型均实现了sql.Scanner接口,其他第三方库也都实现了此接口,如果支持接口类型,那么只需要注册一次sql.Scanner即可,不必把所有的sql.Null类型注册进去

@gqcn
Copy link
Member Author

gqcn commented Mar 4, 2025

@gqcn 目前还存在一些问题

func anyToMyInt(from any, to reflect.Value) error {
	switch x := from.(type) {
	case int:
		println("x=", x)
		to.SetInt(123456)
	default:
		return fmt.Errorf("unsupported type %T(%v)", from, from)
	}
	return nil
}
// 1. 注册为指针类型时,此时自定义转换不生效
conv1 := gconv.NewConverter()
conv1.RegisterAnyConverterFunc(anyToMyInt, reflect.TypeOf((*myInt)(nil)))
var dst struct {
	A myInt
	B myInt
}
err := conv1.Struct(map[string]any{
	"a": 1200,
	"b": 2400,
}, &dst, gconv.StructOption{})

// dst = {1200, 2400}

//2. 注册为值类型,自定义转换会生效
conv2 := gconv.NewConverter()
conv2 .RegisterAnyConverterFunc(anyToMyInt, reflect.TypeOf((myInt)(0)))
var dst struct {
	A *myInt
	B *myInt
}
err := conv2 .Struct(map[string]any{
	"a": 1200,
	"b": 2400,
}, &dst, gconv.StructOption{})

// dst = {123456, 123456}
3. 不支持接口类型,例如sql库里面的sql.Null类型均实现了sql.Scanner接口,其他第三方库也都实现了此接口,如果支持接口类型,那么只需要注册一次sql.Scanner即可,不必把所有的sql.Null类型注册进去

嗯,可以提个pr合并到这个分支一起完善下,我这边最近在准备v2.9的发布内容,咱们一起来提速下这个pr的进度。

wln32 and others added 5 commits March 5, 2025 09:17
2.实例化CachedStructInfo时,直接传入Converter
3.增强(*CachedStructInfo).genFieldConvertFunc的逻辑
1.注册指针类型,但是转换到值类型
2.注册值类型,但是转换到指针类型
3.注册接口类型,转换到具体实现接口的类型
@wln32
Copy link
Member

wln32 commented Mar 5, 2025

@gqcn 关于对RegisterAnyConverterFunc的说明,后续可以放在注释和文档中

RegisterAnyConverterFunc分为非接口注册和接口注册

1.非接口类型注册

这个比较好理解,就是注册了一个具体类型的转换函数

2.接口类型注册

当注册的类型为接口类型时,并不是目标类型为接口类型时才有效而是指实现了接口类型

需要注意的是,当你注册了空接口类型,相当于注册了所有类型,因为所有类型都实现了空接口

假如同时注册了sql.NullInt64和sql.Scanner,那么只有sql.NullInt64会生效,因为它的优先级比较高

举例说明

// 注册sql.Scanner接口
RegisterAnyConverterFunc(anyToSqlNullType,reflect.TypeOf((*sql.Scanner)(nil)))
// 转换函数
func anyToSqlNullType(from any,to reflect.Value)error{
    if to.Kind()!=reflect.Ptr {
        to = to.Addr()
    }
    // 只要经过此函数,都赋值为123456
    return to.Interface().(*sql.Scanner).Scan(123456)
}

// 假设我们要转换的结构体类型
// 因为A,B,C,D四个字段的类型均实现了sql.Scanner接口,
// 所以转换的时候,都会经过anyToSqlNullType函数的逻辑
// 但是E字段并不会经过anyToSqlNullType函数
type sqlNullDst struct {
	A sql.Null[int]
	B sql.Null[float32]
	C sql.NullInt64
	D sql.NullString
    E sql.Scanner 
}

var dst sqlNullDst
gconv.Struct(map[string]any{
	"a": 12,
	"b": 34,
	"c": 56,
	"d": "sqlNullString",
    // 如果这样写会报错:reflect.Value.Convert: value of type string cannot be converted to type sql.Scanner
    "e":"sql.Scanner"
    // 下面的可以正常通过
    // "e":&sql.NullString{String: "123456", Valid: true}
}, &dst)
// 由于经过anyToSqlNullType函数,所以dst的四个字段的值也都为123456
// 并不是a=12,b=34

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants