diff --git a/content/programming-guides/enum.md b/content/programming-guides/enum.md index 6373ca360..dd491eb9c 100644 --- a/content/programming-guides/enum.md +++ b/content/programming-guides/enum.md @@ -184,7 +184,8 @@ PHP is conformant. ### Python {#python} -After 4.22.0, Python is conformant. +After [4.22.0](https://pypi.org/project/protobuf/4.22.0/) which was released +~2023-02-16, Python is conformant. In 4.21.x, Python is conformant by default, but setting `PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python` will cause it to be out of diff --git a/content/reference/go/go-generated-opaque.md b/content/reference/go/go-generated-opaque.md new file mode 100644 index 000000000..c68b6d33e --- /dev/null +++ b/content/reference/go/go-generated-opaque.md @@ -0,0 +1,710 @@ ++++ +title = "Go Generated Code Guide (Opaque)" +weight = 615 +linkTitle = "Generated Code Guide (Opaque)" +description = "Describes exactly what Go code the protocol buffer compiler generates for any given protocol definition." +type = "docs" ++++ + +Any differences between +proto2 and proto3 generated code are highlighted - note that these differences +are in the generated code as described in this document, not the base API, which +are the same in both versions. You should read the +[proto2 language guide](/programming-guides/proto2) +and/or the +[proto3 language guide](/programming-guides/proto3) +before reading this document. + +{{% alert title="Note" color="warning" %}}You are +looking at documentation for the Opaque API, which is the current version. If +you are working with .proto files that use the older Open Struct API (you can +tell by the API level setting in the respective .proto files), see +[Go Generated Code (Open)](/reference/go/go-generated) +for the corresponding documentation. See +[Go Protobuf: The new Opaque API](https://go.dev/blog/protobuf-opaque) for the +introduction of the Opaque API. {{% /alert %}} + +## Compiler Invocation {#invocation} + +The protocol buffer compiler requires a plugin to generate Go code. Install it +using Go 1.16 or higher by running: + +```shell +go install google.golang.org/protobuf/cmd/protoc-gen-go@latest +``` + +This will install a `protoc-gen-go` binary in `$GOBIN`. Set the `$GOBIN` +environment variable to change the installation location. It must be in your +`$PATH` for the protocol buffer compiler to find it. + +The protocol buffer compiler produces Go output when invoked with the `go_out` +flag. The argument to the `go_out` flag is the directory where you want the +compiler to write your Go output. The compiler creates a single source file for +each `.proto` file input. The name of the output file is created by replacing +the `.proto` extension with `.pb.go`. + +Where in the output directory the generated `.pb.go` file is placed depends on +the compiler flags. There are several output modes: + +- If the `paths=import` flag is specified, the output file is placed in a + directory named after the Go package's import path (such as one provided by + the `go_package` option within the `.proto` file). For example, an input + file `protos/buzz.proto` with a Go import path of + `example.com/project/protos/fizz` results in an output file at + `example.com/project/protos/fizz/buzz.pb.go`. This is the default output + mode if a `paths` flag is not specified. +- If the `module=$PREFIX` flag is specified, the output file is placed in a + directory named after the Go package's import path (such as one provided by + the `go_package` option within the `.proto` file), but with the specified + directory prefix removed from the output filename. For example, an input + file `protos/buzz.proto` with a Go import path of + `example.com/project/protos/fizz` and `example.com/project` specified as the + `module` prefix results in an output file at `protos/fizz/buzz.pb.go`. + Generating any Go packages outside the module path results in an error. This + mode is useful for outputting generated files directly into a Go module. +- If the `paths=source_relative` flag is specified, the output file is placed + in the same relative directory as the input file. For example, an input file + `protos/buzz.proto` results in an output file at `protos/buzz.pb.go`. + +Flags specific to `protoc-gen-go` are provided by passing a `go_opt` flag when +invoking `protoc`. Multiple `go_opt` flags may be passed. For example, when +running: + +```shell +protoc --proto_path=src --go_out=out --go_opt=paths=source_relative foo.proto bar/baz.proto +``` + +the compiler will read input files `foo.proto` and `bar/baz.proto` from within +the `src` directory, and write output files `foo.pb.go` and `bar/baz.pb.go` to +the `out` directory. The compiler automatically creates nested output +sub-directories if necessary, but will not create the output directory itself. + +## Packages {#package} + +In order to generate Go code, the Go package's import path must be provided for +every `.proto` file (including those transitively depended upon by the `.proto` +files being generated). There are two ways to specify the Go import path: + +- by declaring it within the `.proto` file, or +- by declaring it on the command line when invoking `protoc`. + +We recommend declaring it within the `.proto` file so that the Go packages for +`.proto` files can be centrally identified with the `.proto` files themselves +and to simplify the set of flags passed when invoking `protoc`. If the Go import +path for a given `.proto` file is provided by both the `.proto` file itself and +on the command line, then the latter takes precedence over the former. + +The Go import path is locally specified in a `.proto` file by declaring a +`go_package` option with the full import path of the Go package. Example usage: + +```proto +option go_package = "example.com/project/protos/fizz"; +``` + +The Go import path may be specified on the command line when invoking the +compiler, by passing one or more `M${PROTO_FILE}=${GO_IMPORT_PATH}` flags. +Example usage: + +```shell +protoc --proto_path=src \ + --go_opt=Mprotos/buzz.proto=example.com/project/protos/fizz \ + --go_opt=Mprotos/bar.proto=example.com/project/protos/foo \ + protos/buzz.proto protos/bar.proto +``` + +Since the mapping of all `.proto` files to their Go import paths can be quite +large, this mode of specifying the Go import paths is generally performed by +some build tool (e.g., [Bazel](https://bazel.build/)) that has +control over the entire dependency tree. If there are duplicate entries for a +given `.proto` file, then the last one specified takes precedence. + +For both the `go_package` option and the `M` flag, the value may include an +explicit package name separated from the import path by a semicolon. For +example: `"example.com/protos/foo;package_name"`. This usage is discouraged +since the package name will be derived by default from the import path in a +reasonable manner. + +The import path is used to determine which import statements must be generated +when one `.proto` file imports another `.proto` file. For example, if `a.proto` +imports `b.proto`, then the generated `a.pb.go` file needs to import the Go +package which contains the generated `b.pb.go` file (unless both files are in +the same package). The import path is also used to construct output filenames. +See the \"Compiler Invocation\" section above for details. + +There is no correlation between the Go import path and the +[`package` specifier](/programming-guides/proto3#packages) +in the `.proto` file. The latter is only relevant to the protobuf namespace, +while the former is only relevant to the Go namespace. Also, there is no +correlation between the Go import path and the `.proto` import path. + +## API level {#apilevel} + +The generated code either uses the Open Struct API or the Opaque API. See the +[Go Protobuf: The new Opaque API](https://go.dev/blog/protobuf-opaque) +blog post for an introduction. + +Depending on the syntax your `.proto` file uses, here is which API will be used: + +`.proto` syntax | API level +--------------- | ---------- +proto2 | Open Struct API +proto3 | Open Struct API +edition 2023 | Open Struct API +edition 2024+ | Opaque API + +You can select the API by setting the `api_level` editions feature in your +`.proto` file. This can be set per file or per message: + +```proto +edition = "2023"; + +package log; + +import "google/protobuf/go_features.proto"; +option features.(pb.go).api_level = API_OPAQUE; + +message LogEntry { … } +``` + +For your convenience, you can also override the default API level with a +`protoc` command-line flag: + +``` +protoc […] --go_opt=default_api_level=API_HYBRID +``` + +To override the default API level for a specific file (instead of all files), +use the `apilevelM` mapping flag (similar to [the `M` flag for import +paths](/reference/go/go-generated/#package)): + +``` +protoc […] --go_opt=apilevelMhello.proto=API_HYBRID +``` + +The command-line flags also work for `.proto` files still using proto2 or proto3 +syntax, but if you want to select the API level from within the `.proto` file, +you need to migrate said file to editions first. + +## Messages {#message} + +Given a simple message declaration: + +```proto +message Artist {} +``` + +the protocol buffer compiler generates a struct called `Artist`. An `*Artist` +implements the +[`proto.Message`](https://pkg.go.dev/google.golang.org/protobuf/proto?tab=doc#Message) +interface. + +The +[`proto` package](https://pkg.go.dev/google.golang.org/protobuf/proto?tab=doc) +provides functions which operate on messages, including conversion to and from +binary format. + +The `proto.Message` interface defines a `ProtoReflect` method. This method +returns a +[`protoreflect.Message`](https://pkg.go.dev/google.golang.org/protobuf/reflect/protoreflect?tab=doc#Message) +which provides a reflection-based view of the message. + +The `optimize_for` option does not affect the output of the Go code generator. + +### Nested Types + +A message can be declared inside another message. For example: + +```proto +message Artist { + message Name { + } +} +``` + +In this case, the compiler generates two structs: `Artist` and `Artist_Name`. + +## Fields + +The protocol buffer compiler generates accessor methods (setters and getters) +for each field defined within a message. + +Note that the generated Go accessor methods always use camel-case naming, even +if the field name in the `.proto` file uses lower-case with underscores +([as it should](/programming-guides/style)). The +case-conversion works as follows: + +1. The first letter is capitalized for export. If the first character is an + underscore, it is removed and a capital X is prepended. +2. If an interior underscore is followed by a lower-case letter, the underscore + is removed, and the following letter is capitalized. + +Thus, you can access the proto field `birth_year` using the `GetBirthYear()` +method in Go, and `_birth_year_2` using `GetXBirthYear_2()`. + +### Singular Scalar Fields (proto2) {#singular-scalar-proto2} + +For either of these field definitions: + +```proto +optional int32 birth_year = 1; +required int32 birth_year = 1; +``` + +the compiler generates the following accessor methods: + +```go +func (m *Artist) GetBirthYear() int32 { ... } +func (m *Artist) SetBirthYear(v int32) { ... } +func (m *Artist) HasBirthYear() bool { ... } +func (m *Artist) ClearBirthYear() { ... } +``` + +The accessor method `GetBirthYear()` returns the `int32` value in `birth_year` +or the default value if the field is unset. If the default is not explicitly +set, the [zero value](https://golang.org/ref/spec#The_zero_value) of +that type is used instead (`0` for numbers, the empty string for strings). + +For other scalar field types (including `bool`, `bytes`, and `string`), `int32` +is replaced with the corresponding Go type according to the +[scalar value types table](/programming-guides/proto2#scalar). + +### Singular Scalar Fields (proto3) {#singular-scalar-proto3} + +For this field definition: + +```proto +int32 birth_year = 1; +optional int32 first_active_year = 2; +``` + +the compiler generates the following accessor methods: + +```go +func (m *Artist) GetBirthYear() int32 { ... } +func (m *Artist) SetBirthYear(v int32) { ... } +// NOTE: No HasBirthYear() or ClearBirthYear() methods; +// proto3 fields only have presence when declared as optional: +// /programming-guides/field_presence.md + +func (m *Artist) GetFirstActiveYear() int32 { ... } +func (m *Artist) SetFirstActiveYear(v int32) { ... } +func (m *Artist) HasFirstActiveYear() bool { ... } +func (m *Artist) ClearFirstActiveYear() { ... } +``` + +The accessor method `GetBirthYear()` returns the `int32` value in `birth_year` +or the [zero value](https://golang.org/ref/spec#The_zero_value) of +that type if the field is unset (`0` for numbers, the empty string for strings). + +For other scalar field types (including `bool`, `bytes`, and `string`), `int32` +is replaced with the corresponding Go type according to the +[scalar value types table](/programming-guides/proto3#scalar). +Unset values in the proto will be represented as the +[zero value](https://golang.org/ref/spec#The_zero_value) of that type +(`0` for numbers, the empty string for strings). + +### Singular Message Fields {#singular-message} + +Given the message type: + +```proto +message Band {} +``` + +For a message with a `Band` field: + +```proto +// proto2 +message Concert { + optional Band headliner = 1; + // The generated code is the same result if required instead of optional. +} + +// proto3 +message Concert { + Band headliner = 1; +} +``` + +The compiler will generate a Go struct with the following accessor methods: + +```go +type Concert struct { ... } + +func (m *Concert) GetHeadliner() *Band { ... } +func (m *Concert) SetHeadliner(v *Band) { ... } +func (m *Concert) HasHeadliner() bool { ... } +func (m *Concert) ClearHeadliner() { ... } +``` + +The `GetHeadliner()` accessor method is safe to call even if `m` is nil. This +makes it possible to chain get calls without intermediate `nil` checks: + +```go +var m *Concert // defaults to nil +log.Infof("GetFoundingYear() = %d (no panic!)", m.GetHeadliner().GetFoundingYear()) +``` + +If the field is unset, the getter will return the default value of the field. +For messages, the default value is a nil pointer. + +Contrary to getters, setters do not perform nil checks for you. Therefore, you +cannot safely call setters on possibly-nil messages. + +### Repeated Fields {#repeated} + +For repeated fields, the accessor methods use a slice type. For this message +with a repeated field: + +```proto +message Concert { + // Best practice: use pluralized names for repeated fields: + // /programming-guides/style#repeated-fields + repeated Band support_acts = 1; +} +``` + +the compiler generates a Go struct with the following accessor methods: + +```go +type Concert struct { ... } + +func (m *Concert) GetSupportActs() []*Band { ... } +func (m *Concert) SetSupportActs(v []*Band) { ... } +``` + +Likewise, for the field definition `repeated bytes band_promo_images = 1;` the +compiler will generate accessors working with the `[][]byte` type. For a +repeated [enumeration](#enum) `repeated MusicGenre genres = 2;`, the compiler +generates accessors working with the `[]MusicGenre` type. + +The following example shows how to construct a `Concert` message using a +[builder](#builders). + +```go +concert := Concert_builder{ + SupportActs: []*Band{ + {}, // First element. + {}, // Second element. + }, +}.Build() +``` + +Alternatively, you can use setters: + +```go +concert := &Concert{} +concert.SetSupportActs([]*Band{ + {}, // First element. + {}, // Second element. +}) +``` + +To access the field, you can do the following: + +```go +support := concert.GetSupportActs() // support type is []*Band. +b1 := support[0] // b1 type is *Band, the first element in support_acts. +``` + +### Map Fields {#map} + +Each map field generates accessors working with type `map[TKey]TValue` where +`TKey` is the field's key type and `TValue` is the field's value type. For this +message with a map field: + +```proto +message MerchItem {} + +message MerchBooth { + // items maps from merchandise item name ("Signed T-Shirt") to + // a MerchItem message with more details about the item. + map items = 1; +} +``` + +the compiler generates a Go struct with the following accessor methods: + +```go +type MerchBooth struct { ... } + +func (m *MerchBooth) GetItems() map[string]*MerchItem { ... } +func (m *MerchBooth) SetItems(v map[string]*MerchItem) { ... } +``` + +### Oneof Fields {#oneof} + +For a oneof field, the protobuf compiler generates accessors for each of the +[singular fields](#singular-scalar-proto2) within the oneof. + +For this message with a oneof field: + +```proto +package account; +message Profile { + oneof avatar { + string image_url = 1; + bytes image_data = 2; + } +} +``` + +the compiler generates a Go struct with the following accessor methods: + +```go +type Profile struct { ... } + +func (m *Profile) WhichAvatar() case_Profile_Avatar { ... } +func (m *Profile) GetImageUrl() string { ... } +func (m *Profile) GetImageData() []byte { ... } + +func (m *Profile) SetImageUrl(v string) { ... } +func (m *Profile) SetImageData(v []byte) { ... } + +func (m *Profile) HasAvatar() bool { ... } +func (m *Profile) HasImageUrl() bool { ... } +func (m *Profile) HasImageData() bool { ... } + +func (m *Profile) ClearAvatar() { ... } +func (m *Profile) ClearImageUrl() { ... } +func (m *Profile) ClearImageData() { ... } +``` + +The following example shows how to set the field using a [builder](#builders): + +```go +p1 := accountpb.Profile_builder{ + ImageUrl: proto.String("https://example.com/image.png"), +}.Build() +``` + +...or, equivalently, using a setter: + +```go +// imageData is []byte +imageData := getImageData() +p2 := &accountpb.Profile{} +p2.SetImageData(imageData) +``` + +To access the field, you can use a switch statement on the `WhichAvatar()` +result: + +```go +switch m.WhichAvatar() { +case accountpb.Profile_ImageUrl_case: + // Load profile image based on URL + // using m.GetImageUrl() + +case accountpb.Profile_ImageData_case: + // Load profile image based on bytes + // using m.GetImageData() + +case accountpb.Profile_Avatar_not_set_case: + // The field is not set. + +default: + return fmt.Errorf("Profile.Avatar has an unexpected new oneof field %v", x) +} +``` + +### Builders {#builders} + +Builders are a convenient way to construct and initialize a message within a +single expression, especially when working with nested messages like unit tests. + +Contrary to builders in other languages (like Java), Go protobuf builders are +not meant to be passed around between functions. Instead, call `Build()` +immediately and pass the resulting proto message instead, using setters to +modify fields. + +## Enumerations {#enum} + +Given an enumeration like: + +```proto +message Venue { + enum Kind { + KIND_UNSPECIFIED = 0; + KIND_CONCERT_HALL = 1; + KIND_STADIUM = 2; + KIND_BAR = 3; + KIND_OPEN_AIR_FESTIVAL = 4; + } + Kind kind = 1; + // ... +} +``` + +the protocol buffer compiler generates a type and a series of constants with +that type: + +```go +type Venue_Kind int32 + +const ( + Venue_KIND_UNSPECIFIED Venue_Kind = 0 + Venue_KIND_CONCERT_HALL Venue_Kind = 1 + Venue_KIND_STADIUM Venue_Kind = 2 + Venue_KIND_BAR Venue_Kind = 3 + Venue_KIND_OPEN_AIR_FESTIVAL Venue_Kind = 4 +) +``` + +For enums within a message (like the one above), the type name begins with the +message name: + +```go +type Venue_Kind int32 +``` + +For a package-level enum: + +```proto +enum Genre { + GENRE_UNSPECIFIED = 0; + GENRE_ROCK = 1; + GENRE_INDIE = 2; + GENRE_DRUM_AND_BASS = 3; + // ... +} +``` + +the Go type name is unmodified from the proto enum name: + +```go +type Genre int32 +``` + +This type has a `String()` method that returns the name of a given value. + +The `Enum()` method initializes freshly allocated memory with a given value and +returns the corresponding pointer: + +```go +func (Genre) Enum() *Genre +``` + +The protocol buffer compiler generates a constant for each value in the enum. +For enums within a message, the constants begin with the enclosing message's +name: + +```go +const ( + Venue_KIND_UNSPECIFIED Venue_Kind = 0 + Venue_KIND_CONCERT_HALL Venue_Kind = 1 + Venue_KIND_STADIUM Venue_Kind = 2 + Venue_KIND_BAR Venue_Kind = 3 + Venue_KIND_OPEN_AIR_FESTIVAL Venue_Kind = 4 +) +``` + +For a package-level enum, the constants begin with the enum name instead: + +```go +const ( + Genre_GENRE_UNSPECIFIED Genre = 0 + Genre_GENRE_ROCK Genre = 1 + Genre_GENRE_INDIE Genre = 2 + Genre_GENRE_DRUM_AND_BASS Genre = 3 +) +``` + +The protobuf compiler also generates a map from integer values to the string +names and a map from the names to the values: + +```go +var Genre_name = map[int32]string{ + 0: "GENRE_UNSPECIFIED", + 1: "GENRE_ROCK", + 2: "GENRE_INDIE", + 3: "GENRE_DRUM_AND_BASS", +} +var Genre_value = map[string]int32{ + "GENRE_UNSPECIFIED": 0, + "GENRE_ROCK": 1, + "GENRE_INDIE": 2, + "GENRE_DRUM_AND_BASS": 3, +} +``` + +Note that the `.proto` language allows multiple enum symbols to have the same +numeric value. Symbols with the same numeric value are synonyms. These are +represented in Go in exactly the same way, with multiple names corresponding to +the same numeric value. The reverse mapping contains a single entry for the +numeric value to the name which appears first in the .proto file. + +## Extensions (proto2) {#extensions} + +Given an extension definition: + +```proto +extend Concert { + optional int32 promo_id = 123; +} +``` + +The protocol buffer compiler will generate a +[`protoreflect.ExtensionType`](https://pkg.go.dev/google.golang.org/protobuf/reflect/protoreflect?tab=doc#ExtensionType) +value named `E_Promo_id`. This value may be used with the +[`proto.GetExtension`](https://pkg.go.dev/google.golang.org/protobuf/proto?tab=doc#GetExtension), +[`proto.SetExtension`](https://pkg.go.dev/google.golang.org/protobuf/proto?tab=doc#SetExtension), +[`proto.HasExtension`](https://pkg.go.dev/google.golang.org/protobuf/proto?tab=doc#HasExtension), +and +[`proto.ClearExtension`](https://pkg.go.dev/google.golang.org/protobuf/proto?tab=doc#ClearExtension) +functions to access an extension in a message. The `GetExtension` function and +`SetExtension` functions respectively return and accept an `interface{}` value +containing the extension value type. + +For singular scalar extension fields, the extension value type is the +corresponding Go type from the +[scalar value types table](/programming-guides/proto3#scalar). + +For singular embedded message extension fields, the extension value type is +`*M`, where `M` is the field message type. + +For repeated extension fields, the extension value type is a slice of the +singular type. + +For example, given the following definition: + +```proto +extend Concert { + optional int32 singular_int32 = 1; + repeated bytes repeated_strings = 2; + optional Band singular_message = 3; +} +``` + +Extension values may be accessed as: + +```go +m := &somepb.Concert{} +proto.SetExtension(m, extpb.E_SingularInt32, int32(1)) +proto.SetExtension(m, extpb.E_RepeatedString, []string{"a", "b", "c"}) +proto.SetExtension(m, extpb.E_SingularMessage, &extpb.Band{}) + +v1 := proto.GetExtension(m, extpb.E_SingularInt32).(int32) +v2 := proto.GetExtension(m, extpb.E_RepeatedString).([][]byte) +v3 := proto.GetExtension(m, extpb.E_SingularMessage).(*extpb.Band) +``` + +Extensions can be declared nested inside of another type. For example, a common +pattern is to do something like this: + +```proto +message Promo { + extend Concert { + optional int32 promo_id = 124; + } +} +``` + +In this case, the `ExtensionType` value is named `E_Promo_Concert`. + +## Services {#service} + +The Go code generator does not produce output for services by default. If you +enable the [gRPC](https://www.grpc.io/) plugin (see the +[gRPC Go Quickstart guide](https://github.com/grpc/grpc-go/tree/master/examples)) +then code will be generated to support gRPC. diff --git a/content/reference/go/go-generated.md b/content/reference/go/go-generated.md index bce4cb855..3d1b5dc45 100644 --- a/content/reference/go/go-generated.md +++ b/content/reference/go/go-generated.md @@ -1,7 +1,7 @@ +++ -title = "Go Generated Code Guide" +title = "Go Generated Code Guide (Open)" weight = 610 -linkTitle = "Generated Code Guide" +linkTitle = "Generated Code Guide (Open)" description = "Describes exactly what Go code the protocol buffer compiler generates for any given protocol definition." type = "docs" +++ @@ -15,6 +15,14 @@ and/or the [proto3 language guide](/programming-guides/proto3) before reading this document. +{{% alert title="Note" color="warning" %}}You are +looking at documentation for the old generated code API (Open Struct API). +See +[Go Generated Code (Opaque)](/reference/go/go-generated-opaque) +for the corresponding documentation of the (new) Opaque API. See +[Go Protobuf: The new Opaque API](https://go.dev/blog/protobuf-opaque) for the +introduction of the Opaque API. {{% /alert %}} + ## Compiler Invocation {#invocation} The protocol buffer compiler requires a plugin to generate Go code. Install it @@ -128,6 +136,54 @@ in the `.proto` file. The latter is only relevant to the protobuf namespace, while the former is only relevant to the Go namespace. Also, there is no correlation between the Go import path and the `.proto` import path. +## API level {#apilevel} + +The generated code either uses the Open Struct API or the Opaque API. See the +[Go Protobuf: The new Opaque API](https://go.dev/blog/protobuf-opaque) +blog post for an introduction. + +Depending on the syntax your `.proto` file uses, here is which API will be used: + +`.proto` syntax | API level +--------------- | ---------- +proto2 | Open Struct API +proto3 | Open Struct API +edition 2023 | Open Struct API +edition 2024+ | Opaque API + +You can select the API by setting the `api_level` editions feature in your +`.proto` file. This can be set per file or per message: + +```proto +edition = "2023"; + +package log; + +import "google/protobuf/go_features.proto"; +option features.(pb.go).api_level = API_OPAQUE; + +message LogEntry { … } +``` + +For your convenience, you can also override the default API level with a +`protoc` command-line flag: + +``` +protoc […] --go_opt=default_api_level=API_HYBRID +``` + +To override the default API level for a specific file (instead of all files), +use the `apilevelM` mapping flag (similar to [the `M` flag for import +paths](/reference/go/go-generated/#package)): + +``` +protoc […] --go_opt=apilevelMhello.proto=API_HYBRID +``` + +The command-line flags also work for `.proto` files still using proto2 or proto3 +syntax, but if you want to select the API level from within the `.proto` file, +you need to migrate said file to editions first. + ## Messages {#message} Given a simple message declaration: @@ -219,6 +275,9 @@ an accessor method `GetBirthYear()` which returns the `int32` value in [zero value](https://golang.org/ref/spec#The_zero_value) of that type if the field is unset (`0` for numbers, the empty string for strings). +The `FirstActiveYear` struct field will be of type `*int32`, because it is +marked `optional`. + For other scalar field types (including `bool`, `bytes`, and `string`), `int32` is replaced with the corresponding Go type according to the [scalar value types table](/programming-guides/proto3#scalar). diff --git a/content/reference/go/opaque-faq.md b/content/reference/go/opaque-faq.md new file mode 100644 index 000000000..1e460978b --- /dev/null +++ b/content/reference/go/opaque-faq.md @@ -0,0 +1,380 @@ ++++ +title = "Go Opaque API FAQ" +weight = 670 +linkTitle = "Opaque API FAQ" +description = "A list of frequently asked questions about the Opaque API." +type = "docs" ++++ + + + +The Opaque API is the latest version of the Protocol Buffers implementation for +the Go programming language. The old version is now called Open Struct API. See +the [Go Protobuf: The new Opaque API](https://go.dev/blog/protobuf-opaque) blog +post for an introduction. + +This FAQ answers common questions about the new API and the migration process. + +## Which API Should I Use When Creating a New .proto File? {#which} + +We recommend you select the Opaque API for new development. Protobuf Edition +2024 (see +[Protobuf Editions Overview](/editions/overview/)) will +make the Opaque API the default. + +## How Do I Enable the New Opaque API for My Messages? {#enable} + +With Protobuf Edition 2023 (current at the time of writing), you can select the +Opaque API by setting the `api_level` editions feature to `API_OPAQUE` in your +`.proto` file. This can be set per file or per message: + +```proto +edition = "2023"; + +package log; + +import "google/protobuf/go_features.proto"; +option features.(pb.go).api_level = API_OPAQUE; + +message LogEntry { … } +``` + +Protobuf Edition 2024 will default to the Opaque API, meaning you will not need +extra imports or options anymore: + +```proto +edition = "2024"; + +package log; + +message LogEntry { … } +``` + +The release date estimate for Protobuf Edition 2024 is early 2025. + +For your convenience, you can also override the default API level with a +`protoc` command-line flag: + +``` +protoc […] --go_opt=default_api_level=API_HYBRID +``` + +To override the default API level for a specific file (instead of all files), +use the `apilevelM` mapping flag (similar to +[the `M` flag for import paths](/reference/go/go-generated/#package)): + +``` +protoc […] --go_opt=apilevelMhello.proto=API_HYBRID +``` + +The command-line flags also work for `.proto` files still using proto2 or proto3 +syntax, but if you want to select the API level from within the `.proto` file, +you need to migrate said file to editions first. + +## How Do I Enable Lazy Decoding? {#lazydecoding} + +1. Migrate your code to use the opaque implementation. +1. Set the `[lazy = true]` option on the proto submessage fields that should be + lazily decoded. +1. Run your unit and integration tests, and then roll out to a staging + environment. + +## Where Can I Ask Questions or Report Issues? {#questions} + +If you found an issue with the `open2opaque` migration tool (such as incorrectly +rewritten code), please report it in the +[open2opaque issue tracker](https://github.com/golang/open2opaque/issues). + +If you found an issue with Go Protobuf, please report it in the +[Go Protobuf issue tracker](https://github.com/golang/protobuf/issues/). + +## What Are the Benefits of the Opaque API? {#benefits} + +The Opaque API comes with numerous benefits: + +* It uses a more efficient memory representation, thereby reducing memory and + Garbage Collection cost. +* It makes lazy decoding possible, which can significantly improve + performance. +* It fixes a number of sharp edges. Bugs resulting from pointer address + comparison, accidental sharing, or undesired use of Go reflection are all + prevented when using the Opaque API. +* It makes the ideal memory layout possible by enabling profile-driven + optimizations. + +See +[the Go Protobuf: The new Opaque API blog post](https://go.dev/blog/protobuf-opaque) +for more details on these points. + +## Which Is Faster, Builders or Setters? {#builders-faster} + +Generally, code using builders: + +```go +_ = pb.M_builder{ + F: &val, +}.Build() +``` + +is slower than the following equivalent: + +```go +m := &pb.M{} +m.SetF(val) +``` + +for the following reasons: + +1. The `Build()` call iterates over all fields in the message (even ones that + are not explicitly set) and copies their values (if any) to the final + message. This linear performance matters for messages with many fields. +1. There's a potential extra heap allocation (`&val`). +1. The builder can be significantly larger and use more memory in presence of + oneof fields. Builders have a field per oneof union member while the message + can store the oneof itself as a single field. + +Aside from runtime performance, if binary size is a concern for you, avoiding +builders will result in less code. + +## How Do I Use Builders? {#builders-how} + +Builders are designed to be used as *values* and with an immediate `Build()` +call. Avoid using pointers to builders or storing builders in variables. + +```go {.good} +m := pb.M_builder{ + // ... +}.Build() +``` + +```go {.bad} +// BAD: Avoid using a pointer +m := (&pb.M_builder{ + // ... +}).Build() +``` + +```go {.bad highlight="content:b.:= "} +// BAD: avoid storing in a variable +b := pb.M_builder{ + // ... +} +m := b.Build() +``` + +Proto messages are immutable in some other languages, hence users tend to pass +the builder type into function calls when constructing a proto message. Go proto +messages are mutable, hence there's no need for passing the builder into +function calls. Simply pass the proto message. + +```go {.bad} +// BAD: avoid passing a builder around +func populate(mb *pb.M_builder) { + mb.Field1 = proto.Int32(4711) + //... +} +// ... +mb := pb.M_builder{} +populate(&mb) +m := mb.Build() +``` + +```go {.good} +func populate(mb *pb.M) { + mb.SetField1(4711) + //... +} +// ... +m := &pb.M{} +populate(m) +``` + +Builders are designed to imitate the composite literal construction of the Open +Struct API, not as an alternative representation of a proto message. + +The recommended pattern is also more performant. The intended use of `Build()` +where it is called directly on the builder struct literal can be optimized well. +A separate call to `Build()` is much harder to optimize, as the compiler may not +easily identify which fields are populated. If the builder lives longer, there's +also a high chance that small objects like scalars have to be heap allocated and +later need to be freed by the garbage collector. + +## Should I Use Builders or Setters? {#builders-vs-setters} + +When constructing an empty protocol buffer, you should use `new` or an empty +composite literals. Both are equivalently idiomatic to construct a zero +initialized value in Go and are more performant than an empty builder. + +```go {.good} +m1 := new(pb.M) +m2 := &pb.M{} +``` + +```go {.bad} +// BAD: avoid: unnecessarily complex +m1 := pb.M_builder{}.Build() +``` + +In cases where you need to construct non-empty protocol buffers, you have the +choice between using setters or using builders. Either is fine, but most people +will find builders more readable. If the code you are writing needs to perform +well, +[setters are generally slightly more performant than builders](#builders-faster). + +```go {.good} +// Recommended: using builders +m1 := pb.M1_builder{ + Submessage: pb.M2_builder{ + Submessage: pb.M3_builder{ + String: proto.String("hello world"), + Int: proto.Int32(42), + }.Build(), + Bytes: []byte("hello"), + }.Build(), +}.Build() +``` + +```go +// Also okay: using setters +m3 := &pb.M3{} +m3.SetString("hello world") +m3.SetInt(42) +m2 := &pb.M2{} +m2.SetSubmessage(m3) +m2.SetBytes([]byte("hello")) +m1 := &pb.M1{} +m1.SetSubmessage(m2) +``` + +You can combine the use of builder and setters if certain fields require +conditional logic before setting. + +```go {.good} +m1 := pb.M1_builder{ + Field1: value1, +}.Build() +if someCondition() { + m1.SetField2(value2) + m1.SetField3(value3) +} +``` + +## How Can I Influence open2opaque’s Builder Behavior? {#builders-flags} + +The `open2opaque` tool’s `--use_builders` flag can have the following values: + +* `--use_builders=everywhere`: always use builders, no exceptions. +* `--use_builders=tests`: use builders only in tests, setters otherwise. +* `--use_builders=nowhere`: never use builders. + +## How Much Performance Benefit Can I Expect? {#performance-benefit} + +This depends heavily on your workload. The following questions can guide your +performance exploration: + +* How big a percentage of your CPU usage is Go Protobuf? Some workloads, like + logs analysis pipelines that compute statistics based on Protobuf input + records, can spend about 50% of their CPU usage in Go Protobuf. Performance + improvements will likely be clearly visible in such workloads. On the other + end of the spectrum, in programs that only spend 3-5% of their CPU usage in + Go Protobuf, performance improvements will often be insignificant compared + to other opportunities. +* How amenable to lazy decoding is your program? If large portions of the + input messages are never accessed, lazy decoding can save a lot of work. + This pattern is usually encountered in jobs like proxy servers (which pass + through the input as-is), or logs analysis pipelines with high selectivity + (which discard many records based on a high-level predicate). +* Do your message definitions contain many elementary fields with explicit + presence? The Opaque API uses a more efficient memory representation for + elementary fields like integers, booleans, enums and floats, but not + strings, repeated fields or submessages. + +## How Does Proto2, Proto3, and Editions Relate to the Opaque API? {#proto23editions} + +The terms proto2 and proto3 refer to different syntax versions in your `.proto` +files. [Protobuf Editions](/editions/overview) is the +successor to both proto2 and proto3. + +The Opaque API affects only the generated code in `.pb.go` files, not what you +write in your `.proto` files. + +The Opaque API works the same, independent of which syntax or edition your +`.proto` files use. However, if you want to select the Opaque API on a per-file +basis (as opposed to using a command-line flag when you are running `protoc`), +you must migrate the file to editions first. See +[How Do I Enable the New Opaque API for My Messages?](#enable) for details. + +## Why Only Change the Memory Layout of Elementary Fields? {#memorylayout} + +The +[announcement blog post’s “Opaque structs use less memory” section](https://go.dev/blog/protobuf-opaque#lessmemory) +explains: + +> This performance improvement [modeling field presence more efficiently] +> depends heavily on your protobuf message shape: The change only affects +> elementary fields like integers, booleans, enums and floats, but not strings, +> repeated fields or submessages. + +A natural follow-up question is why strings, repeated fields, and submessages +remain pointers in the Opaque API. The answer is twofold. + +### Consideration 1: Memory Usage + +Representing submessages as values instead of pointers would increase memory +usage: each Protobuf message type carries internal state, which would consume +memory even when the submessage is not actually set. + +For strings and repeated fields, the situation is more nuanced. Let’s compare +the memory usage of using a string value compared to a string pointer: + +Go variable type | set? | [word]s | #bytes +---------------- | ---- | ------------------------ | ------ +`string` | yes | 2 (data, len) | 16 +`string` | no | 2 (data, len) | 16 +`*string` | yes | 1 (data) + 2 (data, len) | 24 +`*string` | no | 1 (data) | 8 + +[word]: https://en.wikipedia.org/wiki/Word_(computer_architecture) + +(The situation is similar for slices, but slice headers need 3 words: data, len, +cap.) + +If your string fields are overwhelmingly not set, using a pointer saves RAM. Of +course, this saving comes at the cost of introducing more allocations and +pointers into the program, which increases load on the Garbage Collector. + +The advantage of the Opaque API is that we can change the representation without +any changes to user code. The current memory layout was optimal for us when we +introduced it, but if we measured today or 5 years into the future, maybe we +would have chosen a different layout. + +As described in the +[announcement blog post’s “Making the ideal memory layout possible” section](https://go.dev/blog/protobuf-opaque#idealmemory), +we aim to make these optimization decisions on a per-workload basis in the +future. + +### Consideration 2: Lazy Decoding + +Aside from the memory usage consideration, there is another restriction: fields +for which [lazy decoding](#lazydecoding) is enabled must be represented by +pointers. + +Protobuf messages are safe for concurrent access (but not concurrent mutation), +so if two different goroutines trigger lazy decoding, they need to coordinate +somehow. This coordination is implemented through using the +[`sync/atomic` package](https://pkg.go.dev/sync/atomic), which can update +pointers atomically, but not slice headers (which exceed a [word]). + +While `protoc` currently only permits lazy decoding for (non-repeated) +submessages, this reasoning holds for all field types. diff --git a/content/reference/go/opaque-migration-manual.md b/content/reference/go/opaque-migration-manual.md new file mode 100644 index 000000000..ce94afedb --- /dev/null +++ b/content/reference/go/opaque-migration-manual.md @@ -0,0 +1,668 @@ ++++ +title = "Go Opaque API: Manual Migration" +weight = 660 +linkTitle = "Opaque API: Manual Migration" +description = "Describes a manual migration to the Opaque API." +type = "docs" ++++ + +The Opaque API is the latest version of the Protocol Buffers implementation for +the Go programming language. The old version is now called Open Struct API. See +the [Go Protobuf: Releasing the Opaque API](https://go.dev/blog/protobuf-opaque) +blog post for an introduction. + +This is a user guide for migrating Go Protobuf usages from the older Open Struct +API to the new Opaque API. + +{{% alert title="Warning" color="warning" %}} You +are looking at the manual migration guide. Typically you’re better off using the +`open2opaque` tool to automate the migration. See +[Opaque API Migration](/reference/go/opaque-migration) +instead. {{% /alert %}} + +The [Generated Code Guide](/reference/go/go-generated) +provides more detail. This guide compares the old and new API side-by-side. + +### Message Construction + +Suppose there is a protobuf message defined like this: + +```proto +message Foo { + uint32 uint32 = 1; + bytes bytes = 2; + oneof union { + string string = 4; + MyMessage message = 5; + } + enum Kind { … }; + Kind kind = 9; +} +``` + +Here is an example of how to construct this message from literal values: + + + + + + + + + + +
Open Struct API (old)Opaque API (new)
+ +```go +m := &pb.Foo{ + Uint32: proto.Uint32(5), + Bytes: []byte("hello"), +} +``` + + + +```go +m := pb.Foo_builder{ + Uint32: proto.Uint32(5), + Bytes: []byte("hello"), +}.Build() +``` + +
+ +As you can see, the builder structs allow for an almost 1:1 translation between +Open Struct API (old) and Opaque API (new). + +Generally, prefer using builders for readability. Only in rare cases, like +creating Protobuf messages in a hot inner loop, might it be preferable to use +setters instead of builders. See +[the Opaque API FAQ: Should I use builders or setters?](/reference/go/opaque-faq#builders-vs-setters) +for more detail. + +An exception to the above example is when working with [oneofs](#oneofs): The +Open Struct API (old) uses a wrapper struct type for each oneof case, whereas +the Opaque API (new) treats oneof fields like regular message fields: + + + + + + + + + + +
Open Struct API (old)Opaque API (new)
+ +```go +m := &pb.Foo{ + Uint32: myScalar, // could be nil + Union: &pb.Foo_String{myString}, + Kind: pb.Foo_SPECIAL_KIND.Enum(), +} +``` + + + +```go +m := pb.Foo_builder{ + Uint32: myScalar, + String: myString, + Kind: pb.Foo_SPECIAL_KIND.Enum(), +}.Build() +``` + +
+ +For the set of Go struct fields associated with a oneof union, only one field +may be populated. If multiple oneof case fields are populated, the last one (in +field declaration order in your .proto file) wins. + +### Scalar fields + +Suppose there is a message defined with a scalar field: + +```proto +message Artist { + int32 birth_year = 1; +} +``` + +Protobuf message fields for which Go uses scalar types (bool, int32, int64, +uint32, uint64, float32, float64, string, []byte, and enum) will have `Get` and +`Set` accessor methods. Fields with +[explicit presence](/programming-guides/field_presence/) +will also have `Has` and `Clear` methods. + +For a field of type `int32` named `birth_year`, the following accessor methods +will be generated for it: + +```go +func (m *Artist) GetBirthYear() int32 +func (m *Artist) SetBirthYear(v int32) +func (m *Artist) HasBirthYear() bool +func (m *Artist) ClearBirthYear() +``` + +`Get` returns a value for the field. If the field is not set or the message +receiver is nil, it returns the default value. The default value is the +[zero value](https://go.dev/ref/spec#The_zero_value), unless explicitly set with +the default option. + +`Set` stores the provided value into the field. It panics when called on a nil +message receiver. + +For bytes fields, calling `Set` with a nil []byte will be considered set. For +example, calling `Has` immediately after returns true. Calling `Get` immediately +after will return a zero-length slice (can either be nil or empty slice). Users +should use `Has` for determining presence and not rely on whether `Get` returns +nil. + +`Has` reports whether the field is populated. It returns false when called on a +nil message receiver. + +`Clear` clears the field. It panics when called on a nil message receiver. + +**Example code snippets using a string field in:** + + + + + + + + + + +
Open Struct API (old)Opaque API (new)
+ +```go +// Getting the value. +s := m.GetBirthYear() + +// Setting the field. +m.BirthYear = proto.Int32(1989) + +// Check for presence. +if s.BirthYear != nil { … } + +// Clearing the field. +m.BirthYear = nil +``` + + + +```go +// Getting the field value. +s := m.GetBirthYear() + +// Setting the field. +m.SetBirthYear(1989) + +// Check for presence. +if m.HasBirthYear() { … } + +// Clearing the field +m.ClearBirthYear() +``` + +
+ +### Message fields + +Suppose there is a message defined with a message-typed field: + +```proto +message Band {} + +message Concert { + Band headliner = 1; +} +``` + +Protobuf message fields of type message will have `Get`, `Set`, `Has` and +`Clear` methods. + +For a message-typed field named `headliner`, the following accessor methods will +be generated for it: + +```go +func (m *Concert) GetHeadliner() *Band +func (m *Concert) SetHeadliner(*Band) +func (m *Concert) HasHeadliner() bool +func (m *Concert) ClearHeadliner() +``` + +`Get` returns a value for the field. It returns nil if not set or when called on +a nil message receiver. Checking if `Get` returns nil is equivalent to checking +if `Has` returns false. + +`Set` stores the provided value into the field. It panics when called on a nil +message receiver. Calling `Set` with a nil pointer is equivalent to calling +`Clear`. + +`Has` reports whether the field is populated. It returns false when called on a +nil message receiver. + +`Clear` clears the field. It panics when called on a nil message receiver. + +**Example code snippets** + + + + + + + + + + +
Open Struct API (old)Opaque (new)
+ +```go +// Getting the value. +b := m.GetHeadliner() + +// Setting the field. +m.Headliner = &pb.Band{} + +// Check for presence. +if s.Headliner != nil { … } + +// Clearing the field. +m.Headliner = nil +``` + + + +```go +// Getting the value. +s := m.GetHeadliner() + +// Setting the field. +m.SetHeadliner(&pb.Band{}) + +// Check for presence. +if m.HasHeadliner() { … } + +// Clearing the field +m.ClearHeadliner() +``` + +
+ +### Repeated fields + +Suppose there is a message defined with a repeated message-typed field: + +```proto +message Concert { + repeated Band support_acts = 2; +} +``` + +Repeated fields will have `Get` and `Set` methods. + +`Get` returns a value for the field. It returns nil if the field is not set or +the message receiver is nil. + +`Set` stores the provided value into the field. It panics when called on a nil +message receiver. `Set` will store a copy of the slice header that is provided. +Changes to the slice contents are observable in the repeated field. Hence, if +`Set` is called with an empty slice, calling `Get` immediately after will return +the same slice. For the wire or text marshaling output, a passed-in nil slice is +indistinguishable from an empty slice. + +For a repeated message-typed field named `support_acts` on message `Concert`, +the following accessor methods will be generated for it: + +```go +func (m *Concert) GetSupportActs() []*Band +func (m *Concert) SetSupportActs([]*Band) +``` + +**Example code snippets** + + + + + + + + + + +
Open Struct API (old)Opaque API (new)
+ +```go +// Getting the entire repeated value. +v := m.GetSupportActs() + +// Setting the field. +m.SupportActs = v + +// Get an element in a repeated field. +e := m.SupportActs[i] + +// Set an element in a repeated field. +m.SupportActs[i] = e + +// Get the length of a repeated field. +n := len(m.GetSupportActs()) + +// Truncate a repeated field. +m.SupportActs = m.SupportActs[:i] + +// Append to a repeated field. +m.SupportActs = append(m.GetSupportActs(), e) +m.SupportActs = append(m.GetSupportActs(), v...) + +// Clearing the field. +m.SupportActs = nil +``` + + + +```go +// Getting the entire repeated value. +v := m.GetSupportActs() + +// Setting the field. +m.SetSupportActs(v) + +// Get an element in a repeated field. +e := m.GetSupportActs()[i] + +// Set an element in a repeated field. +m.GetSupportActs()[i] = e + +// Get the length of a repeated field. +n := len(m.GetSupportActs()) + +// Truncate a repeated field. +m.SetSupportActs(m.GetSupportActs()[:i]) + +// Append to a repeated field. +m.SetSupportActs(append(m.GetSupportActs(), e)) +m.SetSupportActs(append(m.GetSupportActs(), v...)) + +// Clearing the field. +m.SetSupportActs(nil) +``` + +
+ +### Maps + +Suppose there is a message defined with a map-typed field: + +```proto +message MerchBooth { + map items = 1; +} +``` + +Map fields will have `Get` and `Set` methods. + +`Get` returns a value for the field. It returns nil if the field is not set or +the message receiver is nil. + +`Set` stores the provided value into the field. It panics when called on a nil +message receiver. `Set` will store a copy of the provided map reference. Changes +to the provided map are observable in the map field. + +For a map field named `items` on message `MerchBooth`, the following accessor +methods will be generated for it: + +```go +func (m *MerchBooth) GetItems() map[string]*MerchItem +func (m *MerchBooth) SetItems(map[string]*MerchItem) +``` + +**Example code snippets** + + + + + + + + + + +
Open Struct API (old)Opaque API (new)
+ +```go +// Getting the entire map value. +v := m.GetItems() + +// Setting the field. +m.Items = v + +// Get an element in a map field. +v := m.Items[k] + +// Set an element in a map field. +// This will panic if m.Items is nil. +// You should check m.Items for nil +// before doing the assignment to ensure +// it won't panic. +m.Items[k] = v + +// Delete an element in a map field. +delete(m.Items, k) + +// Get the size of a map field. +n := len(m.GetItems()) + +// Clearing the field. +m.Items = nil +``` + + + +```go +// Getting the entire map value. +v := m.GetItems() + +// Setting the field. +m.SetItems(v) + +// Get an element in a map field. +v := m.GetItems()[k] + +// Set an element in a map field. +// This will panic if m.GetItems() is nil. +// You should check m.GetItems() for nil +// before doing the assignment to ensure +// it won't panic. +m.GetItems()[k] = v + +// Delete an element in a map field. +delete(m.GetItems(), k) + +// Get the size of a map field. +n := len(m.GetItems()) + +// Clearing the field. +m.SetItems(nil) +``` + +
+ +### Oneofs + +For each oneof union grouping, there will be a `Which`, `Has` and `Clear` method +on the message. There will also be a `Get`, `Set`, `Has`, and `Clear` method on +each oneof case field in that union. + +Suppose there is a message defined with oneof fields `image_url` and +`image_data` in oneof `avatar` like: + +```proto +message Profile { + oneof avatar { + string image_url = 1; + bytes image_data = 2; + } +} +``` + +The generated Opaque API for this oneof will be: + +```go +func (m *Profile) WhichAvatar() case_Profile_Avatar { … } +func (m *Profile) HasAvatar() bool { … } +func (m *Profile) ClearAvatar() { … } + +type case_Profile_Avatar protoreflect.FieldNumber + +const ( + Profile_Avatar_not_set_case case_Profile_Avatar = 0 + Profile_ImageUrl_case case_Profile_Avatar = 1 + Profile_ImageData_case case_Profile_Avatar = 2 +) +``` + +`Which` reports which case field is set by returning the field number. It +returns 0 when none are set or when called on a nil message receiver. + +`Has` reports whether any of the fields within the oneof is set. It returns +false when called on a nil message receiver. + +`Clear` clears the currently set case field in the oneof. It panics on a nil +message receiver. + +The generated Opaque API for each oneof case field will be: + +```go +func (m *Profile) GetImageUrl() string { … } +func (m *Profile) GetImageData() []byte { … } + +func (m *Profile) SetImageUrl(v string) { … } +func (m *Profile) SetImageData(v []byte) { … } + +func (m *Profile) HasImageUrl() bool { … } +func (m *Profile) HasImageData() bool { … } + +func (m *Profile) ClearImageUrl() { … } +func (m *Profile) ClearImageData() { … } +``` + +`Get` returns a value for the case field. It will return the zero value if the +case field is not set or when called on a nil message receiver. + +`Set` stores the provided value into the case field. It also implicitly clears +the case field that was previously populated within the oneof union. Calling +`Set` on a oneof message case field with nil value will set the field to an +empty message. It panics when called on a nil message receiver. + +`Has` reports whether the case field is set or not. It returns false when called +on a nil message receiver. + +`Clear` clears the case field. If it was previously set, the oneof union is also +cleared. If the oneof union is set to different field, it will not clear the +oneof union. It panics when called on a nil message receiver. + +**Example code snippets** + + + + + + + + + + + +
Open Struct API (old)Opaque API (new)
+ +```go +// Getting the oneof field that is set. +switch m.GetAvatar().(type) { +case *pb.Profile_ImageUrl: + … = m.GetImageUrl() +case *pb.Profile_ImageData: + … = m.GetImageData() +} + +// Setting the fields. +m.Avatar = &pb.Profile_ImageUrl{"http://"} +m.Avatar = &pb.Profile_ImageData{img} + +// Checking whether any oneof field is set +if m.Avatar != nil { … } + +// Clearing the field. +m.Avatar = nil + +// Checking if a specific field is set. +_, ok := m.GetAvatar().(*pb.Profile_ImageUrl) +if ok { … } + +// Clearing a specific field +_, ok := m.GetAvatar().(*pb.Profile_ImageUrl) +if ok { + m.Avatar = nil +} + +// Copy a oneof field. +m.Avatar = src.Avatar +``` + + + +```go +// Getting the oneof field that is set. +switch m.WhichAvatar() { +case pb.Profile_ImageUrl_case: + … = m.GetImageUrl() +case pb.Profile_ImageData_case: + … = m.GetImageData() +} + +// Setting the fields. +m.SetImageUrl("http://") +m.SetImageData([]byte("…")) + +// Checking whether any oneof field is set +if m.HasAvatar() { … } + +// Clearing the field. +m.ClearAvatar() + +// Checking if a specific field is set. +if m.HasImageUrl() { … } + +// Clearing a specific field. +m.ClearImageUrl() + +// Copy a oneof field +switch src.WhichAvatar() { +case pb.Profile_ImageUrl_case: + m.SetImageUrl(src.GetImageUrl()) +case pb.Profile_ImageData_case: + m.SetImageData(src.GetImageData()) +} +``` + +
+ +### Reflection + +Code that use Go `reflect` package on proto message types to access struct +fields and tags will no longer work when migrating away from the Open Struct +API. Code will need to migrate to use +[protoreflect](https://pkg.go.dev/google.golang.org/protobuf/reflect/protoreflect) + +Some common libraries do use Go `reflect` under the hood, examples are: + +* [encoding/json](https://pkg.go.dev/encoding/json/) + * Use + [protobuf/encoding/protojson](https://pkg.go.dev/google.golang.org/protobuf/encoding/protojson). +* [pretty](https://pkg.go.dev/github.com/kr/pretty) +* [cmp](https://pkg.go.dev/github.com/google/go-cmp/cmp) + * To use `cmp.Equal` properly with protobuf messages, use + [protocmp.Transform](https://pkg.go.dev/google.golang.org/protobuf/testing/protocmp#Transform) diff --git a/content/reference/go/opaque-migration.md b/content/reference/go/opaque-migration.md new file mode 100644 index 000000000..7d0868422 --- /dev/null +++ b/content/reference/go/opaque-migration.md @@ -0,0 +1,213 @@ ++++ +title = "Go Opaque API Migration" +weight = 650 +linkTitle = "Opaque API Migration" +description = "Describes the automated migration to the Opaque API." +type = "docs" ++++ + +The Opaque API is the latest version of the Protocol Buffers implementation for +the Go programming language. The old version is now called Open Struct API. See +the [Go Protobuf: Releasing the Opaque API](https://go.dev/blog/protobuf-opaque) +blog post for an introduction. + +The migration to the Opaque API happens incrementally, on a per-proto-message or +per-`.proto`-file basis, by setting the Protobuf Editions feature `api_level` +option to one of its possible values: + +* `API_OPEN` selects the Open Struct API; this was the only API before + December 2024. +* `API_HYBRID` is a step between Open and Opaque: The Hybrid API also includes + accessor methods (so you can update your code), but still exports the struct + fields as before. There is no performance difference; this API level only + helps with the migration. +* `API_OPAQUE` selects the Opaque API. + +Today, the default is `API_OPEN`, but the upcoming +[Protobuf Edition 2024](/editions/overview) will change +the default to `API_OPAQUE`. + +To use the Opaque API before Edition 2024, set the `api_level` like so: + +```proto +edition = "2023"; + +package log; + +import "google/protobuf/go_features.proto"; +option features.(pb.go).api_level = API_OPAQUE; + +message LogEntry { … } +``` + +Before you can change the `api_level` to `API_OPAQUE` for existing files, all +existing usages of the generated proto code need to be updated. The +`open2opaque` tool helps with this. + +For your convenience, you can also override the default API level with a +`protoc` command-line flag: + +``` +protoc […] --go_opt=default_api_level=API_OPAQUE +``` + +To override the default API level for a specific file (instead of all files), +use the `apilevelM` mapping flag (similar to +[the `M` flag for import paths](/reference/go/go-generated/#package)): + +``` +protoc […] --go_opt=apilevelMhello.proto=API_OPAQUE +``` + +The command-line flags also work for `.proto` files still using proto2 or proto3 +syntax, but if you want to select the API level from within the `.proto` file, +you need to migrate said file to editions first. + +## Automated migration {#automated} + +We try to make migrating existing projects to the Opaque API as easy as possible +for you: our open2opaque tool does most of the work! + +To install the migration tool, use: + +``` +go install google.golang.org/open2opaque@latest +``` + +{{% alert title="Note" color="info" %}}If +you encounter any issues with the automated migration approach, refer to the +[Opaque API: Manual Migration](/reference/go/opaque-migration-manual) +guide. {{% /alert %}} + +### Project Preparation {#projectprep} + +Ensure your build environment and project are using recent-enough versions of +Protocol Buffers and Go Protobuf: + +1. Update the protobuf compiler (protoc) from + [the protobuf release page](https://github.com/protocolbuffers/protobuf/releases/latest) + to version 29.0 or newer. + +1. Update the protobuf compiler Go plugin (protoc-gen-go) to version 1.36.0 or + newer: + + ``` + go install google.golang.org/protobuf/cmd/protoc-gen-go@latest + ``` + +1. In each project, update the `go.mod` file to use the protobuf module in + version 1.36.0 or newer: + + ``` + go get google.golang.org/protobuf@latest + ``` + + **Note:** if you are not yet importing `google.golang.org/protobuf`, you + might still be on an older module. See + [the `google.golang.org/protobuf` announcement (from 2020)](https://go.dev/blog/protobuf-apiv2) + and migrate your code before returning to this page. + +### Step 1. Switch to the Hybrid API {#setup} + +Use the `open2opaque` tool to switch your `.proto` files to the Hybrid API: + +``` +open2opaque setapi -api HYBRID $(find . -name "*.proto") +``` + +Your existing code will continue to build. The Hybrid API is a step between the +Open and Opaque API which adds the new accessor methods but keeps struct fields +visible. + +### Step 2. `open2opaque rewrite` {#rewrite} + +To rewrite your Go code to use the Opaque API, run the `open2opaque rewrite` +command: + +``` +open2opaque rewrite -levels=red github.com/robustirc/robustirc/... +``` + +You can specify one or more +[packages or patterns](https://pkg.go.dev/cmd/go#hdr-Package_lists_and_patterns). + +As an example, if you had code like this: + +```go +logEntry := &logpb.LogEntry{} +if req.IPAddress != nil { + logEntry.IPAddress = redactIP(req.IPAddress) +} +logEntry.BackendServer = proto.String(host) +``` + +The tool would rewrite it to use accessors: + +```go +logEntry := &logpb.LogEntry{} +if req.HasIPAddress() { + logEntry.SetIPAddress(redactIP(req.GetIPAddress())) +} +logEntry.SetBackendServer(host) +``` + +Another common example is to initialize a protobuf message with a struct +literal: + +```go +return &logpb.LogEntry{ + BackendServer: proto.String(host), +} +``` + +In the Opaque API, the equivalent is to use a Builder: + +```go +return logpb.LogEntry_builder{ + BackendServer: proto.String(host), +}.Build() +``` + +The tool classifies its available rewrites into different levels. The +`-levels=red` argument enables all rewrites, including those that require human +review. The following levels are available: + +* green: Safe rewrites (high + confidence). Includes most changes the tool makes. These changes do not + require a close look and could even be submitted by automation, without any + human oversight. +* yellow: (reasonable + confidence) These rewrites require human review. They should be correct, but + please review them. +* red: Potentially dangerous + rewrites, changing rare and complicated patterns. These require careful + human review. For example, when an existing function takes a `*string` + parameter, the typical fix of using `proto.String(msg.GetFoo())` does not + work if the function meant to change the field value by writing to the + pointer (`*foo = "value"`). + +Many programs can be fully migrated with only green changes. Before you can +migrate a proto message or file to the Opaque API, you need to complete all +rewrites of all levels, at which point no direct struct access remains in your +code. + +### Step 3. Migrate and Verify {#migrate-and-verify} + +To complete the migration, use the `open2opaque` tool to switch your `.proto` +files to the Opaque API: + +``` +open2opaque setapi -api OPAQUE $(find . -name "*.proto") +``` + +Now, any remaining code that was not rewritten to the Opaque API yet will no +longer compile. + +Run your unit tests, integration tests and other verification steps, if any. + +## Questions? Issues? + +First, check out the +[Opaque API FAQ](/reference/go/opaque-faq). If that +doesn't answer your question or resolve your issue, see +[Where can I ask questions or report issues?](/reference/go/opaque-faq#questions)