From 20c5dce144ba6c6c2af17be2cbdcbe7fb489f9b8 Mon Sep 17 00:00:00 2001 From: Alex Stephen Date: Mon, 18 Mar 2024 22:08:45 -0700 Subject: [PATCH 01/16] adopt AEP-202 --- aep/general/0202/aep.md.j2 | 112 +++++++++++++++++++++++++++++++++++++ aep/general/0202/aep.yaml | 7 +++ 2 files changed, 119 insertions(+) create mode 100644 aep/general/0202/aep.md.j2 create mode 100644 aep/general/0202/aep.yaml diff --git a/aep/general/0202/aep.md.j2 b/aep/general/0202/aep.md.j2 new file mode 100644 index 00000000..716fbe53 --- /dev/null +++ b/aep/general/0202/aep.md.j2 @@ -0,0 +1,112 @@ +# Fields + +The [`google.api.FieldInfo`][field info proto] type, through its accompanying +extension `google.api.field_info`, enriches a field's schema beyond the basic +name and type information. + +## Guidance + +Decorating a field with `google.api.field_info` is only necessary when +explicitly stated in this AEP or another that leverages `google.api.FieldInfo` +information. As such, the guidance herein applies to those scenarios as well. + +### Format + +Fields with a primitive type can still have a specific format. To convey that +type format, the `FieldInfo.Format` enumeration is used via the +`(google.api.field_info).format` extension field. The following guidance conveys +the meaning of and requirements for use of each `FieldInfo.Format` value. + +#### UUID4 + +The `UUID4` format represents a UUID version 4 value as governed by +[RFC 4122][]. It **must** only be used on a field of type `string`. + +Such a value **may** be normalized by the service to entirely lowercase letters. +For example, the value `F47AC10B-58CC-0372-8567-0E02B2C3D479` would be +normalized to `f47ac10b-58cc-0372-8567-0e02b2c3d479`. + +As such, equivalence comparison **must not** be done via primitive text +comparison. Instead, an [RFC 4122][] compliant implementation **must** be used. + +#### IPv4 + +The `IPV4` format represents an IP v4 address as governed by [RFC 791][]. It +**must** only be used on a field of type `string`. + +Such a value **may** be condensed by the service, with leading zeros in each +octet stripped. For example, `001.022.233.040` would be condensed to +`1.22.233.40`. + +As such, equivalence comparison **must not** be done via primitive text +comparison. Instead, an [RFC 791][] compliant implementation **must** be used. + +#### IPv6 + +The `IPV6` format represents an IP v6 address as governed by [RFC 4291][]. It +**must** only be used on a field of type `string`. + +Such a value **may** be normalized by the service to entirely lowercase letters +with zeros compressed, following [RFC 5952][]. For example, the value +`2001:0DB8:0::0` would be normalized to `2001:db8::`. + +As such, equivalence comparison **must not** be done via primitive text +comparison. Instead, an [RFC 4291][] compliant implementation **must** be used. + +#### IPv4 or IPv6 + +The `IPV4_OR_IPV6` value indicates that the field can be either an IP v4 or v6 +address, as described in the [IPv4](#ipv4) and [IPv6](#ipv6) sections. + +#### Format Compatibility + +Adding a format specifier to an existing, unspecified field **is not** backwards +compatible, *unless* the field in question has always conformed to the format +being specified. + +Changing an existing format specifier to a different one in all cases **is not** +backwards compatible. + +#### Extending Format + +Any new `FieldInfo.Format` value **must** be governed by an +[IETF-approved RFC][ietf rfc] or a [Google-approved AEP](./0001.md). + +## Rationale + +#### Why add a format specifier? + +The format of a primitive-typed field can be critical to its usability. Some +programming languages may convey a specific type format as a standalone type, +as Java does with [UUID][java uuid]. Most have specific structural requirements +that are validated by the service, so conveying the format to the user ahead of +time is critical to their experience. + +#### Why discourage primitive equality comparisons? + +The text representations of the supported formats have many nuances and +transforming the value into a canonical representation is non-trivial. As such, +aligning implementations between each consumer and each service without any +issue is infeasiable. + +#### Why document value normalizations? + +While primitive comparison is not recommended for any of the supported formats, +uniform normalization of values is important to set consumer expectations, and +create a user-friendly surface. + +#### Why require an RFC or AEP for new formats? + +Those formats which are sufficiently standardized to merit an RFC or AEP are +stable enough and widely enough known to be incorporated as a supported value +and see usage in Google APIs. Requiring such extra guidance means that governing +the format specification is not the responsibility of the `FieldInfo.Format` +enumeration itself. + +[field info proto]: https://github.com/googleapis/googleapis/blob/master/google/api/field_info.proto +[rfc 4122]: https://datatracker.ietf.org/doc/html/rfc4122 +[rfc 791]: https://datatracker.ietf.org/doc/html/rfc791 +[rfc 4291]: https://datatracker.ietf.org/doc/html/rfc4291#section-2.2 +[rfc 5952]: https://datatracker.ietf.org/doc/html/rfc5952 +[ietf rfc]: https://www.ietf.org/standards/rfcs +[java uuid]: https://docs.oracle.com/javase/8/docs/api/java/util/UUID.html \ No newline at end of file diff --git a/aep/general/0202/aep.yaml b/aep/general/0202/aep.yaml new file mode 100644 index 00000000..d1242a21 --- /dev/null +++ b/aep/general/0202/aep.yaml @@ -0,0 +1,7 @@ +--- +id: 202 +state: reviewing +slug: fields +created: 2023-01-22 +placement: + category: fields From 671cf6ec025923ce01e31b42a90be4311c40efdc Mon Sep 17 00:00:00 2001 From: Alex Stephen Date: Fri, 12 Apr 2024 15:38:28 -0700 Subject: [PATCH 02/16] change format --- aep/general/0202/aep.md.j2 | 72 ++++---------------------------------- 1 file changed, 6 insertions(+), 66 deletions(-) diff --git a/aep/general/0202/aep.md.j2 b/aep/general/0202/aep.md.j2 index 716fbe53..982364af 100644 --- a/aep/general/0202/aep.md.j2 +++ b/aep/general/0202/aep.md.j2 @@ -1,62 +1,13 @@ # Fields -The [`google.api.FieldInfo`][field info proto] type, through its accompanying -extension `google.api.field_info`, enriches a field's schema beyond the basic +The [`buf.validate.field`][protovalidate] type, through its accompanying +extension `buf.validate.field`, enriches a field's schema beyond the basic name and type information. ## Guidance -Decorating a field with `google.api.field_info` is only necessary when -explicitly stated in this AEP or another that leverages `google.api.FieldInfo` -information. As such, the guidance herein applies to those scenarios as well. - -### Format - -Fields with a primitive type can still have a specific format. To convey that -type format, the `FieldInfo.Format` enumeration is used via the -`(google.api.field_info).format` extension field. The following guidance conveys -the meaning of and requirements for use of each `FieldInfo.Format` value. - -#### UUID4 - -The `UUID4` format represents a UUID version 4 value as governed by -[RFC 4122][]. It **must** only be used on a field of type `string`. - -Such a value **may** be normalized by the service to entirely lowercase letters. -For example, the value `F47AC10B-58CC-0372-8567-0E02B2C3D479` would be -normalized to `f47ac10b-58cc-0372-8567-0e02b2c3d479`. - -As such, equivalence comparison **must not** be done via primitive text -comparison. Instead, an [RFC 4122][] compliant implementation **must** be used. - -#### IPv4 - -The `IPV4` format represents an IP v4 address as governed by [RFC 791][]. It -**must** only be used on a field of type `string`. - -Such a value **may** be condensed by the service, with leading zeros in each -octet stripped. For example, `001.022.233.040` would be condensed to -`1.22.233.40`. - -As such, equivalence comparison **must not** be done via primitive text -comparison. Instead, an [RFC 791][] compliant implementation **must** be used. - -#### IPv6 - -The `IPV6` format represents an IP v6 address as governed by [RFC 4291][]. It -**must** only be used on a field of type `string`. - -Such a value **may** be normalized by the service to entirely lowercase letters -with zeros compressed, following [RFC 5952][]. For example, the value -`2001:0DB8:0::0` would be normalized to `2001:db8::`. - -As such, equivalence comparison **must not** be done via primitive text -comparison. Instead, an [RFC 4291][] compliant implementation **must** be used. - -#### IPv4 or IPv6 - -The `IPV4_OR_IPV6` value indicates that the field can be either an IP v4 or v6 -address, as described in the [IPv4](#ipv4) and [IPv6](#ipv6) sections. +Decorating a field with `buf.validate.field` is only necessary when a field +consistently follows a set pattern. #### Format Compatibility @@ -67,11 +18,6 @@ being specified. Changing an existing format specifier to a different one in all cases **is not** backwards compatible. -#### Extending Format - -Any new `FieldInfo.Format` value **must** be governed by an -[IETF-approved RFC][ietf rfc] or a [Google-approved AEP](./0001.md). - ## Rationale #### Why add a format specifier? @@ -100,13 +46,7 @@ create a user-friendly surface. Those formats which are sufficiently standardized to merit an RFC or AEP are stable enough and widely enough known to be incorporated as a supported value and see usage in Google APIs. Requiring such extra guidance means that governing -the format specification is not the responsibility of the `FieldInfo.Format` +the format specification is not the responsibility of the `buf.validate.field` enumeration itself. -[field info proto]: https://github.com/googleapis/googleapis/blob/master/google/api/field_info.proto -[rfc 4122]: https://datatracker.ietf.org/doc/html/rfc4122 -[rfc 791]: https://datatracker.ietf.org/doc/html/rfc791 -[rfc 4291]: https://datatracker.ietf.org/doc/html/rfc4291#section-2.2 -[rfc 5952]: https://datatracker.ietf.org/doc/html/rfc5952 -[ietf rfc]: https://www.ietf.org/standards/rfcs -[java uuid]: https://docs.oracle.com/javase/8/docs/api/java/util/UUID.html \ No newline at end of file +[protovalidate]: https://github.com/bufbuild/protovalidate/tree/main/proto/protovalidate \ No newline at end of file From 29efd039f165de2f9122f74f756683389a996820 Mon Sep 17 00:00:00 2001 From: Alex Stephen Date: Fri, 12 Apr 2024 15:41:04 -0700 Subject: [PATCH 03/16] proto --- aep/general/0202/aep.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/aep/general/0202/aep.yaml b/aep/general/0202/aep.yaml index d1242a21..2ed08238 100644 --- a/aep/general/0202/aep.yaml +++ b/aep/general/0202/aep.yaml @@ -4,4 +4,4 @@ state: reviewing slug: fields created: 2023-01-22 placement: - category: fields + category: protobuf From 64bd912abc96b1bf7b8bdeabcf865a00a424233e Mon Sep 17 00:00:00 2001 From: Alex Stephen <1325798+rambleraptor@users.noreply.github.com> Date: Sun, 19 May 2024 17:22:44 -0700 Subject: [PATCH 04/16] Update aep/general/0202/aep.md.j2 Co-authored-by: Richard Frankel --- aep/general/0202/aep.md.j2 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/aep/general/0202/aep.md.j2 b/aep/general/0202/aep.md.j2 index 982364af..d89cdc4e 100644 --- a/aep/general/0202/aep.md.j2 +++ b/aep/general/0202/aep.md.j2 @@ -1,6 +1,6 @@ # Fields -The [`buf.validate.field`][protovalidate] type, through its accompanying +The [`buf.validate.FieldConstraints`][protovalidate] type, through its accompanying extension `buf.validate.field`, enriches a field's schema beyond the basic name and type information. From bbdfbd1ffd67d7dd9ece88f6fd42503aacfb7663 Mon Sep 17 00:00:00 2001 From: Alex Stephen <1325798+rambleraptor@users.noreply.github.com> Date: Sun, 19 May 2024 17:22:52 -0700 Subject: [PATCH 05/16] Update aep/general/0202/aep.md.j2 Co-authored-by: Richard Frankel --- aep/general/0202/aep.md.j2 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/aep/general/0202/aep.md.j2 b/aep/general/0202/aep.md.j2 index d89cdc4e..4044f8a4 100644 --- a/aep/general/0202/aep.md.j2 +++ b/aep/general/0202/aep.md.j2 @@ -2,7 +2,7 @@ The [`buf.validate.FieldConstraints`][protovalidate] type, through its accompanying extension `buf.validate.field`, enriches a field's schema beyond the basic -name and type information. +name and type information by specifying constraints or validation rules. ## Guidance From 7d015e6c043112cd2b3118a8e916ebaaefee19a9 Mon Sep 17 00:00:00 2001 From: Alex Stephen <1325798+rambleraptor@users.noreply.github.com> Date: Sun, 19 May 2024 17:22:58 -0700 Subject: [PATCH 06/16] Update aep/general/0202/aep.md.j2 Co-authored-by: Richard Frankel --- aep/general/0202/aep.md.j2 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/aep/general/0202/aep.md.j2 b/aep/general/0202/aep.md.j2 index 4044f8a4..18addee2 100644 --- a/aep/general/0202/aep.md.j2 +++ b/aep/general/0202/aep.md.j2 @@ -6,7 +6,7 @@ name and type information by specifying constraints or validation rules. ## Guidance -Decorating a field with `buf.validate.field` is only necessary when a field +Annotating a field with `buf.validate.field` is only necessary when a field consistently follows a set pattern. #### Format Compatibility From 9e8378f459d7f35bb4907a04d861b6ebb3726554 Mon Sep 17 00:00:00 2001 From: Alex Stephen <1325798+rambleraptor@users.noreply.github.com> Date: Sun, 19 May 2024 17:23:29 -0700 Subject: [PATCH 07/16] Update aep/general/0202/aep.md.j2 Co-authored-by: Richard Frankel --- aep/general/0202/aep.md.j2 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/aep/general/0202/aep.md.j2 b/aep/general/0202/aep.md.j2 index 18addee2..80c5b317 100644 --- a/aep/general/0202/aep.md.j2 +++ b/aep/general/0202/aep.md.j2 @@ -9,7 +9,7 @@ name and type information by specifying constraints or validation rules. Annotating a field with `buf.validate.field` is only necessary when a field consistently follows a set pattern. -#### Format Compatibility +#### Backwards compatibility Adding a format specifier to an existing, unspecified field **is not** backwards compatible, *unless* the field in question has always conformed to the format From d7855aba32a616050c83a988c612d8e87f5fc6c7 Mon Sep 17 00:00:00 2001 From: Alex Stephen <1325798+rambleraptor@users.noreply.github.com> Date: Sun, 19 May 2024 17:23:37 -0700 Subject: [PATCH 08/16] Update aep/general/0202/aep.md.j2 Co-authored-by: Richard Frankel --- aep/general/0202/aep.md.j2 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/aep/general/0202/aep.md.j2 b/aep/general/0202/aep.md.j2 index 80c5b317..4301a9c1 100644 --- a/aep/general/0202/aep.md.j2 +++ b/aep/general/0202/aep.md.j2 @@ -15,7 +15,7 @@ Adding a format specifier to an existing, unspecified field **is not** backwards compatible, *unless* the field in question has always conformed to the format being specified. -Changing an existing format specifier to a different one in all cases **is not** +Changing an existing validation rule to a different one in all cases **is not** backwards compatible. ## Rationale From 511125d34ab829508e9c688f14120fae443070ec Mon Sep 17 00:00:00 2001 From: Alex Stephen <1325798+rambleraptor@users.noreply.github.com> Date: Fri, 31 May 2024 11:18:15 -0700 Subject: [PATCH 09/16] Update aep/general/0202/aep.md.j2 Co-authored-by: Richard Frankel --- aep/general/0202/aep.md.j2 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/aep/general/0202/aep.md.j2 b/aep/general/0202/aep.md.j2 index 4301a9c1..b3764693 100644 --- a/aep/general/0202/aep.md.j2 +++ b/aep/general/0202/aep.md.j2 @@ -11,7 +11,7 @@ consistently follows a set pattern. #### Backwards compatibility -Adding a format specifier to an existing, unspecified field **is not** backwards +Adding a validation rule to an existing, unspecified field **is not** backwards compatible, *unless* the field in question has always conformed to the format being specified. From 302d83a0dfe7e725db19fa6b16eea809f5eb987e Mon Sep 17 00:00:00 2001 From: Alex Stephen <1325798+rambleraptor@users.noreply.github.com> Date: Fri, 31 May 2024 11:18:21 -0700 Subject: [PATCH 10/16] Update aep/general/0202/aep.md.j2 Co-authored-by: Richard Frankel --- aep/general/0202/aep.md.j2 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/aep/general/0202/aep.md.j2 b/aep/general/0202/aep.md.j2 index b3764693..0967d74d 100644 --- a/aep/general/0202/aep.md.j2 +++ b/aep/general/0202/aep.md.j2 @@ -12,7 +12,7 @@ consistently follows a set pattern. #### Backwards compatibility Adding a validation rule to an existing, unspecified field **is not** backwards -compatible, *unless* the field in question has always conformed to the format +compatible, *unless* the field in question has always been documented as conforming to the format being specified. Changing an existing validation rule to a different one in all cases **is not** From 4aa9c22118b4e6dfc260eece1a14ce512f177d11 Mon Sep 17 00:00:00 2001 From: Alex Stephen Date: Fri, 31 May 2024 11:23:45 -0700 Subject: [PATCH 11/16] guidance --- aep/general/0202/aep.md.j2 | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/aep/general/0202/aep.md.j2 b/aep/general/0202/aep.md.j2 index 0967d74d..68310eeb 100644 --- a/aep/general/0202/aep.md.j2 +++ b/aep/general/0202/aep.md.j2 @@ -6,8 +6,14 @@ name and type information by specifying constraints or validation rules. ## Guidance -Annotating a field with `buf.validate.field` is only necessary when a field -consistently follows a set pattern. +If a field follows a set pattern, it **should** be annotated with a `buf.validate.field` annotation. + +If a field has a `buf.validate.field` annotation, that annotation **must** be included on all requests and responses. + +If a field has a `buf.validate.field`, any values that do not follow the `buf.validate.field` annotation must return an error. + +If a field has a `buf.validate.field`, an API **must** return an error if it attempts to return a value that does not follow the annotation. + #### Backwards compatibility @@ -49,4 +55,4 @@ and see usage in Google APIs. Requiring such extra guidance means that governing the format specification is not the responsibility of the `buf.validate.field` enumeration itself. -[protovalidate]: https://github.com/bufbuild/protovalidate/tree/main/proto/protovalidate \ No newline at end of file +[protovalidate]: https://github.com/bufbuild/protovalidate/tree/main/proto/protovalidate From fafbd0d7bb2a59d5672aea42351114e17c3e2c26 Mon Sep 17 00:00:00 2001 From: Alex Stephen Date: Fri, 31 May 2024 11:24:34 -0700 Subject: [PATCH 12/16] remove some guidance --- aep/general/0202/aep.md.j2 | 8 -------- 1 file changed, 8 deletions(-) diff --git a/aep/general/0202/aep.md.j2 b/aep/general/0202/aep.md.j2 index 68310eeb..530aca50 100644 --- a/aep/general/0202/aep.md.j2 +++ b/aep/general/0202/aep.md.j2 @@ -47,12 +47,4 @@ While primitive comparison is not recommended for any of the supported formats, uniform normalization of values is important to set consumer expectations, and create a user-friendly surface. -#### Why require an RFC or AEP for new formats? - -Those formats which are sufficiently standardized to merit an RFC or AEP are -stable enough and widely enough known to be incorporated as a supported value -and see usage in Google APIs. Requiring such extra guidance means that governing -the format specification is not the responsibility of the `buf.validate.field` -enumeration itself. - [protovalidate]: https://github.com/bufbuild/protovalidate/tree/main/proto/protovalidate From 5f38c6fe8093767b8fd4380fc342439daf0bf888 Mon Sep 17 00:00:00 2001 From: Alex Stephen <1325798+rambleraptor@users.noreply.github.com> Date: Fri, 14 Jun 2024 19:30:38 +0000 Subject: [PATCH 13/16] Richard changes --- aep/general/0202/aep.md.j2 | 7 ------- 1 file changed, 7 deletions(-) diff --git a/aep/general/0202/aep.md.j2 b/aep/general/0202/aep.md.j2 index 530aca50..5f7fccdd 100644 --- a/aep/general/0202/aep.md.j2 +++ b/aep/general/0202/aep.md.j2 @@ -34,13 +34,6 @@ as Java does with [UUID][java uuid]. Most have specific structural requirements that are validated by the service, so conveying the format to the user ahead of time is critical to their experience. -#### Why discourage primitive equality comparisons? - -The text representations of the supported formats have many nuances and -transforming the value into a canonical representation is non-trivial. As such, -aligning implementations between each consumer and each service without any -issue is infeasiable. - #### Why document value normalizations? While primitive comparison is not recommended for any of the supported formats, From 1eaa4e14e1d3713919b6af96d26ed7450daef746 Mon Sep 17 00:00:00 2001 From: Alex Stephen <1325798+rambleraptor@users.noreply.github.com> Date: Fri, 12 Jul 2024 11:55:38 -0700 Subject: [PATCH 14/16] Update aep.md.j2 --- aep/general/0202/aep.md.j2 | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/aep/general/0202/aep.md.j2 b/aep/general/0202/aep.md.j2 index 5f7fccdd..8d1938f4 100644 --- a/aep/general/0202/aep.md.j2 +++ b/aep/general/0202/aep.md.j2 @@ -1,5 +1,13 @@ # Fields +In many cases, a field's particular value can be represented in many different ways. +String representations of dates are an excellent example, where a date can be represented in multiple ways +and an API cannot necessarily normalize the data without additional information about its format. + +The allowed formats for a field's value must be properly documented and enforced so that the API +can ensure that it receives data in a way that it can properly handle and send it back in a way +that consumers expect. + The [`buf.validate.FieldConstraints`][protovalidate] type, through its accompanying extension `buf.validate.field`, enriches a field's schema beyond the basic name and type information by specifying constraints or validation rules. @@ -36,8 +44,11 @@ time is critical to their experience. #### Why document value normalizations? +In many cases, the API will be responsible for parsing the user input and normalizing it +into a value that the API expects. + While primitive comparison is not recommended for any of the supported formats, uniform normalization of values is important to set consumer expectations, and -create a user-friendly surface. +create a user-friendly surface. [protovalidate]: https://github.com/bufbuild/protovalidate/tree/main/proto/protovalidate From bd557c047a28eb5d2f0733ab104a662229d0bd70 Mon Sep 17 00:00:00 2001 From: Alex Stephen <1325798+rambleraptor@users.noreply.github.com> Date: Fri, 12 Jul 2024 12:00:54 -0700 Subject: [PATCH 15/16] Update aep.yaml --- aep/general/0202/aep.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/aep/general/0202/aep.yaml b/aep/general/0202/aep.yaml index 2ed08238..40d98d7f 100644 --- a/aep/general/0202/aep.yaml +++ b/aep/general/0202/aep.yaml @@ -1,7 +1,7 @@ --- id: 202 -state: reviewing +state: approved slug: fields -created: 2023-01-22 +created: 2023-07-12 placement: category: protobuf From 20dd825b5bda11eb58e7a180ec48207159ba5258 Mon Sep 17 00:00:00 2001 From: Alex Stephen <1325798+rambleraptor@users.noreply.github.com> Date: Fri, 12 Jul 2024 12:01:11 -0700 Subject: [PATCH 16/16] Update aep.yaml --- aep/general/0202/aep.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/aep/general/0202/aep.yaml b/aep/general/0202/aep.yaml index 40d98d7f..be5fe8e2 100644 --- a/aep/general/0202/aep.yaml +++ b/aep/general/0202/aep.yaml @@ -2,6 +2,6 @@ id: 202 state: approved slug: fields -created: 2023-07-12 +created: 2024-07-12 placement: category: protobuf