Merge branch 'main' into docs/update-readme-2024-04-04

docs/README.md

@@ -17,7 +17,7 @@ Godocs on exported library package code (such as `resource`, `operator`, `plugin
 | [Resource Stores](./resource-stores.md) | Describes the various "Store" types in the `resource` package, and why you may want to use one or another |
 | [Design Patterns](./design-patterns.md) | The typical design patterns of an app built with the SDK |
 | [Operators & Event-Based Design](./operators.md) | A brief primer on what operators/controllers are and working with event-based code |
-| [Custom Kinds & CUE](./custom-kinds.md) | A look at writing custom kinds in CUE |
+| [Custom Kinds](./custom-kinds/README.md) | What are kinds, how to write them, and how to use them |
 | [Code Generation](./code-generation.md) | How to use CUE and the CLI for code generation. |
 | [Local Dev Environment Setup](./local-development.md) | How to use the CLI to set up a local development & testing environment |
 | [Kubernetes Concepts](./kubernetes.md) | A primer on some kubernetes concepts which are relevant to using the SDK backed by a kubernetes API server |

docs/custom-kinds/README.md

@@ -0,0 +1,41 @@
+# Kinds
+
+## What is a Kind?
+
+A Kind is a concept [borrowed from Kubernetes](https://kubernetes.io/docs/concepts/overview/working-with-objects/), and is used to describe a type of object which can be instantiated. Think of it as a blueprint, or, in Object-Oriented Programming, a class. A Kind is a specification of the structure of an object, and instances (or objects, in OOP parlance) are called "Objects" or "Resources" (these are often used interchangably).
+
+A Kind is composed of several components:
+* A unique name (also, slightly confusingly, called Kind)
+* A Group it belongs to
+* One or more versions, and a schema for the `spec` and any subresources for each version
+
+(a `spec` is essentially the body of the resource: for a resource like a grafana dashboard, it would be the panels. Subresources are additional payloads that are not considered part of the main body of the resource, but may be used by applications for their purposes. They are returned alongside the `spec` in reads, but must be updated via a separate call, and can have different RBAC associated with them. In kubernetes, `status` is considered a subresource).
+
+In a kubernetes-compatible API server, a kind is identified by the top-level attributes `kind` and `apiVersion` (where the `apiVersion` consists of the `group` and `version`--the group is used to identify the kind, and the version the specific schema). A kind is sometimes referenced as the Group and Kind combination (called GroupKind), or as the totality of Group, Kind, and Version (called GroupVersionKind, or GVK), which is the way to uniquely identify the schema of a resource.
+
+Kinds are the core of apps, as they are the data structure for all app data. Apps interact with an API server to read, write, update, delete, list and watch kinds. For more on this, see [Operators and Event-Based Design](../operators.md).
+
+Kinds belong to groups, which generally correlate to apps. In kubernetes, your kind's identifier when quering the API is `<group>/<version>/<plural>`, with `group` being the kind's full group (excepting very specific circumstances, this is your app name + `.ext.grafana.com`), `version` being the version you wish to use, and `plural` being the plural name of the kind (unless changed, this defaults to `LOWER(<kind>) + 's')`).
+
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="../diagrams/kind-overview-dark.png">
+  <source media="(prefers-color-scheme: light)" srcset="../diagrams/kind-overview.png">
+  <img alt="A diagram of how aspects of a kind are encapsulated" src="../diagrams/kind-overview.png">
+</picture>
+
+
+## Writing Kinds
+
+In a typical app, kinds are created in the API server as a Custom Resource Definition, and in-code as instances of `resource.Kind` (containing implementations of `resource.Codec` and `resource.Object`). An app author can write out a kind in CUE and use the `grafana-app-sdk` CLI to turn that CUE into a Custom Resource Definition JSON (or YAML) file for applying to an API server, go code to use in their operator, and TypeScript code to use in their front-end. The document [Writing Kinds](./writing-kinds.md) discusses how to write kinds in CUE to use with the `grafana-app-sdk` CLI.
+
+Alternatively, an author can write a Custom Resource Definition (CRD) themselves, and write the go and TypeScript types needed to work with the SDK and API server on their own, implementing the relevant interfaces, but this method incurs much more toil and does not guarantee that all three representations of the kind are kept synchronized.
+
+## Kinds in-code
+
+In-code, kinds are used via `resource.Kind` and `resource.Object`, where `resource.Kind` contains kind metadata and the ability to marshal/unmarshal the kind into a `resource.Object`, and `resource.Object` is an interface which concrete types must implement.
+
+To learn more about how to use kinds in your app, check out [Using Kinds](./using-kinds.md).
+
+## Kind Versions
+
+An important part of kinds is that once a version is published, it creates an API contract so a user can request and interface with the kind schema for a specific version, rather than the schema changing as the app evolves (which would then require pinning a specific version of the app if a user wanted to use an older version of the API). If and when you need to update the schema of a kind, you add a new version to the kind (you can also have "mutable" versions, such as `v1alpha1`, in addition to immutable ones like `v1`). For best practices with kind versioning and how to support multiple kind versions in your app, see [Managing Multiple Kind Version](./managing-multiple-versions.md).

docs/custom-kinds/managing-multiple-versions.md

@@ -0,0 +1,206 @@
+# Managing Multiple Kind Versions
+
+There are two kinds of versions for a kind: a fixed version, such as `v1`, and a mutable, alpha or beta version, such as `v1alpha1`. When publishing a kind, be aware that you should never change the definition of a fixed version, as clients will rely on that kind version being constant when working with the API. For more details on kubernetes version conventions, see [Custom Resource Definitions Versions](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/).
+
+## Operators & Multiple Versions
+
+No matter how many versions you have, you should only have one watcher/reconciler per kind, as resource events are not local to a version. When you create an informer, you are specifying a specific version you want to consume _all_ events for the kind as, and your converter hook will be called to convert the object in each event into the requested version (if necessary). Typically, this means that you want your watcher/reconciler to consume the latest version, as it will usually contain the most information.
+
+## Adding a New Version
+
+When you need to add a new version to a kind, it's rather simple in the standard CUE definition:
+```cue
+{
+    // Existing kind information
+    versions: {
+        "v1": { // Current version
+            schema: {
+                // schema
+            } 
+        }
+        "v2": { // New version
+            schema: {
+                // new schema
+            }
+        }
+    }
+}
+```
+
+However, now that you have two versions, you'll need to be able to support both of them, since any user can request either version from the API server. 
+By default, the API server will convert between these versions by simply taking the JSON schema from one and pushing it into the other (depending on what is stored), 
+but often that is not a good enough conversion, and you'll need to define how to convert between versions yourself.
+
+To do this, you'll need to add a Conversion Webhook to your operator. If you're using the `simple.Operator` type, this can be done either in the initial config with the `simple.OperatorConfig.Webhooks.Converters` field, or via the `ConvertKind` method on `simple.Operator`. If you are not using `simple.Operator`, you'll need to create a `k8s.WebhookServer` controller to add to your operator with `k8s.NewWebhookServer` (you may already have this if you're using [Admission Control](./admission-control.md), as Conversion webhooks are exposed on the same server as Validation and Mutation webhooks). All of these different methods require two things: a `k8s.Converter`-implementing type, and TLS information. Let's go over each, then give some examples.
+
+### Converter
+
+`k8s.Converter` is defined as:
+```go
+// Converter describes a type which can convert a kubernetes kind from one API version to another.
+// Typically there is one converter per-kind, but a single converter can also handle multiple kinds.
+type Converter interface {
+	// Convert converts a raw kubernetes kind into the target APIVersion.
+	// The RawKind argument will contain kind information and the raw kubernetes object,
+	// and the returned bytes are expected to be a raw kubernetes object of the same kind and targetAPIVersion
+	// APIVersion. The returned kubernetes object MUST have an apiVersion that matches targetAPIVersion.
+	Convert(obj RawKind, targetAPIVersion string) ([]byte, error)
+}
+```
+`k8s.RawKind` contains the raw (JSON) bytes of an object, and kind information (Group, Version, Kind). To implement `k8s.Converter` we need a function which can accept any version of our kind and return any version of our kind. When we register `k8s.Converter`s with the `WebhookServer`, we can generally assume that the input 
+`RawKind` will be of the Group and Kind we specify, and the output will also be of the Group and Kind we specify, so it's safe to error on anything unexpected. Let's put together a very simple converter for an object with two versions defined as:
+```cue
+myKind: {
+    kind: "MyKind"
+    versions: {
+        "v1": { 
+            schema: {
+                foo: string
+            } 
+        }
+        "v2": { 
+            schema: {
+                foo: string
+                bar: string
+            }
+        }
+    }
+}
+```
+```go
+type MyKindConverter struct {}
+
+func (m *MyKindConverter) Convert(obj k8s.RawKind, targetAPIVersion string) ([]byte, error) {
+    // We shouldn't ever see this, but just in case...
+    if targetAPIVersion == obj.APIVersion {
+        return nil, obj.Raw
+    }
+    targetGVK := schema.FromAPIVersionAndKind(targetAPIVersion, obj.Kind)
+
+    if obj.Version == "v1" {
+        // Only allowed conversion is to v2 (as we already checked v1 => v1 above)
+        if targetGVK.Version != "v2" {
+            return nil, fmt.Errorf("cannot convert into unknown version %s", targetGVK.Version)
+        }
+        src := v1.MyKind{}
+        err := v1.Kind().Codec(resource.KindEncodingJSON).Read(bytes.NewReader(obj.Raw), &src)
+        if err != nil {
+            return nil, fmt.Errorf("unable to parse kind")
+        }
+        dst := v2.MyKind{}
+        // Copy metadata
+        src.ObjectMeta.DeepCopyInto(&dst.ObjectMeta)
+        // Set GVK
+        dst.SetGroupVersionKind(targetGVK)
+        // Set values
+        dst.Spec.Foo = src.Spec.Foo
+        buf := bytes.Buffer{}
+        err := v2.Kind().Write(&dst, &buf, resource.KindEncodingJSON)
+        return buf.Bytes(), err
+    }
+
+    if obj.Version == "v2" {
+        // Only allowed conversion is to v1 (as we already checked v2 => v2 above)
+        if targetGVK.Version != "v1" {
+            return nil, fmt.Errorf("cannot convert into unknown version %s", targetGVK.Version)
+        }
+        src := v2.MyKind{}
+        err := v2.Kind().Codec(resource.KindEncodingJSON).Read(bytes.NewReader(obj.Raw), &src)
+        if err != nil {
+            return nil, fmt.Errorf("unable to parse kind")
+        }
+        dst := v1.MyKind{}
+        // Copy metadata
+        src.ObjectMeta.DeepCopyInto(&dst.ObjectMeta)
+        // Set GVK
+        dst.SetGroupVersionKind(targetGVK)
+        // Set values
+        dst.Spec.Foo = src.Spec.Foo
+        buf := bytes.Buffer{}
+        err := v1.Kind().Write(&dst, &buf, resource.KindEncodingJSON)
+        return buf.Bytes(), err
+    }
+
+    return nil, fmt.Errorf("unknown source version %s", obj.Version)
+}
+```
+
+Now, depending on the way we are creating our operator, we can register the converter webhook:
+
+#### `simple.Operator`
+
+**Using OperatorConfig**
+```go
+runner, err := simple.NewOperator(simple.OperatorConfig{
+    Name:       "my-operator",
+	KubeConfig: kubeConfig.RestConfig,
+    Webhooks:   simple.WebhookConfig{
+        Enabled:   true,
+        TLSConfig: k8s.TLSConfig{
+            CertPath: "/path/to/cert",
+            KeyPath:  "/path/to/key",
+        },
+        Converters: map[metav1.GroupKind]k8s.Converter{
+            metav1.GroupKind{Group: v1.Group(), Kind: v1.Kind()}: &MyKindConverter{},
+        },
+    },
+})
+```
+**Using ConvertKind**
+```go
+runner, err := simple.NewOperator(simple.OperatorConfig{
+    Name:       "my-operator",
+	KubeConfig: kubeConfig.RestConfig,
+    Webhooks:   simple.WebhookConfig{ // Webhook information is still required in config
+        Enabled:   true,
+        TLSConfig: k8s.TLSConfig{
+            CertPath: "/path/to/cert",
+            KeyPath:  "/path/to/key",
+        },
+    },
+})
+
+err = runner.ConvertKind(metav1.GroupKind{Group: v1.Group(), Kind: v1.Kind()}, &MyKindConverter{})
+```
+
+#### `k8s.NewWebhookServer`
+
+**In WebhookServerConfig**
+```go
+ws, err = k8s.NewWebhookServer(k8s.WebhookServerConfig{
+    Port:      8443,
+    TLSConfig: k8s.TLSConfig{
+        CertPath: "/path/to/cert",
+        KeyPath:  "/path/to/key",
+    },
+    KindConverters: map[metav1.GroupKind]k8s.Converter{
+        metav1.GroupKind{Group: v1.Group(), Kind: v1.Kind()}: &MyKindConverter{},
+    },
+})
+
+err = runner.AddController(ws)
+```
+
+**Using AddConverter**
+```go
+ws, err = k8s.NewWebhookServer(k8s.WebhookServerConfig{
+    Port:      8443,
+    TLSConfig: k8s.TLSConfig{
+        CertPath: "/path/to/cert",
+        KeyPath:  "/path/to/key",
+    },
+})
+
+ws.AddConverter(&MyKindConverter{}, metav1.GroupKind{Group: v1.Group(), Kind: v1.Kind()})
+
+err = runner.AddController(ws)
+```
+
+### Using the Converter Webhook
+
+If you use `grafana-app-sdk project local generate`, you can set `converting: true` in the `webhooks` section of `local/config.yaml`. This will set the webhook conversion strategy in your CRD and make sure your operator service exposes your webhook port. However, this only applies for local setup. 
+
+> [!WARNING]  
+> To register your conversion webhook in production, you must currently manually update the generated CRD file!
+
+Update your generated CRD file to add the `conversion` block as described in [the kubernetes documentation](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/#configure-customresourcedefinition-to-use-conversion-webhooks) before deploying the CRD to take advantage of the converter webhook you have written