Skip to content

Commit

Permalink
Implement tracked resources (radius-project#6204)
Browse files Browse the repository at this point in the history
# Description

This change implements 'tracked' resources for for every resource
created in the Radius plane. This means that UCP will observe each
operation and maintain its own state for the lifecycle of each resource.
The purpose of this is to power multiple other pieces of infrastructure.
New in this pull-request it's possible to list the resources (of all
types) in a resource group. This is served from the 'tracked' resources
maintained by UCP and does not need to query all resource providers or
their databases. Future changes will implement cascading deletion for
resource groups as well as notifications for resource modifications.

## Type of change



- This pull request fixes a bug in Radius and has an approved issue
(issue link required).
- This pull request adds or changes features of Radius and has an
approved issue (issue link required).


## Auto-generated summary

<!--
GitHub Copilot for docs will auto-generate a summary of the PR
-->

<!--
copilot:all
-->
### <samp>🤖 Generated by Copilot at 969afb9</samp>

### Summary
📝🌐🔄

<!--
1. 📝 - This emoji represents the addition of comments, documentation,
and test fixtures, which are all related to writing and explaining the
code.
2. 🌐 - This emoji represents the addition of the `location` field, which
is related to the global scope and distribution of the UCP service.
3. 🔄 - This emoji represents the addition of conversion functions,
serialization and deserialization methods, and async operation
controllers, which are all related to transforming and processing data.
-->
This pull request adds support for generic and proxy resources in the
UCP service. It introduces a new `GenericResource` type in the UCP API
and datamodel packages, and implements the conversion, serialization,
and client functions for it. It also adds a new `Resources` group to the
UCP API, which allows listing and querying resources in UCP. It updates
the UCP backend service and controller to use the new resource type and
operation method for tracked resources. It adds unit tests,
documentation, and configuration settings for the new features. It also
improves the logging and error handling of async operations, and removes
some unused dependencies.

> _Sing, O Muse, of the mighty deeds of the UCP service_
> _That tracks and lists the myriad resources in the cloud_
> _And how the skillful coders added new fields and types_
> _To make the `GenericResource` more versatile and proud_

### Walkthrough
* Add a new `GenericResource` type and its conversion functions to
support storing and listing resources in UCP regardless of their actual
type
([link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-f2d1781ff7fc84fff544f3a071c4bad77c26a30777d6d95f39cbd49ee16b6ba7R1-R49),
[link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-f845c7096c463030307454c370c23460c9a806f229fe959e226cc9e81222ad70R1-R73),
[link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-893ab293df1ca8739cc792cd8e8c5c5691cb0a308cec7f4ef2f06ecce4019761L1-R21),
[link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-6f7e7aae5544d15eeb60fd6452c8cdcdd56c68c1c6c5a9910511ef0aef14aa77R170-R188),
[link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-6f7e7aae5544d15eeb60fd6452c8cdcdd56c68c1c6c5a9910511ef0aef14aa77R232-R258),
[link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-6f300088351b430ca2464beef84ad3069228f02d95a3dcd3641926ad1c2620fbR387-R429),
[link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-6f300088351b430ca2464beef84ad3069228f02d95a3dcd3641926ad1c2620fbR558-R631),
[link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-4f5a8ccf4f4002068030786c91856b61579168b6c60a91b815f6d525552bf907R1-R47),
[link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-c71c1646c3c381abc13b22a43803aadd6725bdc1b956f957452fa0955032a723R1-R86))
* Add a new `Resources` group and its client to the UCP API, which
contains the methods for listing and querying resources in UCP
([link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-df683afb7070d6ba64f3b2a742db37199c9827b9ba03cc95b48777ad2e601644R57-R61),
[link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-416356e403db5b572d2c0f372baef2abfeb3000363efc6d41f55bdc65f47ebafR121-R125),
[link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-a0503cdfd9eb8e97f731ed907b35daf38dde994f995701b0fd69ca1699883adfR1-R108),
[link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-c74c23d29da9a5ac2ea4388c4cfd23f42018e45c93a29f71c4c5eeadebdfa4c6R132-R137))
* Add a new `TrackedResourceProcessController` type and its tests to the
UCP backend, which performs background processing on tracked resources
([link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-58df6e92977a97ec7d0156ae7e0dafc869190c32b82b448d3e684ad7586170bfR1-R94),
[link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-f966a886f9d05281b158e9739ff6c1e1e0bffe560230e7d5d8305dbedef6e309R1-R180),
[link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-8aded038840f1f8802bdb81f53055a340da414b4d53a57b2af19bc6463599388L68-R93))
* Update the `UCPProviderName` constant to `System.Resources` to align
with the new resource type for tracked resources
([link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-8aded038840f1f8802bdb81f53055a340da414b4d53a57b2af19bc6463599388L23-R33))
* Add some logging and error handling to the
`updateResourceAndOperationStatus` function in the `worker.go` file,
which updates the resource and operation status in the storage
([link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-5d9b9964242d77606f4868e3fe049978ea1c3f1903afcfeb60c76e19d9785fc5R254-R261),
[link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-5d9b9964242d77606f4868e3fe049978ea1c3f1903afcfeb60c76e19d9785fc5L350-R360))
* Add a comment to the `generate-openapi-spec` target in the `Makefile`
to describe its purpose
([link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-cc76a2a14994ce1ed06bd12ba9665a1d20a17992f345a7f8ca06afc934da2a92L35-R35))
* Add a new field `location` to the UCP service configuration and
deployment files, and document its meaning
([link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-ceba0600c7e49ffd65b0a2fd7bf1798d9f6f6f531db64e051bb00ff29c7dcd93R12),
[link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-be9c0b61a26165c87503e667cec005520aea94da66f0dd1eedfff863efd266acR13),
[link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-c5a6e900bac29ec26476b731e62862ee9afb8f9f67225da0aa8fd1d052f8183fR220))
* Remove some unused dependencies and imports from the `go.mod` and
`worker.go` files
([link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6L14),
[link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6L70),
[link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6R99),
[link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6L107),
[link](https://github.com/radius-project/radius/pull/6204/files?diff=unified&w=0#diff-5d9b9964242d77606f4868e3fe049978ea1c3f1903afcfeb60c76e19d9785fc5L24))

Signed-off-by: ytimocin <[email protected]>
Signed-off-by: Ryan Nowak <[email protected]>
Co-authored-by: Yetkin Timocin <[email protected]>
  • Loading branch information
rynowak and ytimocin authored Dec 10, 2023
1 parent 6d10e1c commit efa7880
Show file tree
Hide file tree
Showing 27 changed files with 2,634 additions and 35 deletions.
1 change: 1 addition & 0 deletions cmd/ucpd/ucp-self-hosted-dev.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
# - Talk to Portable Resources' Providers on port 8081
# - Disables metrics and profiler
#
location: 'global'
storageProvider:
provider: "apiserver"
apiserver:
Expand Down
1 change: 1 addition & 0 deletions deploy/Chart/templates/ucp/configmaps.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ data:
ucp-config.yaml: |-
# Radius configuration file.
# See https://github.com/radius-project/radius/blob/main/docs/contributing/contributing-code/contributing-code-control-plane/configSettings.md for more information.
location: 'global'
storageProvider:
provider: "apiserver"
apiserver:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,7 @@ ucp:

### UCP
```yaml
location: 'global'
storageProvider:
provider: "apiserver"
apiserver:
Expand Down
16 changes: 12 additions & 4 deletions pkg/armrpc/asyncoperation/worker/worker.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ import (
"encoding/json"
"errors"
"fmt"
"net/http"
"runtime/debug"
"strings"
"time"
Expand Down Expand Up @@ -252,6 +251,14 @@ func (w *AsyncRequestProcessWorker) runOperation(ctx context.Context, message *q

logger.Info("Start processing operation.")
result, err := asyncCtrl.Run(asyncReqCtx, asyncReq)

code := ""
if result.Error != nil {
code = result.Error.Code
}

logger.Info("Operation returned", "success", result.Error == nil, "code", code, "provisioningState", result.ProvisioningState(), "err", err)

// There are two cases when asyncReqCtx is canceled.
// 1. When the operation is timed out, w.completeOperation will be called in L186
// 2. When parent context is canceled or done, we need to requeue the operation to reprocess the request.
Expand All @@ -262,6 +269,7 @@ func (w *AsyncRequestProcessWorker) runOperation(ctx context.Context, message *q
result.SetFailed(armErr, false)
logger.Error(err, "Operation Failed")
}

w.completeOperation(ctx, message, result, asyncCtrl.StorageClient())
}
trace.SetAsyncResultStatus(result, span)
Expand Down Expand Up @@ -347,10 +355,10 @@ func (w *AsyncRequestProcessWorker) updateResourceAndOperationStatus(ctx context
return err
}

opType, _ := v1.ParseOperationType(req.OperationType)

err = updateResourceState(ctx, sc, rID.String(), state)
if err != nil && !(opType.Method == http.MethodDelete && errors.Is(&store.ErrNotFound{ID: rID.String()}, err)) {
if errors.Is(err, &store.ErrNotFound{}) {
logger.Info("failed to update the provisioningState in resource because it no longer exists.")
} else if err != nil {
logger.Error(err, "failed to update the provisioningState in resource.")
return err
}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
/*
Copyright 2023 The Radius Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package resourcegroups

import (
"context"
"errors"
"fmt"
"net/http"

v1 "github.com/radius-project/radius/pkg/armrpc/api/v1"
ctrl "github.com/radius-project/radius/pkg/armrpc/asyncoperation/controller"
"github.com/radius-project/radius/pkg/ucp/datamodel"
"github.com/radius-project/radius/pkg/ucp/frontend/controller/resourcegroups"
"github.com/radius-project/radius/pkg/ucp/resources"
"github.com/radius-project/radius/pkg/ucp/store"
"github.com/radius-project/radius/pkg/ucp/trackedresource"
"github.com/radius-project/radius/pkg/ucp/ucplog"
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

var _ ctrl.Controller = (*TrackedResourceProcessController)(nil)

type updater interface {
Update(ctx context.Context, downstreamURL string, originalID resources.ID, version string) error
}

// TrackedResourceProcessController is the async operation controller to perform background processing on tracked resources.
type TrackedResourceProcessController struct {
ctrl.BaseController

// Updater is the utility struct that can perform updates on tracked resources. This can be modified for testing.
updater updater
}

// NewTrackedResourceProcessController creates a new TrackedResourceProcessController controller which is used to process resources asynchronously.
func NewTrackedResourceProcessController(opts ctrl.Options) (ctrl.Controller, error) {
transport := otelhttp.NewTransport(http.DefaultTransport)
return &TrackedResourceProcessController{ctrl.NewBaseAsyncController(opts), trackedresource.NewUpdater(opts.StorageClient, &http.Client{Transport: transport})}, nil
}

// Run retrieves a resource from storage, parses the resource ID, and updates our tracked resource entry in the background.
func (c *TrackedResourceProcessController) Run(ctx context.Context, request *ctrl.Request) (ctrl.Result, error) {
resource, err := store.GetResource[datamodel.GenericResource](ctx, c.StorageClient(), request.ResourceID)
if errors.Is(err, &store.ErrNotFound{}) {
return ctrl.NewFailedResult(v1.ErrorDetails{Code: v1.CodeNotFound, Message: fmt.Sprintf("resource %q not found", request.ResourceID), Target: request.ResourceID}), nil
} else if err != nil {
return ctrl.Result{}, err
}

originalID, err := resources.Parse(resource.Properties.ID)
if err != nil {
return ctrl.Result{}, err
}

downstreamURL, err := resourcegroups.ValidateDownstream(ctx, c.StorageClient(), originalID)
if errors.Is(err, &resourcegroups.NotFoundError{}) {
return ctrl.NewFailedResult(v1.ErrorDetails{Code: v1.CodeNotFound, Message: err.Error(), Target: request.ResourceID}), nil
} else if errors.Is(err, &resourcegroups.InvalidError{}) {
return ctrl.NewFailedResult(v1.ErrorDetails{Code: v1.CodeInvalid, Message: err.Error(), Target: request.ResourceID}), nil
} else if err != nil {
return ctrl.Result{}, fmt.Errorf("failed to validate downstream: %w", err)
}

logger := ucplog.FromContextOrDiscard(ctx)
logger.Info("Processing tracked resource", "resourceID", originalID)
err = c.updater.Update(ctx, downstreamURL.String(), originalID, resource.Properties.APIVersion)
if errors.Is(err, &trackedresource.InProgressErr{}) {
// The resource is still being processed, so we can sleep for a while.
result := ctrl.Result{}
result.SetFailed(v1.ErrorDetails{Code: v1.CodeConflict, Message: err.Error(), Target: request.ResourceID}, true)

return result, nil
} else if err != nil {
return ctrl.Result{}, err
}

logger.Info("Completed processing tracked resource", "resourceID", originalID)
return ctrl.Result{}, nil
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,180 @@
/*
Copyright 2023 The Radius Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package resourcegroups

import (
"context"
"fmt"
"testing"

"github.com/golang/mock/gomock"
v1 "github.com/radius-project/radius/pkg/armrpc/api/v1"
"github.com/radius-project/radius/pkg/armrpc/asyncoperation/controller"
"github.com/radius-project/radius/pkg/to"
"github.com/radius-project/radius/pkg/ucp/api/v20231001preview"
"github.com/radius-project/radius/pkg/ucp/datamodel"
"github.com/radius-project/radius/pkg/ucp/resources"
"github.com/radius-project/radius/pkg/ucp/store"
"github.com/radius-project/radius/pkg/ucp/trackedresource"
"github.com/radius-project/radius/test/testcontext"
"github.com/stretchr/testify/require"
)

func Test_Run(t *testing.T) {
setup := func(t *testing.T) (*TrackedResourceProcessController, *mockUpdater, *store.MockStorageClient) {
ctrl := gomock.NewController(t)
storageClient := store.NewMockStorageClient(ctrl)

pc, err := NewTrackedResourceProcessController(controller.Options{StorageClient: storageClient})
require.NoError(t, err)

updater := mockUpdater{}
pc.(*TrackedResourceProcessController).updater = &updater
return pc.(*TrackedResourceProcessController), &updater, storageClient
}

id := resources.MustParse("/planes/test/local/resourceGroups/test-rg/providers/Applications.Test/testResources/my-resource")
trackingID := trackedresource.IDFor(id)

plane := datamodel.Plane{
Properties: datamodel.PlaneProperties{
Kind: datamodel.PlaneKind(v20231001preview.PlaneKindUCPNative),
ResourceProviders: map[string]*string{
"Applications.Test": to.Ptr("https://localhost:1234"),
},
},
}
resourceGroup := datamodel.ResourceGroup{}
data := datamodel.GenericResourceFromID(id, trackingID)

// Most of the heavy lifting is done by the updater. We just need to test that we're calling it correctly.
t.Run("Success", func(t *testing.T) {
pc, _, storageClient := setup(t)

storageClient.EXPECT().
Get(gomock.Any(), trackingID.String(), gomock.Any()).
Return(&store.Object{Data: data}, nil).Times(1)

storageClient.EXPECT().
Get(gomock.Any(), "/planes/"+trackingID.PlaneNamespace(), gomock.Any()).
Return(&store.Object{Data: plane}, nil).Times(1)

storageClient.EXPECT().
Get(gomock.Any(), trackingID.RootScope(), gomock.Any()).
Return(&store.Object{Data: resourceGroup}, nil).Times(1)

result, err := pc.Run(testcontext.New(t), &controller.Request{ResourceID: trackingID.String()})
require.Equal(t, controller.Result{}, result)
require.NoError(t, err)
})

t.Run("retry", func(t *testing.T) {
pc, updater, storageClient := setup(t)

storageClient.EXPECT().
Get(gomock.Any(), trackingID.String(), gomock.Any()).
Return(&store.Object{Data: data}, nil).Times(1)

storageClient.EXPECT().
Get(gomock.Any(), "/planes/"+trackingID.PlaneNamespace(), gomock.Any()).
Return(&store.Object{Data: plane}, nil).Times(1)

storageClient.EXPECT().
Get(gomock.Any(), trackingID.RootScope(), gomock.Any()).
Return(&store.Object{Data: resourceGroup}, nil).Times(1)

// Force a retry.
updater.Result = &trackedresource.InProgressErr{}

expected := controller.Result{}
expected.SetFailed(v1.ErrorDetails{Code: v1.CodeConflict, Message: updater.Result.Error(), Target: trackingID.String()}, true)

result, err := pc.Run(testcontext.New(t), &controller.Request{ResourceID: trackingID.String()})
require.Equal(t, expected, result)
require.NoError(t, err)
})

t.Run("Failure (resource not found)", func(t *testing.T) {
pc, _, storageClient := setup(t)

storageClient.EXPECT().
Get(gomock.Any(), trackingID.String(), gomock.Any()).
Return(nil, &store.ErrNotFound{}).Times(1)

expected := controller.NewFailedResult(v1.ErrorDetails{
Code: v1.CodeNotFound,
Message: fmt.Sprintf("resource %q not found", trackingID.String()),
Target: trackingID.String(),
})

result, err := pc.Run(testcontext.New(t), &controller.Request{ResourceID: trackingID.String()})
require.Equal(t, expected, result)
require.NoError(t, err)
})

t.Run("Failure (validate downstream: not found)", func(t *testing.T) {
pc, _, storageClient := setup(t)

storageClient.EXPECT().
Get(gomock.Any(), trackingID.String(), gomock.Any()).
Return(&store.Object{Data: data}, nil).Times(1)

storageClient.EXPECT().
Get(gomock.Any(), "/planes/"+trackingID.PlaneNamespace(), gomock.Any()).
Return(nil, &store.ErrNotFound{}).Times(1)

expected := controller.NewFailedResult(v1.ErrorDetails{
Code: v1.CodeNotFound,
Message: "plane \"/planes/test/local\" not found",
Target: trackingID.String(),
})

result, err := pc.Run(testcontext.New(t), &controller.Request{ResourceID: trackingID.String()})
require.Equal(t, expected, result)
require.NoError(t, err)
})

t.Run("Failure (validate downstream: invalid downstream)", func(t *testing.T) {
pc, _, storageClient := setup(t)

storageClient.EXPECT().
Get(gomock.Any(), trackingID.String(), gomock.Any()).
Return(&store.Object{Data: data}, nil).Times(1)

storageClient.EXPECT().
Get(gomock.Any(), "/planes/"+trackingID.PlaneNamespace(), gomock.Any()).
Return(&store.Object{Data: datamodel.Plane{}}, nil).Times(1)

expected := controller.NewFailedResult(v1.ErrorDetails{
Code: v1.CodeInvalid,
Message: "unexpected plane type ",
Target: trackingID.String(),
})

result, err := pc.Run(testcontext.New(t), &controller.Request{ResourceID: trackingID.String()})
require.Equal(t, expected, result)
require.NoError(t, err)
})
}

type mockUpdater struct {
Result error
}

func (u *mockUpdater) Update(ctx context.Context, downstreamURL string, originalID resources.ID, version string) error {
return u.Result
}
26 changes: 25 additions & 1 deletion pkg/ucp/backend/service.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,17 @@ import (
"context"
"fmt"

v1 "github.com/radius-project/radius/pkg/armrpc/api/v1"
ctrl "github.com/radius-project/radius/pkg/armrpc/asyncoperation/controller"
"github.com/radius-project/radius/pkg/armrpc/asyncoperation/worker"
"github.com/radius-project/radius/pkg/armrpc/hostoptions"
"github.com/radius-project/radius/pkg/ucp/api/v20231001preview"
"github.com/radius-project/radius/pkg/ucp/backend/controller/resourcegroups"
"github.com/radius-project/radius/pkg/ucp/datamodel"
)

const (
UCPProviderName = "ucp"
UCPProviderName = "System.Resources"
)

// Service is a service to run AsyncReqeustProcessWorker.
Expand Down Expand Up @@ -65,5 +70,24 @@ func (w *Service) Run(ctx context.Context) error {
}
}

opts := ctrl.Options{
DataProvider: w.StorageProvider,
}

err := RegisterControllers(ctx, w.Controllers, opts)
if err != nil {
return err
}

return w.Start(ctx, workerOpts)
}

// RegisterControllers registers the controllers for the UCP backend.
func RegisterControllers(ctx context.Context, registry *worker.ControllerRegistry, opts ctrl.Options) error {
err := registry.Register(ctx, v20231001preview.ResourceType, v1.OperationMethod(datamodel.OperationProcess), resourcegroups.NewTrackedResourceProcessController, opts)
if err != nil {
return err
}

return nil
}
Loading

0 comments on commit efa7880

Please sign in to comment.