Feature: Check core package health endpoint

[cmoulliard]

Have you searched for this feature request?

• I searched but did not find similar requests

Problem Statement

Until now, the localbuild reconciles the resources as Deployment, StatefulSet and more precisely if the status: AvailableReplicas is >= 1.

While such a check guarantees that a pod has been created and is running, we have no guarantee that the core Applications: gitea, ingress, argocd are healthy and their HTTP endpoint are accessible.

This is why I would like to propose that we add a new test able to call the healthEndpoint of a core application if it exists and has been set part of the

type EmbeddedInstallation struct {
	name         string
	resourcePath string
	namespace    string

	// skips waiting on expected resources to become ready
	skipReadinessCheck bool

	// name and gvk pair for resources that need to be monitored
	monitoredResources map[string]schema.GroupVersionKind
	customization      v1alpha1.PackageCustomization
	resourceFS         embed.FS

	// resources that need to be created without using static manifests or gitops
	unmanagedResources []client.Object

	healthEndpoint    string // endpoint + port
}

Dummy example of such a check

func (e *EmbeddedInstallation) CheckHealthEndpoint(ctx context.Context, cli client.Client) (ctrl.Result, error) {
	logger := log.FromContext(ctx)
	
	labelSelector := client.MatchingLabels{
		"app.kubernetes.io/name":      "ingress-nginx",
		"app.kubernetes.io/component": "controller",
	}

	// List Pods with the specified label in the given namespace
	var podList corev1.PodList
	err := cli.List(ctx, &podList, client.InNamespace(e.namespace), labelSelector)
	if err != nil {
		// Handle error if listing fails
		return ctrl.Result{}, err
	}

	if len(podList.Items) == 0 {
		// If no Pods found, log a message and skip reconciliation
		logger.V(1).Info("No Pods found with the specified label")
		return ctrl.Result{RequeueAfter: 5 * time.Second}, nil
	}

	var pod = podList.Items[0]

	// Construct the health endpoint URL
	// Assumes health check is on port 8080 and path `/health`
	healthURL := fmt.Sprintf("http://%s:10254/healtz", pod.Status.PodIP)

	// Set up an HTTP client with a timeout
	client := http.Client{
		Timeout: 5 * time.Second,
	}

	// Send a GET request to the health endpoint
	resp, err := client.Get(healthURL)
	if err != nil {
		// Log the error and requeue if the Pod is not healthy
		return ctrl.Result{}, err
	}
	defer resp.Body.Close()

	// Check if the health endpoint response is successful (200 OK)
	if resp.StatusCode == http.StatusOK {
		return ctrl.Result{}, err
	} else {
		// Requeue if the health check is not successful
		return ctrl.Result{RequeueAfter: 5 * time.Second}, nil
	}
}

If we do that, then we should get rid of errors like: argocd not able to call gitea, gitea client not able to set a token, etc

As such a check will fail as we run the command from the CLI where it is not possible to call the HTTP's pod health endpoint using the pod.Status.poDIP, then we could maybe leverage this operator tool: https://github.com/kuberhealthy/kuberhealthy?tab=readme-ov-file

WDYT ? @nabuskey

Possible Solution

See problem statement

Alternatives Considered

n/a