Skip to content

Commit

Permalink
Adding documentation for Backstage Terraform Integrations
Browse files Browse the repository at this point in the history 
Signed-off-by: Elamaran Shanmugam <[email protected]>
  • Loading branch information
@elamaran11
elamaran11 committed Apr 25, 2024
1 parent 79b5df7 commit 42794b8
Showing 1 changed file with 79 additions and 130 deletions.
209 changes: 79 additions & 130 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,15 +1,3 @@
Table of Contents
=================

* [IDP Builder](#idp-builder)
* [About](#about)
* [Quickstart](#quickstart)
* [Running the idpbuilder](#running-the-idpbuilder)
* [Use](#use)
* [Architecture](#architecture)
* [Extending the IDP builder](#extending-the-idpbuilder)
* [Developer notes](#developer-notes)

# IDP Builder

Internal development platform binary launcher.
Expand All @@ -27,6 +15,8 @@ This can be useful in several ways:

## Quickstart

If you are interested in running idpbuilder in Codespaces through your browser, check out the [Codespaces](#running-in-codespaces) section.

### Download and install the idpbuilder

Download the latest release with the commands:
Expand All @@ -44,22 +34,33 @@ tar xzf idpbuilder.tar.gz

Alternatively, you can download the latest binary from [the latest release page](https://github.com/cnoe-io/idpbuilder/releases/latest).

### Using the idpbuilder
## Using the idpbuilder

### Basic usage

The most basic command which creates a Kubernetes Cluster (Kind cluster) with the core packages installed.

```bash
./idpbuilder create
```

This is the most basic command which creates a Kubernetes Cluster (Kind cluster) with the core packages installed.

<details>
<summary>What are the core packages?</summary>

* **ArgoCD** is the GitOps solution to deploy manifests to Kubernetes clusters. In this project, a package is an ArgoCD application.
* **Gitea** server is the in-cluster Git server that ArgoCD can be configured to sync resources from. You can sync from local file systems to this.
* **Ingress-nginx** is used as a method to access in-cluster resources such as ArgoCD UI and Gitea UI.

See the [Architecture](#Architecture) section for more information on further information on how core packages are installed and configured.
#### Core package versions

| Name | Version |
| -------- | ------- |
| Argo CD | v2.10.7 |
| Gitea | v9.5.1 |
| Nginx | v1.8.1 |

The default manifests for the core packages are available [here](pkg/controllers/localbuild/resources).
See the [contribution doc](./CONTRIBUTING.md) for more information on how core packages are installed and configured.

</details>

Expand All @@ -69,19 +70,36 @@ Once idpbuilder finishes provisioning cluster and packages, you can access GUIs
* ArgoCD: https://argocd.cnoe.localtest.me:8443/
* Gitea: https://gitea.cnoe.localtest.me:8443/

You can obtain credentials for these packages by running the following commands:
You can obtain credentials for them by running the following command:

```bash
# argoCD password. username is admin.
kubectl -n argocd get secret argocd-initial-admin-secret \
-o go-template='{{ range $key, $value := .data }}{{ printf "%s: %s\n" $key ($value | base64decode) }}{{ end }}'

# gitea admin credentials
kubectl get secrets -n gitea gitea-admin-secret \
-o go-template='{{ range $key, $value := .data }}{{ printf "%s: %s\n" $key ($value | base64decode) }}{{ end }}'
./idpbuilder get secrets
```

#### Example commands
<details>
<summary> The "get secrets" command </summary>

The `get secrets` command retrieves the following:
- ArgoCD initial admin password.
- Gitea admin user credentials.
- Any secrets labeled with `cnoe.io/cli-secret=true`.

You can think of the command as executing the following kubectl commands:

```bash
kubectl -n argocd get secret argocd-initial-admin-secret
kubectl get secrets -n gitea gitea-admin-secret
kubectl get secrets -A -l cnoe.io/cli-secret=true
```
In addition, secrets labeled with `cnoe.io/package-name` can be specified with the `-p` flag. For example, for Gitea:

```bash
./idpbuilder get secrets -p gitea
```

</details>

### Example commands

**For more advanced use cases, check out the [examples](./examples) directory.**

Expand All @@ -91,29 +109,31 @@ You can specify the kubernetes version by using the `--kube-version` flag. Suppo
./idpbuilder create --kube-version v1.27.3
```

If you need to expose more ports between the docker container and the kubernetes host, you can use the `--extra-ports` flag. For example:
If you want to specify your own kind configuration file, use the `--kind-config` flag.

```
./idpbuilder create --extra-ports 22:32222
./idpbuilder create --build-name local --kind-config ./my-kind.yaml`
```

If you want to specify your own kind configuration file, use the `--kind-config` flag.
If you want to specify ArgoCD configmap.

```
./idpbuilder create --build-name local --kind-config ./my-kind.yaml`
./idpbuilder create --package-custom-file=argocd:pkg/k8s/test-resources/input/argocd-cm.yaml
```

You can also specify the name of build. This name is used as part of the cluster, namespace, and git repositories.
Run the following commands for available flags and subcommands:

```
./idpbuilder create --build-name localdev
./idpbuilder --help
./idpbuilder create --help
```

### Custom Packages

Idpbuilder supports specifying custom packages using the flag `--package-dir` flag. This flag expects a directory containing ArgoCD application files.

Let's take a look at [this example](examples/basic). This example defines two custom package directories to deploy to the cluster.
Examples of using custom packages are available in the [example](./examples) directory.
Let's take a look at [this example](examples/basic). This defines two custom package directories to deploy to the cluster.

To deploy these packages, run the following commands from this repository's root.

Expand Down Expand Up @@ -167,105 +187,34 @@ You can also view the updated Application spec by going to this address: https:/
The second package directory defines two normal ArgoCD applications referencing a remote repository.
They are applied as-is.

## Architecture

idpbuilder is made of two phases: CLI and Kubernetes controllers.

![idpbuilder.png](docs/images/idpbuilder.png)

### CLI

When the idpbuilder binary is executed, it starts with the CLI phase.

1. This is the phase where command flags are parsed and translated into relevant Go structs' fields. Most notably the [`LocalBuild`](https://github.com/cnoe-io/idpbuilder/blob/main/api/v1alpha1/localbuild_types.go) struct.
2. Create a Kind cluster, then update the kubeconfig file.
3. Once the kind cluster is started and relevant fields are populated, Kubernetes controllers are started:
* `LocalbuildReconciler` responsible for bootstrapping the cluster with absolute necessary packages. Creates Custom Resources (CRs) and installs embedded manifests.
* `RepositoryReconciler` responsible for creating and managing Gitea repository and repository contents.
* `CustomPackageReconciler` responsible for managing custom packages.
4. They are all managed by a single Kubernetes controller manager.
5. Once controllers are started, CRs corresponding to these controllers are created. For example for Backstage, it creates a GitRepository CR and ArgoCD Application.
6. CLI then waits for these CRs to be ready.

### Controllers

During this phase, controllers act on CRs created by the CLI phase. Resources such as Gitea repositories and ArgoCD applications are created.

#### LocalbuildReconciler

`LocalbuildReconciler` bootstraps the cluster using embedded manifests. Embedded manifests are yaml files that are baked into the binary at compile time.
1. Install core packages. They are essential services that are needed for the user experiences we want to enable:
* Gitea. This is the in-cluster Git server that hosts Git repositories.
* Ingress-nginx. This is necessary to expose services inside the cluster to the users.
* ArgoCD. This is used as the packaging mechanism. Its primary purpose is to deploy manifests from gitea repositories.
2. Once they are installed, it creates `GitRepository` CRs for core packages. This CR represents the git repository on the Gitea server.
3. Create ArgoCD applications for the apps. Point them to the Gitea repositories. From here on, ArgoCD manages the core packages.

Once core packages are installed, it creates the other embedded applications: Backstage and Crossplane.
1. Create `GitRepository` CRs for the apps.
2. Create ArgoCD applications for the apps. Point them to the Gitea repositories.

#### RepositoryReconciler

`RepositoryReconciler` creates Gitea repositories.
The content of the repositories can either be sourced from Embedded file system or local file system.

#### CustomPackageReconciler

`CustomPackageReconciler` parses the specified ArgoCD application files. If they specify repository URL with the scheme `cnoe://`,
it creates `GitRepository` CR with source specified as local, then creates ArgoCD application with the repository URL replaced.

For example, if an ArgoCD application is specified as the following.

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
source:
repoURL: cnoe://busybox
```

Then, the actual object created is this.

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
source:
repoURL: http://my-gitea-http.gitea.svc.cluster.local:3000/giteaAdmin/idpbuilder-localdev-my-app-busybox.git
```

## Developer notes

If you want to contribute and extend the existing project, make sure that you have installed go (>= 1.21) and cloned this project.
Next, you can build it `make` or launch the `main.go` within your IDE or locally `./idpbuilder`.

You can override the kind configuration generated by the idpbuilder. For that purpose, look to the
console to grab the config and save it in a file:
```text
########################### Our kind config ############################
# Kind kubernetes release images https://github.com/kubernetes-sigs/kind/releases
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
...
######################### config end ############################
```
Next, import it `./idpbuilder create --kindConfig <path to the config file>`

### Default manifests installed by idpbuilder

The default manifests for the core packages are available [here](pkg/controllers/localbuild/resources).
These are generated by scripts. If you want to make changes to them, see below.

#### ArgoCD

ArgoCD manifests are generated using a bash script available [here](./hack/argo-cd/generate-manifests.sh).
This script runs kustomize to modify the basic installation manifests provided by ArgoCD. Modifications include:

1. Prevent notification and dex pods from running. This is done to keep the number of pods running low by default.
2. Use the annotation tracking instead of the default label tracking. Annotation tracking allows you to avoid [problems caused by the label tracking method](https://argo-cd.readthedocs.io/en/stable/user-guide/resource_tracking/). In addition, this configuration is required when using Crossplane.
## Contributing

If you'd like to contribute to the project or know the architecture and internals of this project, check out the [contribution doc](./CONTRIBUTING.md).

## Running in Codespaces

1. Create a Codespaces instance. ![img](examples/ref-implementation/images/codespaces-create.png)
2. Wait for it to be ready. It may take several minutes.
3. Get the latest release of idpbuilder:
```bash
version=$(curl -Ls -o /dev/null -w %{url_effective} https://github.com/cnoe-io/idpbuilder/releases/latest)
version=${version##*/}
curl -L -o ./idpbuilder.tar.gz "https://github.com/cnoe-io/idpbuilder/releases/download/${version}/idpbuilder-$(uname | awk '{print tolower($0)}')-$(uname -m | sed 's/x86_64/amd64/').tar.gz"
tar xzf idpbuilder.tar.gz
```
4. Run idpbuilder:
```
idpbuilder create --protocol http \
--host ${CODESPACE_NAME}-8080.${GITHUB_CODESPACES_PORT_FORWARDING_DOMAIN} \
--port 8080 --use-path-routing
```
5. Because Codespaces gives a single externally routable host name for an instance, idpbuilder must deploy with path based routing.
This means ArgoCD and Gitea UIs are given with the following commands.
* ArgoCD: `echo https://${CODESPACE_NAME}-8080.${GITHUB_CODESPACES_PORT_FORWARDING_DOMAIN}/argocd`
* Gitea: `echo https://${CODESPACE_NAME}-8080.${GITHUB_CODESPACES_PORT_FORWARDING_DOMAIN}/gitea`
6. Note that not all examples work with path based routing.

## Extending the IDP builder

We are actively working to include more patterns and examples of extending idpbuilder to get started easily.

We are actively working to include more patterns and examples of extending idpbuilder to get started easily.

0 comments on commit 42794b8

Please sign in to comment.