diff --git a/docs/assembly-ploigos-contribute.adoc b/docs/assembly-ploigos-contribute.adoc index d27e9a1..145e9f0 100644 --- a/docs/assembly-ploigos-contribute.adoc +++ b/docs/assembly-ploigos-contribute.adoc @@ -6,7 +6,7 @@ ifdef::ProjectVersion[:parent-ProjectVersion: {ProjectVersion}] include::vars/ploigos.adoc[] -[id="{ProjectNameID}"] +[id="{ProjectNameID}-contribute"] = Contributing :context: assembly-{ProjectNameID} diff --git a/docs/assembly-ploigos-customize.adoc b/docs/assembly-ploigos-customize.adoc index 73676b6..9f84188 100644 --- a/docs/assembly-ploigos-customize.adoc +++ b/docs/assembly-ploigos-customize.adoc @@ -6,7 +6,7 @@ ifdef::ProjectVersion[:parent-ProjectVersion: {ProjectVersion}] include::vars/ploigos.adoc[] -[id="{ProjectNameID}"] +[id="{ProjectNameID}-customize"] = Customize Workflow :context: assembly-{ProjectNameID} diff --git a/docs/assembly-ploigos-deploy.adoc b/docs/assembly-ploigos-deploy.adoc index 2cb83a0..9cf24e9 100644 --- a/docs/assembly-ploigos-deploy.adoc +++ b/docs/assembly-ploigos-deploy.adoc @@ -6,7 +6,7 @@ ifdef::ProjectVersion[:parent-ProjectVersion: {ProjectVersion}] include::vars/ploigos.adoc[] -[id="{ProjectNameID}"] +[id="{ProjectNameID}-deploy"] = Deploy using existing infrastructure :context: assembly-{ProjectNameID} diff --git a/docs/assembly-ploigos-workflow.adoc b/docs/assembly-ploigos-workflow.adoc index 4ce8be3..42e92c5 100644 --- a/docs/assembly-ploigos-workflow.adoc +++ b/docs/assembly-ploigos-workflow.adoc @@ -6,12 +6,10 @@ ifdef::ProjectVersion[:parent-ProjectVersion: {ProjectVersion}] include::vars/ploigos.adoc[] -[id="{ProjectNameID}"] +[id="{ProjectNameID}-workflow"] = {ProjectName} - An Opinionated CI/CD Workflow :context: assembly-{ProjectNameID} -At its core the {ProjectName} defines an opinionated workflow used to deliver software from peoples heads to production. This documentation module defines what that that opinionated workflow is. - include::modules/workflow/con-ploigos-workflow-abstracted.adoc[leveloffset=+1] include::modules/workflow/ref-ploigos-workflow-tools.adoc[leveloffset=+1] include::modules/workflow/con-ploigos-workflow-infrastructure.adoc[leveloffset=+1] diff --git a/docs/common/ploigos-terms-definitions.adoc b/docs/common/ploigos-terms-definitions.adoc index 28b32cf..7c5785a 100644 --- a/docs/common/ploigos-terms-definitions.adoc +++ b/docs/common/ploigos-terms-definitions.adoc @@ -1,10 +1,8 @@ [id="ploigos-terms-definitions-{context}"] = {ProjectName} Terms and Definitions -{ProjectShortName}:: {ProjectName} +Workflow:: A {ProjectName} procedure as represented by a drawing. -Workflow:: A {ProjectShortName} procedure as represented by a drawing. +Workflow Abstraction:: A {ProjectName} Workflow as represented by a drawing with no specified tooling to implement that steps of the workflow. -Workflow Abstraction:: A {ProjectShortName} Workflow as represented by a drawing with no specified tooling to implement that steps of the workflow. - -Workflow Implimentaiton:: An implimentation of a {ProjectShortName} Workflow Abstraction with specific tooling. +Workflow Implementation:: An implementation of a {ProjectName} Workflow Abstraction with specific tooling. diff --git a/docs/modules/workflow/con-ploigos-workflow-abstracted.adoc b/docs/modules/workflow/con-ploigos-workflow-abstracted.adoc index 976e19c..1832edd 100644 --- a/docs/modules/workflow/con-ploigos-workflow-abstracted.adoc +++ b/docs/modules/workflow/con-ploigos-workflow-abstracted.adoc @@ -1,100 +1,63 @@ -[id="{ProjectNameID}-workflow-abstracted", reftext="{ProjectName} Abstracted Workflow"] +[id="{ProjectNameID}-workflow-overview", reftext="{ProjectName} Overview"] += Overview -When discussing methods for getting ideas from peoples' heads to production, groups often get lost in talking about technology and tools and avoid the hard work of deciding on the abstract process, or workflow. This section has pictorial renderings of the {ProjectName} opinionated workflow in an abstract form, without any specific technologies or tools. This can be used as a primer for a team to first discuss and decide on their own specific {ProjectName} workflow, and then use the provided templates to draw it to avoid the https://www.jpattonassociates.com/wp-content/uploads/2018/07/glad-we-all-agree-1.png["I'm glad we all agree"] problem that inevitably arises if an abstract concept is not concretely defined by drawing it. +_What is Ploigos?_ +Ploigos is an opinionated workflow used to transform ideas into delivered software in a production environment. Ploigos can be divided into two major components: the Idea Delivery Workflow and the CI/CD Workflow. -[id="{ProjectNameID}-workflow-abstracted-high-level", reftext="{ProjectName} Abstracted Workflow - High Level"] +The Idea Delivery Workflow is an abstract process workflow that defines a high level view of how an organization can best take ideas that solve business problems and implement the solution in a well defined procedure. This process focuses on the specific actions that need to take place without identifying specific tools for each step. Ploigos provides a general use Idea Delivery Workflow that can be further customized to fit specific projects as needed. A workflow first approach identifies the organizational behaviors and expected outcomes to define what quality is. The workflow codifies and enforces these to ensure all aspects of security, compliance, trust, and privacy are addressed. + +The CI/CD Workflow is the implementation of the software development portion of the Idea Delivery Workflow. Ploigos creates a framework for a modular, extensible, and opinionated CD/CD pipeline. Modularity and extensibility are accomplished by defining one or more '*Steps*' in the CD/CD workflow, and automating the *Step* by creating a *StepImplementer*. As part of fulfilling the opinionated aspect, Ploigos provides a number of pre-defined Steps, StepImplementers, and several CI/CD Workflows. + + +[id="{ProjectNameID}-workflow-idea-delivery-high-level", reftext="{ProjectName} Idea Delivery Workflow - High Level"] = Idea Delivery Workflow -In this high level rendering of the {ProjectName} workflow, the majority of steps are Predefined Procedures. That is, they are placeholders for more complex sub workflows containing multiple Procedures, the details of which are defined in <<{ProjectNameID}-workflow-abstracted-detailed>>. This is a "one slide" view of the workflow. +Ploigos defines a reference high level workflow that covers the process of taking the genesis of an idea, idea development, integration, and review. In this high level rendering of the {ProjectName} workflow, the majority of steps are _Predefined Procedures_. These procedures are placeholders for more complex sub workflows containing multiple Procedures, the details of which are defined in later sections. This simplification allows for a 'one slide' view of the workflow. [id="{ProjectNameID}-workflow-abstracted-high-level-image", reftext="{ProjectName} Abstracted Workflow - High Level Image"] -image::ploigos_workflows-Ploigos_Idea_Delivery_Workflow-vertical.png[alt="({ProjectName}) Workflow - High Level",title="({ProjectName}) - High Level ",caption="Workflow Image: ",link=images/ploigos_workflows-Ploigos_Idea_Delivery_Workflow-vertical.png] +image::ploigos_workflows-Ploigos_Idea_Delivery_Workflow-vertical.png[alt="({ProjectName}) Workflow - High Level",title="({ProjectName}) - High Level",caption="Workflow Image: ",link=images/ploigos_workflows-Ploigos_Idea_Delivery_Workflow-vertical.png] -[id="{ProjectNameID}-workflow-components-high-level-{context}"] - -The {ProjectName} Workflow Components are the high level groupings of more complex sub workflows. The following table defines the purpose of these high-level components without emphasis on the more detailed steps each component encapsulates, or the technology of how those get realized. +[id="{ProjectNameID}-workflow-idea-steps-{context}"] == Idea Delivery Workflow Steps Each predefined process represents an encapsulation of steps that are typically necessary in a complete end-to-end pipeline -[cols="a,a,a",options="header"] -|=== -| Predefined Process -| Purpose -| <<{ProjectNameID}-workflow-components-detailed-level-{context}, Components / Steps>> - -| Prioritize Ideas -| Move idea from person's head into a work management tool to track item through to completion. -| Steps vary based on team's backlog prioritization and work planning process - -| Development -| Lay foundation for isolated development environment for developer to begin coding -| -* <> -* <> -* <> - -| Create or update Merge Request to Release Branch -| Deployment of build artifacts into a temporary development environment that will occur when a merge request is created (from a feature branch to the primary release branch), and for any subsequent changes to the merging feature branch -| -* Deploy or Update MR#-DEV Environment -* Validate MR#-DEV Environment Configuration -* Run User Acceptance Tests (UAT) -* Run Runtime Vulnerability Scans -* Run Performance Tests (Limited) -* Mark Merge Request Ready for Peer Review and Merging - -| Detect Change -| Allows pipeline to detect and respond to a defined set of changes in source code repository -| -* <> - -| CI/CD -| Build artifacts are created and tested automatically on a continual basis . Provide developer with ability to make code changes with instant (realtime) deployment of those changes into live development runtime environment. An ability to bypass pipeline to reduce friction during development on feature branches. -| -* Generate Metadata -* Tag Source Code with Metadata -* Run Unit Tests -* Package Application -* Static Code Analysis -* Push Packaged Application to Repository with Metadata -* Create Container Image -* Push Container Image to Repository with Metadata -* Run Container Image Unit Tests -* Run Static Compliance Container Image Scan -* Run Static Vulnerability Container Image Scan -* Push Trusted Container Image to Repository with Metadata -* Deploy or Update TEST Environment -* Validate TEST Environment Configuration -* Run User Acceptance Tests (UAT) -* Run Runtime Vulnerability Scans -* Run Performance Tests (Full) -* Deploy or Update PROD Environment -* Validate PROD Environment Configuration -* Run Canary Testing - - -| Peer Review -| To collaborate with others in the process of reviewing code in order to make sure it works, and in order to improve it where possible -| + +Prioritize Ideas:: +New ideas are placed in a work management tool, refined, and prioritized. The work management tool allows for organization and tracking of ideas as they move through the workflow. Steps will vary based on existing organizational structure and specific tool chosen, but basic requirements involve being able to: record new ideas, backlog management, assign ideas to developers, and track progress and status. + +Development:: +The development procedure where an idea is implemented in software. This includes an isolated development environment for developer to begin coding. + +Create or Update Merge Request to Release Branch:: +The procedure for how merges happen into the release branch in the source code repository. Deployment of build artifacts into a temporary development environment that will occur when a merge request is created (from a feature branch to the primary release branch), and for any subsequent changes to the merging feature branch. Acceptance of a merge request is automated, contingent on passing all gates defined in the CI/CD pipeline for the merging branch, and passing all gates of the CI/CD pipeline after the merge into the primary branch is complete. + +Detect Change:: +The procedure for detecting change in the source code repository, and using that event to initiate appropriate action in the CI/CD procedure. The event can be a change in a feature branch, release branch, a new merge request, etc. + + +CI/CD:: +Build artifacts are created and tested automatically on a continual basis. This provides developers with the ability to make code changes with instant (realtime) deployment of those changes into live development runtime environment. An ability to bypass pipeline to reduce friction during development on feature branches. +The detailed CI/CD Process Workflow section provides more details on this predefined process. + +Retrospective:: +An introspection of the processes used by the team to deliver the software to evaluate the effectiveness of the overall process itself. Allows for modifying a predefined procedure or even modifying the Idea Delivery Workflow itself based on team feedback. + +Peer Review:: +Collaborate with others in the process of reviewing code in order to make sure it works, the code conforms to organizational coding standards, and in order to improve the code where possible. + * Mark Merge Request as Work In Progress (WIP) * Peer Review * Merge to Release Branch * Delete MR#-DEV Deployment Environment -| Retrospective -| -| - -| Release -| -| -|=== +Release:: +Predefined process for how software is released. = CI/CD Process Workflow @@ -103,6 +66,25 @@ image::ploigos_workflows-Ploigos_CI_CD_Workflow_Processes_-_v1_0_0-vertical.png[ [id="{ProjectNameID}-workflow-components-processes-level-{context}"] +Setup:: + +Initial setup of the CI/CD pipeline. This step includes deployment and configuration of a pipeline using Ploigos + +Continuous Integration:: + +Development work that happens on the deployed CI/CD pipeline + +Continuous Deployment - DEV:: + +Deployment of the application into a development environment for the developer to review + +Continuous Deployment - TEST:: + +Deployment of the application into the testing environment for functionality, performance, and acceptance testing. + +Continuous Deployment - PROD:: + +Deployment of the application into the production environment. = Minimum Workflow @@ -111,6 +93,59 @@ image::ploigos_workflows-Ploigos_CI_CD_Workflow_Steps_-_Minimum_-_v1_0_0.png[alt == Minimum Workflow Steps +Detect Change:: + - Detect new/changed/merged branches + + To bring an idea from development into a release (and ultimately production) a developer will create a merge request from feature branch to the primary release branch. The merge request should initially be created as WIP, which indicates this is a "Work in progress" and not yet ready to be merged. The act of creating the merge request from a feature branch to the release branch should trigger the pipeline to be run on the new feature branch. + + + - Start CI/CD workflow for changed branch + + The capability of the CI tool to detect actions at the source control tool. For actions "new merge request" or "changed merge request", the pipeline will run and the subject will be feature branch being merged. For "merge of feature branch to release branch" the pipeline will run and the subject will be the primary release branch. + +Setup:: + - Setup workflow step runner + - Setup PGP keys + +Continuous Integration:: + - Generate metadata + + The pipeline will generate a semantic version based on other metadata, to produce version and image tag to uniquely identify artifacts associated with the pipeline run. This information gets applied to runtime artifacts and container image as labels. + + - Tag Source Code + + This step will take the version created in the "generate metadata" step to tag the source in source control. + + - Package Application Artifact + + Build runtime artifacts, distribution archives, and other necessary artifacts required to run application. + + - Push Application Artifact to repository + + Transfer runtime artifacts into a centralized artifact repository for distribution. + + - Create Container Image + - Push container image to registry + +Continuous Deployment - DEV (Feature Branch):: + - Deploy or Update to dev environment + + Provide a temporary environment for deployment of code changes associated with a feature. If the environment does not already exist, the environment will be created. The lifetime of the environment is limited to the time it takes to implement the feature and merge the changes into the release branch of the primary code repo. At which point the development environment will be deleted. + +Continuous Deployment - TEST (Release Branch):: + - Deploy or Update to test environment + + Deploy image built from the latest release branch to the test environment. + +Continuous Deployment - PROD (Release Branch):: + - Deploy or Update to production environment + + Deploy tested code to shared prod environment with latest feature available to end users + +Report:: + - Generate and publish workflow report + + Provide central dashboard with published test results as an indicator of overall health of system = Standard Workflow @@ -119,215 +154,116 @@ image::ploigos_workflows-Ploigos_CI_CD_Workflow_Steps_-_Standard_-_v1_0_0.png[al == Standard Workflow Steps -[cols="20a,50a,30a",options="header"] -|=== -| Step -| Purpose -| <<{ProjectNameID}-workflow-tool-purposes-{context}, Implementing Tool Category>> - -| [[detailed-component-fork-repository, Fork Repository]] -Fork Repository -| Common with open source projects, a developer will not have direct access to the original repository, so developer will fork the repo and make the changes in own version of the repo and then "pull request" change back to the original repo. -| -* Source Control Tool - -| [[detailed-component-create-idea-branch, Create Idea Branch on Fork of Repository]] -Create Idea Branch on Fork of Repository -| Changes are made to a new branch in forked repo. The branch will follow naming convention that conveys the feature being worked on. -| -* Source Control Tool - -| [[detailed-component-create-wip-merge-request, Create WIP Merge Request to Release Branch on prime repository]] -Create WIP Merge Request to Release Branch on prime repository -| To bring an idea from development into a release (and ultimately production) a developer will create a merge request from feature branch to the primary release branch. The merge request should initially be created as WIP, which indicates this is a "Work in progress" and not yet ready to be merged. The act of creating the merge request from a feature branch to the release branch should trigger the pipeline to be run on the new feature branch. -| -* Source Control Tool - -| [[detailed-component-detect-changed-merge-request, Detect new, changed, or merged, Merge Request to Release Branch]] -Detect new, changed, or merged, Merge Request to Release Branch -| The capability of the CI tool to detect actions at the source control tool. For actions "new merge request" or "changed merge request", the pipeline will run and the subject will be feature branch being merged. For "merge of feature branch to release branch" the pipeline will run and the subject will be the primary release branch. -| -* CI Tool -* Source Control Tool - -| Generate Metadata -| The pipeline will generate a semantic version based on other metadata, to produce version and image tag to uniquely identify artifacts associated with the pipeline run. This information gets applied to runtime artifacts and container image as labels. -| -* CI Tool - -| Tag Source Code with Metadata -| This step will take the version created in the "generate metadata" step to tag the source in source control. -| -* Source Control Tool - -| Run Unit Tests -| Validate that each unit of the software performs as designed. -| -* Application Language Unit Test Tool - -| Package Application -| Build runtime artifacts, distribution archives, and other necessary artifacts required to run application. -| -* Application Language Packaging Tool - -| Static Code Analysis -| The pipeline will perform static analysis on source code to identify defects, vulnerabilities, programmatic and stylistic problems as early in the development life cycle as possible. For example, static analysis is completed prior to building, scanning and deploying the image. -| -* Static Code Analysis Tool - -| Push Packaged Application to Repository with Metadata -| Transfer runtime artifacts into a centralized artifact repository for distribution. -| -* Binary Artifact Upload Tool -* Artifact Repository - -| Create Container Image -| Create the minimal container image that the application will need to run, including the packaged application artifacts. -| -* Container Image Build Tool - -| Run Container Image Unit Tests -| Test container images, verify functionality, and validate the structure and content of the container themselves. -| -* Container Image Unit Test Tool - -| Run Static Compliance Container Image Scan -| Ensure adherence to an organization's security compliance policy by your container image. -| -* Container Image Scanning Tool - -| Run Static Vulnerability Container Image Scan -| Identify software vulnerabilities in your container image. -| -* Container Image Scanning Tool - -| Push Trusted Container Image to Repository with Metadata -| Transfer the verified image to centralized repository with metadata applied as labels to the image. -| -* Container Image Upload Tool -* Image Registry - -| Deploy or Update MR#-DEV Environment -| Provide a temporary environment for deployment of code changes associated with a feature. If the environment does not already exist, the environment will be created. The lifetime of the environment is limited to the time it takes to implement the feature and merge the changes into the release branch of the primary code repo. At which point the development environment will be deleted. -| -* Continuous Deployment Tool - -| Validate MR#-DEV Environment Configuration -| To validate the development test environment matches a given baseline of required objects, and configuration of those objects are correct. Requirements for this step can often come from an enterprise security and compliance team. -| -* Validate Environment Configuration Tool - -| Run User Acceptance Tests (UAT) -| Assess if the system can support day-to-day business and user scenarios and ensure the system is sufficient and correct for business usage. -| -* UAT Tool - -| Run Runtime Vulnerability Scans -| Analyze the run-time activity of a container for any vulnerabilities or weak runtime security that may not manifest during static analysis. -| -* Runtime Vulnerability Scanning Tool - -| Run Performance Tests (Limited) -| To identify and eliminate the performance bottlenecks in the application. -| -* Performance Testing Tool - -| Mark Merge Request Ready for Peer Review and Merging -| The new code must have a specific number of approving reviewers before the code can be merged. This ensures the quality and completeness of the solution. Typically the peer review process is managed by the source control tool. -| -* Source Control Tool - -| Remove "WIP" from Merge Request -| This step is an indicator that the new code is of sufficient quality (in developer's opinion) to be merged into the main branch of the primary repository. Typically this step is done by the developer, and involves a change to the name and state (from "draft" merge request) within the source control tool. -| -* Source Control Tool - -| Connect IDE to MR#-DEV Environment -| If the merge request is still considered by the developer to be a work in progress, development will continue. The developer's IDE should support (typically via plugins) the ability to connect directly to the development environment. -| -* IDE & Container Platform - -| Live Development and Testing in MR#-DEV Environment -| Code changes made inside the developer's IDE should automatically be moved to a live environment quickly and with minimal friction. The development tooling should facilitate iterating and deploying new versions of the code, as well as testing. -| -* IDE & Container Platform - -| Commit Change to Idea Branch on Fork of Repository -| The developer will make updates to idea branch (or feature branch) on his/her forked repository. This action will cause the pipeline to run against the feature branch, and allows development to perform code update/deploy iterations until code is suitable for review and merge to main branch. -| -* Source Control Tool - -| Peer Review -| Collaborate with teammates on code change to ensure the quality and completeness of the solution. -| -* Peer Review Tool - -| Merge to Release Branch -| Once peer review determines code ready, the developer will merge code from feature branch into the main branch of the primary repo. This action will cause the pipeline to run against the release branch and trigger deployment to shared test environment. -| -* Source Control Tool - -| Delete MR#-DEV Deployment Environment -| Once merge from feature branch to main branch is complete, clean up the environment infrastructure, so as to minimize resource consumption. -| -* Kubernetes Resources Creation Tool - -| Mark Merge Request as Work In Progress (WIP) -| Generally, a merge request will initially be created in this draft state, and remain in this state for several development iteration of code update, deploy, test, and peer review. -| -* Source Control Tool - -| Deploy or Update TEST Environment -| Deploy image built from the latest release branch to the test environment. -| -* Continuous Deployment Tool - -| Validate Test Environment Configuration -| Using predefined rules, validate the configuration files used to deploy the test environment -| -* Validate Environment Configuration Tool - -| Run Performance Tests (Full) -| Execute tests to determine the speed, responsiveness and stability -| -* Performance Testing Tool - -| Create PROD Environment -| Create PROD Environment as-needed -| -* Kubernetes Resources Creation Tool - -| Deploy or Update PROD Environment -| Deploy tested code to shared prod environment with latest feature available to end users -| -* Continuous Deployment Tool - -| Validate Prod Environment Configuration -| Verify that the deployment environment has been built successfully and configured according to predefined specifications and rules -| -* Validate Environment Configuration Tool - -| Run Canary Testing -| Allows you to roll out new code/features to a subset of end-users as an initial test. -| -* Canary Testing Tool - -| Collect, Bundle, & Publish Test Reports and Metadata -| Provide central dashboard with published test results as an indicator of overall health of system -| -* CI Tool - -| Collect Lessons Learned -| Collect, understand and act upon positive and negative lessons learned. -| -* Discussion - -| Celebrate -| Work hard - now play hard! -| -* Discussion - -|=== +Detect Change:: + - Detect new/changed/merged branches + + To bring an idea from development into a release (and ultimately production) a developer will create a merge request from feature branch to the primary release branch. The merge request should initially be created as WIP, which indicates this is a "Work in progress" and not yet ready to be merged. The act of creating the merge request from a feature branch to the release branch should trigger the pipeline to be run on the new feature branch. + + + - Start CI/CD workflow for changed branch + + The capability of the CI tool to detect actions at the source control tool. For actions "new merge request" or "changed merge request", the pipeline will run and the subject will be feature branch being merged. For "merge of feature branch to release branch" the pipeline will run and the subject will be the primary release branch. + +Setup:: + - Setup workflow step runner + - Setup PGP keys + +Continuous Integration:: + - Generate metadata + + The pipeline will generate a semantic version based on other metadata, to produce version and image tag to uniquely identify artifacts associated with the pipeline run. This information gets applied to runtime artifacts and container image as labels. + + - Tag Source Code + + This step will take the version created in the "generate metadata" step to tag the source in source control. + + - Run Unit Tests + + Validate that each unit of the software performs as designed. + + - Package Application Artifact + + Build runtime artifacts, distribution archives, and other necessary artifacts required to run application. + + - Run Static Code Analysis + + The pipeline will perform static analysis on source code to identify defects, vulnerabilities, programmatic and stylistic problems as early in the development life cycle as possible. For example, static analysis is completed prior to building, scanning and deploying the image. + + - Push Application Artifact to repository + + Transfer runtime artifacts into a centralized artifact repository for distribution. + + - Create Container Image + + Assemble the minimal container image that the application will need to run, including the packaged application artifacts. Test container images, verify functionality, and validate the structure and content of the container themselves. + + - Run Static Image Scan: Compliance + + Ensure adherence to an organization's security compliance policy by your container image. + + - Run Static Image Scan: Vulnerability + + Identify software vulnerabilities in your container image. + + - Push container image to registry + + Transfer the verified image to centralized repository with metadata applied as labels to the image. + + - Sign Container Image + + Sign the container image to allow validating image source and ensure image has not been tampered with. + +Continuous Deployment - DEV (Feature Branch):: + - Deploy or Update to dev environment + + Provide a temporary environment for deployment of code changes associated with a feature. If the environment does not already exist, the environment will be created. The lifetime of the environment is limited to the time it takes to implement the feature and merge the changes into the release branch of the primary code repo. At which point the development environment will be deleted. + + - Validate environment configuration + + To validate the development test environment matches a given baseline of required objects, and configuration of those objects are correct. Requirements for this step can often come from an enterprise security and compliance team. + + - Run user acceptance tests + + Assess if the system can support day-to-day business and user scenarios and ensure the system is sufficient and correct for business usage. + + - Run performance tests (limited) + + Run limited performance tests to ensure basic performance requirements are met + +Continuous Deployment - TEST (Release Branch):: + - Deploy or Update to test environment + + Deploy image built from the latest release branch to the test environment. + + - Validate environment configuration + + Using predefined rules, validate the configuration files used to deploy the test environment + + - Run user acceptance tests + + Run automated user accepting tests + + - Run performance tests (full) + + To identify and eliminate the performance bottlenecks in the application. + +Continuous Deployment - PROD (Release Branch):: + - Deploy or Update to production environment + + Deploy tested code to shared prod environment with latest feature available to end users + + - Validate environment configuration + + Verify that the deployment environment has been built successfully and configured according to predefined specifications and rules + + - Run Canary Testing + + Allows you to roll out new code/features to a subset of end-users as an initial test. + +Report:: + - Generate and publish workflow report + + Provide central dashboard with published test results as an indicator of overall health of system = Workflow Source Files diff --git a/docs/modules/workflow/con-ploigos-workflow-infrastructure.adoc b/docs/modules/workflow/con-ploigos-workflow-infrastructure.adoc index 0316210..e371ac1 100644 --- a/docs/modules/workflow/con-ploigos-workflow-infrastructure.adoc +++ b/docs/modules/workflow/con-ploigos-workflow-infrastructure.adoc @@ -1,7 +1,61 @@ [id="{ProjectNameID}-workflow-infrastructure", reftext="{ProjectName} Infrastructure Requirements"] = Infrastructure Requirements -Reference information on the infrastructure used to develop and test the MVP of the <<{ProjectNameID}-workflow-assembly-{ProjectNameID}-workflow>>. +Below is the infrastructure used to develop and test the Minimum Viable Product(MVP) of the Ploigos CI/CD Workflow Reference Implementation. The most basic requirement for the deployment of the Ploigos CI/CD Workflow is an operational OCP 4.x cluster. + +.Red Hat OpenShift Sizing +[cols="a,a,a,a,a,a",options="header"] +|=== +| Node +| CPUs +| Memory (GB) +| Disk (GB) +| AWS EC2 Instance Type +| Sizing Source + +| Control 0 +| 8 +| 32 +| 120 +| m4.2xlarge +| https://docs.openshift.com/container-platform/latest/scalability_and_performance/recommended-host-practices.html#master-node-sizing_[OCP 4 Docs - Master Node Sizing] + +| Control 1 +| 8 +| 32 +| 120 +| m4.2xlarge +| https://docs.openshift.com/container-platform/latest/scalability_and_performance/recommended-host-practices.html#master-node-sizing_[OCP 4 Docs - Master Node Sizing] + +| Control 2 +| 8 +| 32 +| 120 +| m4.2xlarge +| https://docs.openshift.com/container-platform/latest/scalability_and_performance/recommended-host-practices.html#master-node-sizing_[OCP 4 Docs - Master Node Sizing] + +| Compute 0 +| 8 +| 32 +| 120 +| m4.2xlarge +| Based on Containerized Tool Sizing needs + +| Compute 1 +| 8 +| 32 +| 120 +| m4.2xlarge +| Based on Containerized Tool Sizing needs + +| Compute 2 +| 8 +| 32 +| 120 +| m4.2xlarge +| Based on Containerized Tool Sizing needs +|=== + == Suggested Sizing .Containerized Tool Sizing @@ -33,7 +87,7 @@ Reference information on the infrastructure used to develop and test the MVP of | https://access.redhat.com/documentation/en-us/red_hat_quay/3.2/html/deploy_red_hat_quay_-_basic/preparing_for_red_hat_quay_basic#prerequisites[Quay Docs - Preparing for Red Hat Quay Basic - Prerequisites], https://access.redhat.com/documentation/en-us/red_hat_quay/3/html/deploy_red_hat_quay_on_openshift_with_quay_setup_operator/architecture[Operator and Quay Architecture] -| Red Hat Quay - Operator Based - DB (Crunchy Data PostgrSQL) +| Red Hat Quay - Operator Based - DB (Crunchy Data PostgreSQL) | 2 / Operator Governed | 8 / Operator Governed | @@ -78,56 +132,3 @@ https://access.redhat.com/documentation/en-us/red_hat_quay/3/html/deploy_red_hat | N/A (embedded in CI container) | |=== - -.Red Hat OpenShift Sizing -[cols="a,a,a,a,a,a",options="header"] -|=== -| Node -| CPUs -| Memory (GB) -| Disk (GB) -| AWS EC2 Instance Type -| Sizing Source - -| Master 0 -| 8 -| 32 -| 120 -| m4.2xlarge -| https://docs.openshift.com/container-platform/4.3/scalability_and_performance/recommended-host-practices.html#master-node-sizing_[OCP 4 Docs - Master Node Sizing] - -| Master 1 -| 8 -| 32 -| 120 -| m4.2xlarge -| https://docs.openshift.com/container-platform/4.3/scalability_and_performance/recommended-host-practices.html#master-node-sizing_[OCP 4 Docs - Master Node Sizing] - -| Master 2 -| 8 -| 32 -| 120 -| m4.2xlarge -| https://docs.openshift.com/container-platform/4.3/scalability_and_performance/recommended-host-practices.html#master-node-sizing_[OCP 4 Docs - Master Node Sizing] - -| Compute 0 -| 8 -| 32 -| 120 -| m4.2xlarge -| Based on Containerized Tool Sizing needs - -| Compute 1 -| 8 -| 32 -| 120 -| m4.2xlarge -| Based on Containerized Tool Sizing needs - -| Compute 2 -| 8 -| 32 -| 120 -| m4.2xlarge -| Based on Containerized Tool Sizing needs -|=== diff --git a/docs/modules/workflow/proc-ploigos-workflow-reference-impl.adoc b/docs/modules/workflow/proc-ploigos-workflow-reference-impl.adoc index c896006..b534c63 100644 --- a/docs/modules/workflow/proc-ploigos-workflow-reference-impl.adoc +++ b/docs/modules/workflow/proc-ploigos-workflow-reference-impl.adoc @@ -2,4 +2,175 @@ = Reference Implementation -TODO +The following guide shows how to deploy the reference implementation of the Ploigos Standard Workflow into an OCP 4.x cluster with a simple quarkus application as a demo. + + +== Prerequisites + +* Operational OCP 4.x cluster + + +== Installing the Ploigos Software Factory Operator + +See https://github.com/ploigos/ploigos-software-factory-operator[ploigos-software-factory-operator] for latest information. + + +. Login to an OCP cluster as `cluster-admin` +. Import the RedHatGov Operator Catalog (Create a custom `CatalogSource`) ++ +---- +oc apply -f - << EOF +apiVersion: operators.coreos.com/v1alpha1 +kind: CatalogSource +metadata: + name: redhatgov-operators + namespace: openshift-marketplace +spec: + sourceType: grpc + image: quay.io/redhatgov/operator-catalog:latest + displayName: Red Hat NAPS Community Operators + publisher: RedHatGov +EOF +---- ++ +. Create a project named *devsecops* for the pipeline tooling+ ++ +---- +oc new-project devsecops +---- ++ +. Remove restrictive limit ranges *NOTE: Required for RHPDS clusters* ++ +---- +oc delete limitrange --all -n devsecops +---- ++ +. Install the 'Ploigos Software Factory Operator' +* In the OCP Web Console, navigate to: +** *Operators -> OperatorHub* +** Search for "Ploigos Software Factory Operator" +** Select Operator and click *Install* + +== Deploy the Ploigos Platform and Pipeline + +. In the OCP WebUI, Select the *devsecops* namespace +. Deploy the Ploigos Platform +* Navigate to *Installed Operators -> Ploigos Software Factory* +* Inside the software factory operator screen, click *[+ TSSCPlatform]* +* Wait for the deployment to finish +** Navigate to the _tssc-operator-controller-manager_ pod and watch the logs for the "PLAY RECAP " line +. Deploy the Ploigos Pipeline +* Navigate to *Installed Operators -> Ploigos Software Factory* +* Inside the software factory operator screen, click *[+ TSSCPipeline]* +. Access the deployed Jenkins pod to watch progress +* Navigate to *Routes -> Jenkins* and click the Jenkins Route +* Login to Jenkins with Openshift Credentials +* Navigate to *platform -> reference-quarkus-mvn_jekins_workflow-standard -> main* +* Click "Open Blue Ocean" (Vertical menu on left side) +* Watch the pipeline progress and wait for 100% + +== Add SSH Key to Gitea + +. Obtain the Gitea credentials +* In the *devsecops* namespace, navigate to *Secrets* +* From the secret *tssc-service-account*, copy the username and password +* Navigate to *Routes* and open the Gitea route +. Login to Gitea with the tssc-service-account username/password +. Add SSH Key to account in Gitea +* Click the Profile Icon (top left corner) +* Select Settings +* Select 'Add Key' +* Enter a public key +* Click 'Add Key' + +== Setup the Development Environment with reference application +The following directions are designed to setup a development environment for Ploigos. By creating forks of the official repositories, a developer can customize the workflow as needed. + +. Fork required Ploigos repositories +* Create a fork of: https://github.com/ploigos/ploigos-step-runner +* Create a fork of: https://github.com/ploigos/ploigos-jenkins-library +* Clone repositories above to local machine +. Configure the Jenkinsfile to point to the *ploigos-step-runner* fork +* In Gitea, navigate to the quarkus reference application Jenkinsfile `cicd/Jenkinsfile` +* Add the following to the end of the Jenkinsfile: + + stepRunnerUpdateLibrary: true, + stepRunnerLibSourceUrl: "git+​ https://github.com/[your-step-runner-fork]@[the-branch-to-use​ ]" + +** Replace [your-step-runner-fork] with your fork URL +** Replace [the-branch-to-use] with target branch +* Commit Changes +* Initiate Pull Request +* Upon successful Pull Request build, Merge changes into main branch of the reference-quarkus-mvn-jenkins repository +. Configure the Jenkinsfile to point to the *ploigos-jenkins-library* fork +* In Gitea, navigate to the quarkus reference application Jenkinsfile `cicd/Jenkinsfile` +* Replace the *remote:* line of the Jenkinsfile with the forked jenkins library URL + + // Load the TSSC Jenkins Library + library identifier: 'ploigos-jenkins-library@main', + retriever: modernSCM([ + $class: 'GitSCMSource', + remote: 'https://github.com//ploigos-jenkins-library.git' + ]) + +** NOTE: also update the *library identifier:* line with the branch name if it differs from `main` +* Commit Changes +* Initiate Pull Request +* Upon successful Pull Request build, Merge changes into main branch of the reference-quarkus-mvn-jenkins repository + +== Changing from the Ploigos CI/CD Standard Workflow to the Ploigos CI/CD Minimum Workflow + +. In Gitea, navigate to the quarkus reference application Jenkinsfile `cicd/Jenkinsfile`: + + // Load the Ploigos Jenkins Library + library identifier: 'ploigos-jenkins-library@v0.17.0', + retriever: modernSCM([ + $class: 'GitSCMSource', + remote: 'https://github.com/ploigos/ploigos-jenkins-library.git' + ]) + + // run the pipeline + ploigosWorkflowStandard( + stepRunnerConfigDir: 'cicd/ploigos-step-runner-config/', + pgpKeysSecretName: 'pgp-keys-ploigos-workflow-ref-quarkus-mvn-jenkins-std-fruit', + + workflowServiceAccountName: 'ploigos-workflow-ref-quarkus-mvn-jenkins-std-fruit', + + workflowWorkerImageDefault: 'ploigos/ploigos-ci-agent-jenkins:v0.16.0', + workflowWorkerImageUnitTest: 'ploigos/ploigos-tool-maven:v0.16.0', + workflowWorkerImagePackage: 'ploigos/ploigos-tool-maven:v0.16.0', + workflowWorkerImageStaticCodeAnalysis: 'ploigos/ploigos-tool-sonar:v0.16.0', + workflowWorkerImagePushArtifacts: 'ploigos/ploigos-tool-maven:v0.16.0', + workflowWorkerImageContainerOperations: 'ploigos/ploigos-tool-containers:v0.16.0', + workflowWorkerImageContainerImageStaticComplianceScan: 'ploigos/ploigos-tool-openscap:v0.16.0', + workflowWorkerImageContainerImageStaticVulnerabilityScan: 'ploigos/ploigos-tool-openscap:v0.16.0', + workflowWorkerImageDeploy: 'ploigos/ploigos-tool-argocd:v0.16.0', + workflowWorkerImageValidateEnvironmentConfiguration: 'ploigos/ploigos-tool-config-lint:v0.16.0', + workflowWorkerImageUAT: 'ploigos/ploigos-tool-maven:v0.16.0' + ) + +. Rename the function `ploigosWorkflowStandard` to `ploigosWorkflowMinimal` and Remove *workflowWorker* Lines to conform to the Ploigos Minimal Workflow: + + // Load the Ploigos Jenkins Library + library identifier: 'ploigos-jenkins-library@v0.17.0', + retriever: modernSCM([ + $class: 'GitSCMSource', + remote: 'https://github.com/ploigos/ploigos-jenkins-library.git' + ]) + + // run the pipeline + ploigosWorkflowMinimal( + stepRunnerConfigDir: 'cicd/ploigos-step-runner-config/', + pgpKeysSecretName: 'pgp-keys-ploigos-workflow-ref-quarkus-mvn-jenkins-std-fruit', + + workflowServiceAccountName: 'ploigos-workflow-ref-quarkus-mvn-jenkins-std-fruit', + workflowWorkerImageDefault: 'ploigos/ploigos-ci-agent-jenkins:v0.16.0', + workflowWorkerImagePackage: 'ploigos/ploigos-tool-maven:v0.16.0', + workflowWorkerImagePushArtifacts: 'ploigos/ploigos-tool-maven:v0.16.0', + workflowWorkerImageContainerOperations: 'ploigos/ploigos-tool-containers:v0.16.0', + workflowWorkerImageDeploy: 'ploigos/ploigos-tool-argocd:v0.16.0', + ) + +. Commit Changes +. Initiate Pull Request +. Upon successful Pull Request build, Merge changes into main branch of the reference-quarkus-mvn-jenkins repository