metadata-entry-form

CIOOS Metadata entry form

System Architecture

Below is the system architecture diagram which provides an overview of the data flow and interaction between components within the application:

For a more interactive and detailed view, see the Lucidchart Diagram.

Local Development Installation

To simplify the local development, we highly recommend to clone the repo locally and integrate it via vscode within the predefined Dev Container.

In the container, run the following steps:

1. Copy .env.sample to .env

cp .env.sample .env

2. Login with firebase

firebase login

3. Select firebase project:

firebase use dev

4. Go to firebase-functions directory and start the emulator, this will install the javascript and python functions dependancies and activate the firebase local emulator :

cd firebase-functions
bash emulate-functions.sh 

5. In a second terminal, start the frontent in development. From the base directory of this project run:

npm install
npm start

Monitoring

Monitoring of production site availability is done via the cioos-upptime and notices are posted to the CIOOS cwatch-upptime slack channel. Error collection is performed by sentry and reported in the cioos-metadata-entry-form project.

Deploy to production site at GitHub pages

Pushes to master automatically deploy to https://cioos-siooc.github.io/metadata-entry-form/

Or manually deploy any branch with

npm run deploy

Deploy to dev preview sites

Firebase hosted preview sites are created for all pull requests. Once a pull request is generated, check the pull request comments on github for the link to the preview site. Deployment is handled by the firebase-hosting-pull-request.yml github action. Preview sites are deleted during a pull request close event or after 30 days of inactivity on the pull request. Any commit to the pull request branch will reset the timer.

Deployment and Configuration of Firebase Functions

Our Firebase Functions infrastructure utilizes both GitHub Actions for automated and manual deployments and parameterized configuration for management of environment variables and credentials.

Automated and Manual Deployment with GitHub Actions

We use a GitHub Actions workflow named firebase-deploy for deploying Firebase Functions. This workflow is triggered automatically on push to the main branch but can also be executed manually for feature branches.

Workflow Features

• Automated Deployments on Push to Main: Ensures that any changes merged into the main branch are automatically deployed to Firebase.
• Manual Deployment Option: Allows for manual deployments of specific branches, useful for testing changes in feature branches.
• Environment Variables and Secrets: Uses GitHub Secrets to populate a virtual .env file with necessary configurations for the deployment process.

Manual Deployment Steps

1. Go to the "Actions" tab in the GitHub repository.
2. Select the firebase-deploy workflow.
3. Click "Run workflow", select the branch to deploy, and initiate the workflow.

Deploying to Development Project

To deploy updated Firebase functions to the "cioos-metadata-form-dev-258dc" development project, follow these steps:

1. Login to firebase

   firebase login

2. Ensure your local setup is linked to the correct Firebase project by using the Firebase CLI to login and select the "cioos-metadata-form-dev-258dc" project.

   firebase use cioos-metadata-form-dev-258dc

3. Make necessary changes to your Firebase functions.

4. Deploy the changes by running the command: From the ./firebase-functions/functions directory run the command:

   firebase deploy --only functions

This will deploy the updated functions to the development project. The GitHub Action for deploying to the preview URL is already configured to use this development project, ensuring that any previews generated from pull requests will interact with the updated dev functions instead of the production version.

GitHub Secrets and .env File Creation

The workflow utilizes the following secrets to create the virtual .env file for the deployment process:

# Email notifications account (do not commit real credentials)
GMAIL_USER=
GMAIL_PASS=

# AWS Translate credentials
AWS_REGION=
AWS_ACCESSKEYID=
AWS_SECRETACCESSKEY=

# GitHub token (repo + pages scopes) for issue generation
GITHUB_AUTH=

# Environment settings
# Set REACT_APP_DEV_DEPLOYMENT to true to connect to dev Firebase project
REACT_APP_DEV_DEPLOYMENT=false

# Use local emulators for Firebase services
# Note: if using local emulators, ensure the emulators are running (firebase emulators:start)
REACT_APP_FIREBASE_LOCAL_FUNCTIONS=true
REACT_APP_FIREBASE_LOCAL_DATABASE=false

# Google Cloud API keys
# Prod project: https://console.cloud.google.com/apis/credentials?project=cioos-metadata-form-8d942
REACT_APP_GOOGLE_CLOUD_API_KEY=
# Dev project: https://console.cloud.google.com/apis/credentials?project=cioos-metadata-form-dev-258dc
REACT_APP_GOOGLE_CLOUD_API_KEY_DEV=

Using Parameterized Configuration in Firebase Functions

Our Firebase Functions leverage parameterized configuration for managing sensitive information. This helps prevent functions from being deployed with missing configurations/credentials.

For details on defining and accessing these parameters, refer to the official Firebase documentation.

Deployment Considerations with Parameters

• Local Development: Use a .env file within the functions directory for local development, mirroring the setup of parameters used in production environments.
• Firebase CLI Prompt: The CLI may prompt for parameter values during deployment if they are not preset, ensuring functions are correctly configured.
• Firebase Console Management: Parameters can also be managed within the Firebase Console.

Security Considerations

The use of GitHub Secrets and the creation of a virtual .env file during the workflow run ensures that sensitive information is handled securely, without persisting in the repository or exposing it beyond the lifecycle of the workflow execution.

• Exclude .env Files from Version Control: Ensure .env files are not included in version control to prevent exposure of sensitive data.
• Temporary .env Files: The .env file created during the GitHub Actions workflow is virtual and transient. It exists only for the duration of the workflow run and is not committed to the repository.

Deploying Firebase Realtime Database Security Rules

Overview

Deploying Firebase Realtime Database security rules via the Firebase CLI is recommended to facilitate version control and consistency across development workflows.

Setting Up Rules

• Edit Rules: Modify the Realtime Database security rules directly in the database.rules.json file.
• Version Control: Ensure the rules file is tracked in git to maintain a history of changes.

Define targets

This project has two databases: cioos-metadata-form-8d942 (this is the default/main db for production) and cioos-metadata-form-dev-258dc (dev). Use Firebase CLI targets to manage rules deployment:

firebase target:apply database prod cioos-metadata-form-8d942
firebase target:apply database dev cioos-metadata-form-dev-258dc

Configure firebase.json

Update your firebase.json to map .rules files to your targets.

{
  "database": [
    {
      "target": "prod",
      "rules": "database.rules.json"
    },
    {
      "target": "dev",
      "rules": "database.rules.json"
    }
  ]
}

Deployment

Deploy your database rules using the Firebase CLI:

# Deploy to a specific environment
firebase deploy --only database:dev  # For development
firebase deploy --only database:prod # For production

Important Considerations

• Deployment Overrides: Deploying via the Firebase CLI overwrites any existing rules in the Firebase console. Ensure the .rules file reflects the latest ruleset. Keep the rules file in sync with any console edits to avoid unintended overwrites.
• Version Control: Use version control to track changes and collaborate on rule development.
• Testing: Thoroughly test your rules in a development or staging environment before deploying to production.

Review the Firebase CLI documentation for more details on managing project resources.

Hosting on github and Authentication

When hosting the application in a new place there are a couple of things to update.

• You must add your new domain to the allowed list for authenication in firebase. https://console.firebase.google.com/u/0/project/cioos-metadata-form/authentication/settings https://console.firebase.google.com/u/0/project/cioos-metadata-form-dev-258dc/authentication/settings

• You have to allow your domain under Website restrictions for the firebase browser key https://console.cloud.google.com/apis/credentials/key/405d637a-efd4-48f5-95c6-f0af1d7f4889?project=cioos-metadata-form https://console.cloud.google.com/apis/credentials/key/23d360a3-4b55-43f2-bc1c-b485371c0e07?project=cioos-metadata-form-dev-258dc