From 1eddacb81b36858e83a63ef13aa32c29c5f792c1 Mon Sep 17 00:00:00 2001 From: vinny Date: Fri, 21 May 2021 15:29:33 -0400 Subject: [PATCH 1/6] HARMONY-388: Update README to reflect recent changes and additional guidance on environment setup --- .gitignore | 1 + README.md | 12 ++++-------- 2 files changed, 5 insertions(+), 8 deletions(-) diff --git a/.gitignore b/.gitignore index 282f962..ab71e30 100644 --- a/.gitignore +++ b/.gitignore @@ -113,6 +113,7 @@ venv/ ENV/ env.bak/ venv.bak/ +harmony-ntz # Spyder project settings .spyderproject diff --git a/README.md b/README.md index 275fa99..52525d7 100644 --- a/README.md +++ b/README.md @@ -35,7 +35,7 @@ and update the `.env` with the correct values. #### Python & Project Dependencies (Optional) -If you would like to do local development outside of Docker, install Python, create a Python virtualenv, +If you would like to do local development outside of Docker, install Python, create a Python virtual environment, and install the project dependencies. If you have [pyenv](https://github.com/pyenv/pyenv) and @@ -48,16 +48,12 @@ create a virtualenv: $ pyenv version > .python-version The last step above creates a local .python-version file which will be automatically activated when cd'ing into the -directory if pyenv-virtualenv has been initialized in your shell (See the pyenv-virtualenv docs linked above). +directory if pyenv-virtualenv has been initialized in your shell (See the pyenv-virtualenv docs linked above). You can alternatively create your virtual environment using [Python's venv module](https://docs.python.org/3.7/library/venv.html). Install project dependencies: $ pip install -r requirements/core.txt -r requirements/dev.txt -NOTE: All steps in this README which install dependencies need to be performed while on the NASA VPN -in order to download and install the Harmony Service Lib, which is published on the -[Nexus artifact repository](https://maven.earthdata.nasa.gov/). - ### Development with Docker #### Testing & Running the Service Independently @@ -83,7 +79,7 @@ the local clone of that Harmony Service Lib and any changes made to it: $ LOCAL_SVCLIB_DIR=../harmony-service-lib-py bin/test-in-docker Finally, run the service using an example Harmony operation request -([example/harmony-operation.json](example/harmony-operation.json) as input. This will reflect +([example/harmony-operation.json](example/harmony-operation.json)) as input. This will reflect local changes to this repo, but will not include local changes to the Harmony Service Lib. $ bin/run-in-docker example/harmony-operation.json @@ -96,7 +92,7 @@ To run the example and also include local Harmony Service Lib changes: *Without local Harmony Service Lib changes*: -Be sure your environment is pointed to the Minikube Docker daemon: +If using Minikube, be sure your environment is pointed to the Minikube Docker daemon: $ eval $(minikube docker-env) From 400fd5623974f83f8f04fc0eac4c495e3fbd69c9 Mon Sep 17 00:00:00 2001 From: vinny Date: Fri, 28 May 2021 16:03:55 -0400 Subject: [PATCH 2/6] HARMONY-388: Add comment about pulling image from docker --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index 52525d7..04bd1b7 100644 --- a/README.md +++ b/README.md @@ -56,6 +56,9 @@ Install project dependencies: ### Development with Docker +If you'd rather not build the image locally (as instructed below), +you can simply pull the latest: `docker pull harmonyservices/netcdf-to-zarr` + #### Testing & Running the Service Independently To run unit tests, coverage reports, or run the service on a sample message _outside_ of the From 431aaf33cb49a18c1ec14d45c55bef538ce0c0c1 Mon Sep 17 00:00:00 2001 From: vinny Date: Wed, 2 Jun 2021 10:17:34 -0400 Subject: [PATCH 3/6] HARMONY-388: Use makefile like in other python projects --- Makefile | 25 +++++++++++++++++++++++ README.md | 55 ++++++++++++-------------------------------------- example/dotenv | 6 ++++++ 3 files changed, 44 insertions(+), 42 deletions(-) create mode 100644 Makefile diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..c8aa291 --- /dev/null +++ b/Makefile @@ -0,0 +1,25 @@ +.PHONY: install test lint build-image push-image build-test-image test-in-docker run-in-docker + +install: + pip install -r requirements/core.txt -r requirements/dev.txt + +test: + bin/test + +lint: + flake8 harmony_netcdf_to_zarr + +build-image: + LOCAL_SVCLIB_DIR=${LOCAL_SVCLIB_DIR} bin/build-image + +push-image: + bin/push-image + +build-test-image: + bin/build-test-image + +test-in-docker: + LOCAL_SVCLIB_DIR=${LOCAL_SVCLIB_DIR} bin/test-in-docker + +run-in-docker: + LOCAL_SVCLIB_DIR=${LOCAL_SVCLIB_DIR} bin/run-in-docker example/harmony-operation.json \ No newline at end of file diff --git a/README.md b/README.md index 04bd1b7..d94b93f 100644 --- a/README.md +++ b/README.md @@ -35,61 +35,43 @@ and update the `.env` with the correct values. #### Python & Project Dependencies (Optional) -If you would like to do local development outside of Docker, install Python, create a Python virtual environment, -and install the project dependencies. - -If you have [pyenv](https://github.com/pyenv/pyenv) and -[pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv) installed (recommended), install Python and -create a virtualenv: - - $ pyenv install 3.7.4 - $ pyenv virtualenv 3.7.4 harmony-ntz - $ pyenv activate harmony-ntz - $ pyenv version > .python-version - -The last step above creates a local .python-version file which will be automatically activated when cd'ing into the -directory if pyenv-virtualenv has been initialized in your shell (See the pyenv-virtualenv docs linked above). You can alternatively create your virtual environment using [Python's venv module](https://docs.python.org/3.7/library/venv.html). +If you would like to do local development outside of Docker, install Python (3.7.4), and create a +[Python virtual environment](https://github.com/nasa/harmony-service-example/blob/main/ENVHELP.md). Install project dependencies: - $ pip install -r requirements/core.txt -r requirements/dev.txt + $ make install ### Development with Docker If you'd rather not build the image locally (as instructed below), you can simply pull the latest: `docker pull harmonyservices/netcdf-to-zarr` +Some of the [Makefile](./Makefile) targets referenced below include an optional argument that allows us to use a local copy of +harmony-service-lib-py (which is useful for concurrent development): `$ make target-name LOCAL_SVCLIB_DIR=../harmony-service-lib-py` + #### Testing & Running the Service Independently To run unit tests, coverage reports, or run the service on a sample message _outside_ of the entire Harmony stack, start by building new runtime and test images: -*IMPORTANT*: Be sure to do these steps in a shell in which has *not* been updated to point to +*IMPORTANT*: If Minikube is installed, be sure to do these steps in a shell in which has *not* been updated to point to the Minikube Docker daemon. This is usually done via a shell `eval` command. Doing so will cause tests and the service to fail due to limitations in Minikube. - $ bin/build-image - $ bin/build-test-image + $ make build-image + $ make build-test-image Run unit tests and generate overage reports. This will mount the local directory into the container and run the unit tests. So all tests will reflect local changes to the service. - $ bin/test-in-docker - -You may be concurrently making changes to the Harmony Service Lib. To run the unit tests using -the local clone of that Harmony Service Lib and any changes made to it: - - $ LOCAL_SVCLIB_DIR=../harmony-service-lib-py bin/test-in-docker + $ make test-in-docker Finally, run the service using an example Harmony operation request ([example/harmony-operation.json](example/harmony-operation.json)) as input. This will reflect local changes to this repo, but will not include local changes to the Harmony Service Lib. - $ bin/run-in-docker example/harmony-operation.json - -To run the example and also include local Harmony Service Lib changes: - - $ LOCAL_SVCLIB_DIR=../harmony-service-lib-py bin/run-in-docker example/harmony-operation.json + $ make run-in-docker #### Testing & Running the Service in Harmony @@ -101,20 +83,9 @@ If using Minikube, be sure your environment is pointed to the Minikube Docker da Build the image: - $ bin/build-image - -You can now run a workflow in your local Harmony stack and it will execute using this image. - -*With local Harmony Service Lib changes*: - -To run this service in Harmony *with* a local copy of the Service Lib, build -the image, but specify the location of the local Harmony Service Lib clone: - - $ LOCAL_SVCLIB_DIR=../harmony-service-lib-py bin/build-image + $ make build-image You can now run a workflow in your local Harmony stack and it will execute using this image. -Note that this also means that the image needs to be rebuilt (using the same command) to -include any further changes to the Harmony Service Lib or this service. ### Development without Docker @@ -122,7 +93,7 @@ include any further changes to the Harmony Service Lib or this service. Run tests with coverage reports: - $ bin/test + $ make test Run an example: diff --git a/example/dotenv b/example/dotenv index 157e177..ae18880 100644 --- a/example/dotenv +++ b/example/dotenv @@ -20,3 +20,9 @@ USE_LOCALSTACK=true # The shared secret key used for encrypting & decrypting data in the Harmony message SHARED_SECRET_KEY=_THIS_IS_MY_32_CHARS_SECRET_KEY_ + +# Set to 'true' if running Docker in Docker and the docker daemon is somewhere other than the current context +DIND=true + +# Indicates where docker commands should find the docker daemon +DOCKER_DAEMON_ADDR=host.docker.internal:2375 \ No newline at end of file From a6e75055801ad179d8d3c55ee72b6db65987f3cc Mon Sep 17 00:00:00 2001 From: vinny Date: Wed, 2 Jun 2021 10:24:26 -0400 Subject: [PATCH 4/6] HARMONY-388: Use consistent markdown formatting --- README.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index d94b93f..0cf1435 100644 --- a/README.md +++ b/README.md @@ -44,11 +44,14 @@ Install project dependencies: ### Development with Docker -If you'd rather not build the image locally (as instructed below), -you can simply pull the latest: `docker pull harmonyservices/netcdf-to-zarr` +If you'd rather not build the image locally (as instructed below), you can simply pull the latest image: + + $ docker pull harmonyservices/netcdf-to-zarr Some of the [Makefile](./Makefile) targets referenced below include an optional argument that allows us to use a local copy of -harmony-service-lib-py (which is useful for concurrent development): `$ make target-name LOCAL_SVCLIB_DIR=../harmony-service-lib-py` +`harmony-service-lib-py` (which is useful for concurrent development): + + $ make target-name LOCAL_SVCLIB_DIR=../harmony-service-lib-py #### Testing & Running the Service Independently From 6312fde346b2d43659bacee16c7b50d190a21930 Mon Sep 17 00:00:00 2001 From: vinny Date: Wed, 2 Jun 2021 14:59:24 -0400 Subject: [PATCH 5/6] HARMONY-388: Move ignore lines to consistent place (customizations toward bottom) --- .gitignore | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/.gitignore b/.gitignore index ab71e30..03a877e 100644 --- a/.gitignore +++ b/.gitignore @@ -1,12 +1,3 @@ -deps - -# Temporary output directories -tmp* -.DS_Store -tmp -config-*.json -config.json - # Byte-compiled / optimized / DLL files __pycache__/ *.py[cod] @@ -113,7 +104,6 @@ venv/ ENV/ env.bak/ venv.bak/ -harmony-ntz # Spyder project settings .spyderproject @@ -133,5 +123,13 @@ dmypy.json # Pyre type checker .pyre/ +# Temporary output directories +tmp* +.DS_Store +tmp +config-*.json +config.json +deps + # Pycharm IDE .idea From 21d4163c750dd1002e70df3d2d23b37e37beb946 Mon Sep 17 00:00:00 2001 From: vinny Date: Mon, 7 Jun 2021 10:50:00 -0400 Subject: [PATCH 6/6] HARMONY-388: Remove ENVHELP.md --- README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/README.md b/README.md index 0cf1435..45d0bfc 100644 --- a/README.md +++ b/README.md @@ -35,8 +35,7 @@ and update the `.env` with the correct values. #### Python & Project Dependencies (Optional) -If you would like to do local development outside of Docker, install Python (3.7.4), and create a -[Python virtual environment](https://github.com/nasa/harmony-service-example/blob/main/ENVHELP.md). +If you would like to do local development outside of Docker, install Python (3.7.4), and create a Python virtual environment. Install project dependencies: