Running integration tests locally

[arrabito]

Dear all,

I'm trying to run integration tests locally, since we would like to do the same for our CTADIRAC extension.

Following this guide:
https://dirac.readthedocs.io/en/latest/DeveloperGuide/CodeTesting/index.html#running-integration-tests-locally

on my MacOS I get the following error:

./integration_tests.py create
...
CalledProcessError: Command '['docker', 'compose', '-f', PosixPath('/var/folders/7j/qdb2nq852973tsy1_v99qzrm0000gq/T/tmp_aqrn3_7/ci/docker-compose.yml'), 'ps', '-q', '-a']' returned non-zero exit status 1.

and I've checked that the file /var/folders/7j/qdb2nq852973tsy1_v99qzrm0000gq/T/tmp_aqrn3_7/ci/docker-compose.yml does not exist .

Any idea how to solve this issue?

Thank you,

Luisa

[fstagni]

What happens if you call directly

docker compose -f docker-compose.yml

where docker-compose.yml is https://github.com/DIRACGrid/DIRAC/blob/integration/tests/CI/docker-compose.yml (you need to edit this with some real value).

[arrabito]

So I've copied docker-compose.yml in the top directory:

cp tests/CI/docker-compose.yml .

and modified :

https://github.com/DIRACGrid/DIRAC/blob/integration/integration_tests.py#L583

to use the docker-compose.yml in the top directory:

output_fn = Path(__file__).parent / "docker-compose.yml"

Then I get:

./integration_tests.py create
...
CalledProcessError: Command '['docker', 'compose', '-f', PosixPath('/Users/arrabito/PycharmProjects/DIRAC/docker-compose.yml'), 'ps', '-q', '-a']' returned non-zero exit status 14.

If I run directly docker compose, I get:

(dirac-dev) DIRAC$ docker compose -f docker-compose.yml ps -q -a
WARN[0000] The "CI_REGISTRY_IMAGE" variable is not set. Defaulting to a blank string. 
WARN[0000] The "HOST_OS" variable is not set. Defaulting to a blank string. 
WARN[0000] The "DIRAC_UID" variable is not set. Defaulting to a blank string. 
WARN[0000] The "DIRAC_GID" variable is not set. Defaulting to a blank string. 
WARN[0000] The "CI_REGISTRY_IMAGE" variable is not set. Defaulting to a blank string. 
WARN[0000] The "HOST_OS" variable is not set. Defaulting to a blank string. 
WARN[0000] The "DIRAC_UID" variable is not set. Defaulting to a blank string. 
WARN[0000] The "DIRAC_GID" variable is not set. Defaulting to a blank string. 
WARN[0000] The "MYSQL_VER" variable is not set. Defaulting to a blank string. 
WARN[0000] The "ES_VER" variable is not set. Defaulting to a blank string. 
WARN[0000] The "ES_VER" variable is not set. Defaulting to a blank string. 
WARN[0000] The "IAM_VER" variable is not set. Defaulting to a blank string. 
WARN[0000] The "IAM_VER" variable is not set. Defaulting to a blank string. 
WARN[0000] The "MYSQL_VER" variable is not set. Defaulting to a blank string. 
yaml: line 11: mapping values are not allowed in this context

[arrabito]

So I thought that with this procedure I would have got docker-compose.yml with real values, but I realize that it's not the case.

So I've just replaced variables with real values and now I get:

(dirac-dev) DIRAC$ docker compose -f docker-compose.yml ps -q -a
open /Users/arrabito/PycharmProjects/DIRAC/opensearchproject/opensearch:2.1.0.env: no such file or directory

[fstagni]

This

open /Users/arrabito/PycharmProjects/DIRAC/opensearchproject/opensearch:2.1.0.env

does not look like a good location. It should be this: https://github.com/DIRACGrid/DIRAC/blob/integration/tests/CI/envs/opensearchproject/opensearch%3A2.1.0.env

[arrabito]

OK I've just changed the docker-compose.yml with:

    env_file: "tests/CI/envs/opensearchproject/opensearch:2.1.0.env"
    env_file: "tests/CI/envs/indigoiam/iam-login-service:v1.8.0.env"

and now I get:

(dirac-dev) DIRAC$ docker compose -f docker-compose.yml ps -q -a
not found

[fstagni]

(dirac-dev) DIRAC$ docker compose -f docker-compose.yml ps -q -a
not found

why ps -q -a ?
and, do you have docker compose installed?

[arrabito]

I've used:

ps -q -a

since this is what is used first in the:

https://github.com/DIRACGrid/DIRAC/blob/integration/integration_tests.py

then in principle yes I have docker compose installed, but maybe the command line is slightly different on MacOS, see:

(dirac-dev) DIRAC$ docker compose -f docker-compose.yml

Usage:  docker compose [OPTIONS] COMMAND

Docker Compose

Options:
      --ansi string                Control when to print ANSI control characters ("never"|"always"|"auto") (default "auto")
      --env-file string            Specify an alternate environment file.
  -f, --file stringArray           Compose configuration files
      --profile stringArray        Specify a profile to enable
      --project-directory string   Specify an alternate working directory
                                   (default: the path of the Compose file)
  -p, --project-name string        Project name

Commands:
  build       Build or rebuild services
  convert     Converts the compose file to platform's canonical format
  cp          Copy files/folders between a service container and the local filesystem
  create      Creates containers for a service.
  down        Stop and remove containers, networks
  events      Receive real time events from containers.
  exec        Execute a command in a running container.
  images      List images used by the created containers
  kill        Force stop service containers.
  logs        View output from containers
  ls          List running compose projects
  pause       pause services
  port        Print the public port for a port binding.
  ps          List containers
  pull        Pull service images
  push        Push service images
  restart     Restart containers
  rm          Removes stopped service containers
  run         Run a one-off command on a service.
  start       Start services
  stop        Stop services
  top         Display the running processes
  unpause     unpause services
  up          Create and start containers

Run 'docker compose COMMAND --help' for more information on a command.

should I use build:

docker compose build -f docker-compose.yml

?

[arrabito]

I've managed to run directly:

docker compose -f docker-compose.yml up

Then, after destroying the containers I'm able to run :

./integration_tests.py prepare-environment

but I had to comment this line:

https://github.com/DIRACGrid/DIRAC/blob/integration/integration_tests.py#L199

which for some reasons gives error when there are no containers running.

I'm still adjusting some values in my docker-compose.yml but it looks to work fine now.

Thank you for your help.

[fstagni]

I think your issues are related to your local docker installation.