Hybrid local development

[foarsitter]

Running a single test inside a container gives me a little more overhead then I would like. Therefore I'm using python locally and leveraging docker for all the external services like redis and postgres. I'm calling it hybrid.

I would like to adjust the template for adding hybrid as an option alongside local & docker.

What is your opinion about this approach?

[JamesParrott]

Make a fork and lets have a look at it.

It would need launcher scripts instead of letting docker compose spin everything up. I wouldn't use it, but there's still value for users new to django in making a template for the exact example from the django docs introductory app, but with postgres, nginx and traefik etc. working out of the box, using containers. That would be a perfect opportnuity to trim out some of the hacks in cookiecutter-django too (like the non-standard directory structure, copying everything in the repo into the node container right at the start of the Django dockerfile meaning the docker cache is hardly ever useful, and the python script in a string inside a bash script, that waits for a postgres connection).

But for the core mission of cookiecutter-django, and promise it makes to help: "jump start production-ready Django projects quickly", containerising the django run time has the following advantages:

• Currently local/production.yml is a great reference for what's running in your cookiecutter-django stack.

• Containerising Django allows the env it's in to be easily reproduced for celery, or for the user's custom services that need to use Django.

• Containerising makes the stack more portable and cloud agnostic (unlike running locally, it doesn't necessarily need a server. It could run on a container run time PAAS like fly.io)

• One extra security layer - if there's a vulnerability in Django, attackers are straight into the host OS, not just the container under the non-root user django. I would not assume a container is bullet proof, but it does raise the cost to an attacker.

• The docker network is a little nicer to work with than localhost ports, and avoids having to double check the ports of services running at OS level, against the firewall and network table.

• Easily upgrade or switch Python versions.

Plus others that don't immediately spring to mind.

[foarsitter]

My proposal is to use a local interpreter for development instead of the containerized one. So the production environment wouldn't change.

Your bullets are valid points and should left untouched. The only change I'm proposing is to use a system interpreter so tests are running faster :)

[JamesParrott]

The tests will indeed run faster, which would be nice I agree. Is there a second route to running the tests directly in the repo, or can they be reorganised?

But the subset of tests that can run locally outside a container, will not definitively test any interaction with the container, which is what will actually be deployed. So it could lead to false confidence, or late discovery of bugs.

[foarsitter]

Is there a second route to running the tests directly in the repo, or can they be reorganised?

What are you thinking of?

In my thinking it is straightforward: an interpreter is juist interacting with services. Where those services are living isn't important as long as they available through the network. The docker layer gives me a performance penalty when I'm running tests, so I can better remove it and enjoy an improved DX.

Theoretically this can lead to false confidence or late discovery of bugs, especially when you are developing under windows or osx. But these issues will popup in the pipeline or on the staging/test environment.

[browniebroke]

That's an approach I saw some colleagues use in a previous project, but I never been too bothered with the Docker overhead myself to use it, it's a price I don't mind paying for the benefit of reproducibility. The way we did it was to have both Docker and Hybrid side by side and each developer on project was free to choose the way they wanted. Was useful to keep Docker around in case of tricky bug where an env closer to prod was needed.

Question for you is how do you usually run Celery in this context? Do you start another process on your host machine or do you keep Celery containerized?

[browniebroke]

That would be a perfect opportnuity to trim out some of the hacks in cookiecutter-django too (like the non-standard directory structure, copying everything in the repo into the node container right at the start of the Django dockerfile meaning the docker cache is hardly ever useful, and the python script in a string inside a bash script, that waits for a postgres connection).

@JamesParrott I think we have a few issues tracking some of these:

• Revisiting the project root vs project_slug folder for creating apps #4031
• Applications Folder #3594
• Question about conception #4721
• Why prefer python than bash for wait the service start #4539

However, I don't see what you mean about "copying everything in the repo into the node container right at the start of the Django dockerfile meaning the docker cache is hardly ever useful", could you point us at the problem in a new issue please? It would be good to tackle that here, we don't need a fork 😄