pipenv install mixes up system and project packages

[geoffroy-noel-ddh]

version: develop, 0.5

Issue happened on three different bare-bones vagrant instances on @ginestra's machine. From @jmiguelv, 27 Feb 2019:

issue with pipenv on a new bare bones project, that i have not been able to reproduce, but she has reproduced it in 3 different machines. it goes like this:

1. machine is provisioned
2. python env is apparently working properly
3. upgrade or install a new package with pipenv
4. after that, system and local packages get mixed up
5. and trying to run django throws import errors

recreating the virtual env fixes the issue

Additional info: pipenv not run as sudo

[geoffroy-noel-ddh]

If that happens again, could you:

• run pipenv --venv from the same place & user you run pipenv install
• explain point 4: how the system and project packages get mixed up exactly
• point 5: what's the exact error message?
• any way to reproduce that systematically (i.e. link to repo and commit number, actual commands in the terminal and error messages would be very useful)

[jmiguelv]

The issue relates to this:

PIPENV_IGNORE_VIRTUALENVS

If set, Pipenv will always assign a virtual environment for this project.
By default, Pipenv tries to detect whether it is run inside a virtual environment, and reuses it if possible. This is usually the desired behavior, and enables the user to use any user-built environments with Pipenv.

For some reason in @ginestra's environment the default behaviour does not work as expected.

[ginestra]

Error reproduced on a different machine when trying to upgrade django-kdl-wagtail. Looking at the long list for pip freeze, system and local packages are possibly mixed up again. Stack trace below:

(vagrant) ➜  /vagrant git:(develop) pipenv install
Installing dependencies from Pipfile.lock (d35566)…
An error occurred while installing redis==3.2.0 --hash=sha256:724932360d48e5407e8f82e405ab3650a36ed02c7e460d1e6fddf0f038422b54 --hash=sha256:9b19425a38fd074eb5795ff2b0d9a55b46a44f91f5347995f27e3ad257a7d775! Will try again.
An error occurred while installing django-kdl-wagtail==0.6.8 --hash=sha256:2841887b582b1ea64a2ac9164d7e435f9e2730547b3262dba28f56c1f2b9d130 --hash=sha256:482e83607cbdbe66b61c092a7694483c16b4fd96161dad88625024e6465e9f90! Will try again.
An error occurred while installing django==2.1.7 --hash=sha256:275bec66fd2588dd517ada59b8bfb23d4a9abc5a362349139ddda3c7ff6f5ade --hash=sha256:939652e9d34d7d53d74d5d8ef82a19e5f8bb2de75618f7e5360691b6e9667963! Will try again.
An error occurred while installing django-extensions==2.1.6 --hash=sha256:109004f80b6f45ad1f56addaa59debca91d94aa0dc1cb19678b9364b4fe9b6f4 --hash=sha256:307766e5e6c1caffe76c5d99239d8115d14ae3f7cab2cd991fcffd763dad904b! Will try again.
An error occurred while installing cryptography==2.6 --hash=sha256:067649cb9a23cda9e1accd948e044ae6194160586d4a715c6833af09af278731 --hash=sha256:216ec8319576623d1bb16e2dbe9e52afb49ea4e5f10c78f90253a65cc09ee833 --hash=sha256:23e0cb0b3a76e4093791568b2f5097b71b8c2a88e040d000d185421a51ec0a8c --hash=sha256:41d3d15be7f297275e8b476cbd132c9d6701c5206e6c48c5946ec40ff2abb1bc --hash=sha256:5990496950a45b83304a874fb6e4f9ec12b576b22307d9c6213f71c7b103eec8 --hash=sha256:6ee49acf009aa1b44d10a1498ca9341c3a1a9f4d72aff2b5967436eaa9c056b0 --hash=sha256:78f680a50c7bd2a95998cfcfd33b83d7f51c9e6814a71bf18fab021e4022c98b --hash=sha256:7b0dff05319df4ac66b6acaed1a878096dc3778cae7a767f365c4363062b6b23 --hash=sha256:9abcea0f0036a6559e15ed5b46640ea9620536d8b2c7a8a0e0c1397723780b1c --hash=sha256:c9aca85672608f7bfeac8d824e3c2c357620817b56c17f7518a9b626ec3f0ef1 --hash=sha256:f35a030eddea954738f41e4464cde5b5f25a3cdfd8fd5ab945450ff5e41981aa! Will try again.
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 49/49 — 00:14:28
Installing initially failed dependencies…
  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 5/5 — 00:00:39
(vagrant) ➜  /vagrant git:(develop) ./manage.py migrate
Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/vagrant/.venv/lib/python3.5/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/vagrant/.venv/lib/python3.5/site-packages/django/core/management/__init__.py", line 357, in execute
    django.setup()
  File "/vagrant/.venv/lib/python3.5/site-packages/django/__init__.py", line 16, in setup
    from django.urls import set_script_prefix
  File "/vagrant/.venv/lib/python3.5/site-packages/django/urls/__init__.py", line 1, in <module>
    from .base import (
  File "/vagrant/.venv/lib/python3.5/site-packages/django/urls/base.py", line 8, in <module>
    from .exceptions import NoReverseMatch, Resolver404
  File "/vagrant/.venv/lib/python3.5/site-packages/django/urls/exceptions.py", line 1, in <module>
    from django.http import Http404
  File "/vagrant/.venv/lib/python3.5/site-packages/django/http/__init__.py", line 5, in <module>
    from django.http.response import (
  File "/vagrant/.venv/lib/python3.5/site-packages/django/http/response.py", line 15, in <module>
    from django.core.serializers.json import DjangoJSONEncoder
  File "/vagrant/.venv/lib/python3.5/site-packages/django/core/serializers/__init__.py", line 23, in <module>
    from django.core.serializers.base import SerializerDoesNotExist
  File "/vagrant/.venv/lib/python3.5/site-packages/django/core/serializers/base.py", line 6, in <module>
    from django.db import models
  File "/vagrant/.venv/lib/python3.5/site-packages/django/db/models/__init__.py", line 3, in <module>
    from django.db.models.aggregates import *  # NOQA
  File "/vagrant/.venv/lib/python3.5/site-packages/django/db/models/aggregates.py", line 5, in <module>
    from django.db.models.expressions import Case, Func, Star, When
  File "/vagrant/.venv/lib/python3.5/site-packages/django/db/models/expressions.py", line 7, in <module>
    from django.db.models import fields
  File "/vagrant/.venv/lib/python3.5/site-packages/django/db/models/fields/__init__.py", line 11, in <module>
    from django import forms
  File "/vagrant/.venv/lib/python3.5/site-packages/django/forms/__init__.py", line 6, in <module>
    from django.forms.boundfield import *  # NOQA
  File "/vagrant/.venv/lib/python3.5/site-packages/django/forms/boundfield.py", line 4, in <module>
    from django.forms.widgets import Textarea, TextInput
  File "/vagrant/.venv/lib/python3.5/site-packages/django/forms/widgets.py", line 13, in <module>
    from django.templatetags.static import static
ImportError: No module named 'django.templatetags.static'
(vagrant) ➜  /vagrant git:(develop) pip freeze
asn1crypto==0.24.0
backcall==0.1.0
bcrypt==3.1.6
beautifulsoup4==4.6.0
certifi==2018.11.29
cffi==1.12.2
chardet==3.0.4
cryptography==2.6
decorator==4.3.2
Django==2.1.7
django-activecollab-digger==0.7.2
django-appconf==1.0.2
django-auth-ldap==1.7.0
django-cogwheels==0.2
django-compressor==2.2
django-debug-toolbar==1.11
django-extensions==2.1.6
django-haystack==2.8.1
django-kdl-ldap==1.1
django-kdl-wagtail==0.6.8
django-libsass==0.7
django-model-utils==3.1.2
django-modelcluster==4.3
django-redis==4.10.0
django-taggit==0.23.0
django-treebeard==4.3
djangorestframework==3.9.1
draftjs-exporter==2.1.5
entrypoints==0.3
Fabric3==1.14.post1
filelock==3.0.10
flake8==3.7.5
html5lib==1.0.1
idna==2.8
ipython==7.2.0
ipython-genutils==0.2.0
jedi==0.13.2
libsass==0.17.0
mccabe==0.6.1
paramiko==2.4.2
parso==0.3.3
pexpect==4.6.0
pickleshare==0.7.5
Pillow==5.4.1
pipenv==2018.11.26
pluggy==0.8.1
prompt-toolkit==2.0.8
psycopg2-binary==2.7.7
ptyprocess==0.6.0
py==1.7.0
pyasn1==0.4.5
pyasn1-modules==0.2.4
pycodestyle==2.5.0
pycparser==2.19
pyflakes==2.1.0
Pygments==2.3.1
PyNaCl==1.3.0
pyOpenSSL==19.0.0
pysolr==3.8.1
python-ldap==3.1.0
pytz==2018.9
rcssmin==1.0.6
redis==3.2.0
requests==2.21.0
rjsmin==1.0.12
simplejson==3.16.0
six==1.12.0
sqlparse==0.2.4
toml==0.10.0
tox==3.7.0
tox-pipenv==1.9.0
traitlets==4.3.2
Unidecode==1.0.23
urllib3==1.24.1
virtualenv==16.3.0
virtualenv-clone==0.5.1
wagtail==2.4
wagtail-condensedinlinepanel==0.5.2
wagtailmenus==2.12
wcwidth==0.1.7
webencodings==0.5.1
Willow==1.1
(vagrant) ➜  /vagrant git:(develop) pipenv --venv 
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
/vagrant/.venv
(vagrant) ➜  /vagrant git:(develop) pipenv install --clear
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Installing dependencies from Pipfile.lock (d35566)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 49/49 — 00:05:33

[geoffroy-noel-ddh]

I'm not sure system and project packages are mixed up... pip freeze on my projects is just as long due to all dependencies. If you want to check you can try these two commands:

(vagrant) ➜ /vagrant git:(develop) ✗ deactivate
➜ /vagrant git:(develop) ✗ pip freeze

Now you should get a short list without any project-related packages.

To return to your environment, just exit then vagrant ssh again.

[ginestra]

This morning, after it was fixed (venv recreated) the list was much shorter.

[geoffroy-noel-ddh]

I can't make sense of the error message yet. But in the error stack you can see all the paths start with /vagrant/.venv/ so the problem seems internal to your vritual env. Which project is it?

[geoffroy-noel-ddh]

In case you can still reproduce the error. I'd be interested to see what happens when you try to import the module manually:

(vagrant) ➜  /vagrant git:(develop) ✗ python      
Python 3.5.2 (default, Nov 12 2018, 13:43:14) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import django.templatetags.static as st
>>> st
<module 'django.templatetags.static' from '/vagrant/.venv/lib/python3.5/site-packages/django/templatetags/static.py'>
>>> from django.templatetags.static import static
>>> exit()

The weird thing is that the missing module is part of Django 2.1.7 which is clearly part of the packages listed by your pip freeze. I have installed exactly the same packages before running the above code.

Then try:

ls -lth /vagrant/.venv/lib/python3.5/site-packages/django/templatetags/static.py

[ginestra]

To add to the story: on the machine where everything seemed to be okay after recreating the venv, none of the [dev-packages] were installed. I only found out now, after trying to deploy with Fabric.

[geoffroy-noel-ddh]

That's just as mysterious... presumably the original Pipfile from the repo had fabric in the dev-packages section. Apart from Pipfile being removed or Fabric being uninstalled, I can't see how it would disappear from dev-packages.

[ginestra]

Sorry @geoffroy-noel-ddh I missed a bit while typing, I meant that none of the packages under dev-packages were installed (edited above), it's like that bit was ignored on that machine. Also, when I installed them manually, they are added to the packages list, as I assume a flag is needed to get them added under the dev-packages section?

[ginestra]

Just for info: pipenv update manages to update packages where pipenv install was failing.

[geoffroy-noel-ddh]

Thanks, good to know. Just be aware that current (buggy) version of "pipenv update PACKAGE" tries to upgrade all packages!

pypa/pipenv#2412

[brian-maher]

Fixed in 0.5.1. Was caused by a virtualbox shared folder issue.

[geoffroy-noel-ddh]

Technical note so we remember the decision to return to a set up where the venv is stored on the VM only, outside the shared folder:
VBox seems to have file system syncing bugs under some special circumstances (but can be reproduced systematically). pip install / uninstall was crashing because some temporary folders could not be deleted from within Vagrant despite being empty. The reason is that they were not empty in the host filesystem. (@brian-maher I can't find the VBox ticket about this bug any longer...)

[brian-maher]

https://www.virtualbox.org/ticket/8761?cversion=0&cnum_hist=2