Add support for jupyter notebook #414

mateuslatrova · 2023-11-07T23:09:04Z

This branch is a continuation of PR #404. Copied it to the pipreqsxp fork so that the other members can also contribute.

codecov-commenter · 2023-11-23T15:11:57Z

Codecov Report

Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (f041de9) 90.07% compared to head (8ae4618) 90.16%.

Files	Patch %	Lines
pipreqs/pipreqs.py	92.68%	3 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             next     #414      +/-   ##
==========================================
+ Coverage   90.07%   90.16%   +0.09%     
==========================================
  Files           2        2              
  Lines         262      295      +33     
==========================================
+ Hits          236      266      +30     
- Misses         26       29       +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

mateuslatrova · 2023-11-27T20:22:53Z

@alan-barzilay

We rebased this branch into the next branch to add the poetry changes.

After that, we added a flag --scan-notebooks to give the ability for the user to include jupyter notebooks in the files scanned by pipreqs.

Also, we also did a change in the tests to suppress all the logged warnings and errors during test execution.

At last, we updated the README file explaining how the user can install pipreqs without the dependencies needed for jupyter notebook support.

pipreqs/pipreqs.py

alan-barzilay

nice work, but too much cross contamination in commits. Also, tests are failing, where you expecting them to pass?

pipreqs/pipreqs.py

pyproject.toml

tests/test_pipreqs.py

alan-barzilay · 2023-11-29T13:53:06Z

tests/test_pipreqs.py

+        """
+        Test the function ipynb_2_py() which converts .ipynb file to .py format
+        """
+        expected = pipreqs.get_all_imports(self.compatible_files["original"])


what is self.compatible_files["original"]? a list of expected imports? if so, this is not a good intuitive name

Not exactly. self.compatible_files is a dictionary containing two keywords: original and notebook. The items of the dictionary contain paths to files that have the same imports, but one has the .py extension and other has the .ipynb extension.

This test basically converts both files and check if their imports are the same.

tests/test_pipreqs.py

Fernando-crz · 2023-12-02T14:55:13Z

@alan-barzilay @mateuslatrova Just rebased all the changes, fixing possible cross-contamination errors. Please, feel free to check my changes to make sure everything is ok. Also fixed other changes asked by Alan.

Now, we only need to check if ipython is really a necessary import or not. Shouldn't be that hard to check that.

Fernando-crz · 2023-12-02T15:18:32Z

Actually, considering the coverage error was fixed by adding the installation of ipython as a dependency, we can assume that ipython is indeed necessary.

alan-barzilay · 2023-12-02T19:08:26Z

Actually, considering the coverage error was fixed by adding the installation of ipython as a dependency, we can assume that ipython is indeed necessary.

just for the sake of completion, our working hypothesis is that ipython is necessary to parse magic commands in notebooks, our tests were failing due to a parsing error involving magics.

alan-barzilay

almost done! the next version will probably be the final one!

tests/test_pipreqs.py

pipreqs/pipreqs.py

alan-barzilay · 2023-12-02T19:34:03Z

pipreqs/pipreqs.py

    walk = os.walk(path, followlinks=follow_links)
    for root, dirs, files in walk:
        dirs[:] = [d for d in dirs if d not in ignore_dirs]

        candidates.append(os.path.basename(root))
-        files = [fn for fn in files if os.path.splitext(fn)[1] == ".py"]
+        py_files = [file for file in files if file_ext_is_allowed(file, [".py"])]
+        candidates.extend([os.path.splitext(filename)[0] for filename in py_files])


@Fernando-crz @mateuslatrova help me sanity check my understanding of the files and candidates objects, files will be parsed and processed to get the list of imported packages, but this list will be contaminated by stdlib packages and local imports which will be removed in other processing steps (lines 165 and 171).

The candidates list is simply a list of local files that could be imported (like an utils.py file) but shouldnt be a part of the requirements file. Is that it? If so, isnt candidates an awful name? what are they candidates of? also, this would explain why we are adding dir by dir to the list, as I raised in issue #424 and on our last meeting.

Maybe we could take this opportunity to rename those objects to something more intuitive? like local_files or local_modules since dirs can also be imported.
If you think this should be in another pr, ok

maybe
py_files -> local_files
candidates -> local_modules
(pretty sure thts the correct nomenclature but we could use a double check)

Can we just leave it for another PR? 🫥
We are not completely sure what is going on with those variables. We just copied the logic as it was before.

@Fernando-crz my dude, you gotta understand whats going on and not just blindly reuse code haha
i opened issue #427 to keep track of this

tests/test_pipreqs.py

@mateuslatrova

Credits to @mateuslatrova for the contribution.

@pakio

Credits to @pakio and @mateuslatrova for the contributions

alan-barzilay

This was an important and long coming PR, thank you all for the work! I' m finally comfortable with merging this

alan-barzilay · 2023-12-05T18:13:19Z

pipreqs/pipreqs.py

    walk = os.walk(path, followlinks=follow_links)
    for root, dirs, files in walk:
        dirs[:] = [d for d in dirs if d not in ignore_dirs]

        candidates.append(os.path.basename(root))
-        files = [fn for fn in files if os.path.splitext(fn)[1] == ".py"]
+        py_files = [file for file in files if file_ext_is_allowed(file, [".py"])]
+        candidates.extend([os.path.splitext(filename)[0] for filename in py_files])


@Fernando-crz my dude, you gotta understand whats going on and not just blindly reuse code haha
i opened issue #427 to keep track of this

alan-barzilay mentioned this pull request Nov 8, 2023

Add support for jupyter notebook #404

Closed

mateuslatrova force-pushed the add_notebook_support branch from e37ba5e to 6b0d760 Compare November 8, 2023 18:47

Fernando-crz force-pushed the add_notebook_support branch from 6b0d760 to ed82776 Compare November 22, 2023 19:32

mateuslatrova force-pushed the add_notebook_support branch from ed82776 to 3b16e0b Compare November 23, 2023 15:09

mateuslatrova force-pushed the add_notebook_support branch from 3b16e0b to 932deea Compare November 27, 2023 20:15

mateuslatrova force-pushed the add_notebook_support branch 2 times, most recently from 0f28659 to 25b1722 Compare November 27, 2023 20:32

alan-barzilay reviewed Nov 29, 2023

View reviewed changes

pipreqs/pipreqs.py Show resolved Hide resolved

alan-barzilay requested changes Nov 29, 2023

View reviewed changes

Fernando-crz force-pushed the add_notebook_support branch 2 times, most recently from 201e6c0 to ef26256 Compare November 29, 2023 18:44

define pipreqs default encoding to utf-8

db92ddf

Fernando-crz force-pushed the add_notebook_support branch 2 times, most recently from 3fc7d66 to a0a44c1 Compare December 2, 2023 14:23

alan-barzilay requested changes Dec 2, 2023

View reviewed changes

Fernando-crz force-pushed the add_notebook_support branch from a0a44c1 to 61bc4a0 Compare December 4, 2023 19:28

suppress errors and warnings in unit tests

97a0f5e

Credits to @mateuslatrova for the contribution.

Fernando-crz force-pushed the add_notebook_support branch from 61bc4a0 to 97a0f5e Compare December 4, 2023 19:30

Add support for jupyter notebooks

8ae4618

Credits to @pakio and @mateuslatrova for the contributions

alan-barzilay approved these changes Dec 5, 2023

View reviewed changes

alan-barzilay merged commit b50b4a7 into bndr:next Dec 5, 2023
8 checks passed

alan-barzilay mentioned this pull request Dec 5, 2023

Jupyter Notebooks support #51

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for jupyter notebook #414

Add support for jupyter notebook #414

mateuslatrova commented Nov 7, 2023

codecov-commenter commented Nov 23, 2023 •

edited

Loading

mateuslatrova commented Nov 27, 2023

alan-barzilay left a comment

alan-barzilay Nov 29, 2023

Fernando-crz Dec 2, 2023

Fernando-crz commented Dec 2, 2023 •

edited

Loading

Fernando-crz commented Dec 2, 2023

alan-barzilay commented Dec 2, 2023

alan-barzilay left a comment

alan-barzilay Dec 2, 2023

alan-barzilay Dec 2, 2023

Fernando-crz Dec 4, 2023

alan-barzilay Dec 5, 2023

alan-barzilay left a comment

alan-barzilay Dec 5, 2023

Add support for jupyter notebook #414

Add support for jupyter notebook #414

Conversation

mateuslatrova commented Nov 7, 2023

codecov-commenter commented Nov 23, 2023 • edited Loading

Codecov Report

mateuslatrova commented Nov 27, 2023

alan-barzilay left a comment

Choose a reason for hiding this comment

alan-barzilay Nov 29, 2023

Choose a reason for hiding this comment

Fernando-crz Dec 2, 2023

Choose a reason for hiding this comment

Fernando-crz commented Dec 2, 2023 • edited Loading

Fernando-crz commented Dec 2, 2023

alan-barzilay commented Dec 2, 2023

alan-barzilay left a comment

Choose a reason for hiding this comment

alan-barzilay Dec 2, 2023

Choose a reason for hiding this comment

alan-barzilay Dec 2, 2023

Choose a reason for hiding this comment

Fernando-crz Dec 4, 2023

Choose a reason for hiding this comment

alan-barzilay Dec 5, 2023

Choose a reason for hiding this comment

alan-barzilay left a comment

Choose a reason for hiding this comment

alan-barzilay Dec 5, 2023

Choose a reason for hiding this comment

codecov-commenter commented Nov 23, 2023 •

edited

Loading

Fernando-crz commented Dec 2, 2023 •

edited

Loading