Compiling Python packages with LPython

[czgdp1807]

Following is the plan to achieve the goal mentioned in issue title,

1. Write down small type annotated test packages in Python syntax. The nesting can vary from just 1 folder/package having an __init__.py file to 3-4 folders/subpackages one inside the other each level having an __init__.py. We can write packages for different sorting algorithms, graph algorithms and some practical stuff like that. We should try not to use workarounds but write code in a way that feels natural to us.
2. Try to compile those test packages starting from easy ones (with just 1 folder) to difficult ones. Do sprints to quickly make them work then send PRs each having a single fix.
3. Then proceed with small packages from PyPI (there must be many). Try to compile and fix LPython as described in step 2.
4. Once all of the above is done, move on to advanced Python packages.

Alongside we should also try to interface with CPython code (#703).

@certik What do you say?

[certik]

Yes, we should get LPython working with actual Python packages, that we write ourselves to ensure we stick to the subset that LPython supports. And the goal here is to fix up the inevitable bugs that we either know about or that we will hit. After most things work, it will allow people to upload packages to PyPi and to have dependencies, etc. We can use regular Python tooling to install dependencies and download/upload from PyPi. We have to improve LPython to be able to compile such a package with dependencies.

We probably also need to add a mode to compile the library and all dependencies to some kind of a "mod" file that stores the ASR in it, for quick compilation, so that when you modify a script, only that script has to be parsed and compiled to ASR, all other modules can be just deserialized. LFortran works this way, and it works great.

Finally, we should then start to write libraries of reusable Python code:

• NumPy (we'll ship with LPython)
• All the packages from the Python standard library (we'll ship with LPython)
• Matplotlib
• SciPy

Effectively we need "lite" versions of such libraries, written in pure LPython. We might later venture into web servers too (Flask, etc.), but I would start with numerical / scientific libraries.

We can start creating packages in the Python standard library as external 3rd party packages, and then we'll simply download them / pull them into the LPython distribution, since we need to ship with them by default. The same with NumPy.

Ideall LPython itself (AST->ASR) has support for all the features that are needed, including a small subset of NumPy, such that all of NumPy and all of the Python Standard Library can be implemented as a regular LPython package, no magic / help from the compiler. Equivalent to user code.

[czgdp1807]

I think we can re-use the serialization-deserialisation mechanism of LFortran and use it here to compile Python modules till ASR level. In fact, modules written by us right now compile (parsed + AST->ASR) only when we import them. However LFortran pre-compiles them till ASR and loads them if we import in our program. LPython should behave in a similar manner I think. Also, the question is how to integrate LPython with a build system (writing build files in simple words) so that LPython compiles all the intrinsic modules defined in it when its being built from source.

[certik]

Yes. I think CPython compiles every module into .pyc, so LPython would compile every module to .lpy (or even to .pyc to reuse the extension), and the compiled module would be just serialized ASR. Just like LFortran works with the .mod files.

[czgdp1807]

I see. Let me try implementing this idea then. Let's see how far we can get.

[czgdp1807]

In order to achieve first and second point in #992 (comment), I think we need to write CMakeLists.txt for the small test package. For now we can use add_custom_command to generate LLVM object files and then link those object files together. So at the end for a single sub-package a static/shared binary will be generated. Does this make sense?

[certik]

For end users the main mode that we want to support is automatic compilation, just like CPython works.

Is the CMake based mode just for testing that we can compile things by hand.

[czgdp1807]

I think LPython won't be detected as a compiler by a CMake, right? So add_custom_command will help in compiling those .py files to .o. Basically I am saying that pipeline should be same as C++/C projects because we are generating binaries for Python files same as what Clang/GCC do for C/C++ files.

For end users the main mode that we want to support is automatic compilation, just like CPython works.

By automatic compilation you mean that a module will be compiled to pyc file only when its imported? AFAIK, CPython creates pyc files only when we run the file importing those modules. I am not sure about this though.

[certik]

We will probably support both modes:

• use cmake
• not use cmake, just do lpython a.py and if a.py imports a package b, the package b gets automatically compiled (and we can support some command line flag to compile just the package b into a list of .pyc files.)

I think most users would prefer the second approach.

Or to rephrase it -- the cmake build system can be automatically generated by LPython, so why bother and not just compile things directly? We know all the information.

[czgdp1807]

I see. Makes sense. Second approach would be much easier. We will have to implement timestamp strategy to check whether the file has been changed or not. Do you know how to fetch the timestamp the last time a file was modified on disk a.k.a modification time? https://stackoverflow.com/a/40504396 seems to be a promising way to do get the modification time of a file but if you know of anything better then we can go for that as well.

[czgdp1807]

So I am planning to implement the strategy in my above comment. However the problem is that when you auto-compile and re-use the auto-compiled modules then symbol table IDs depend on the order in which you compile those modules. This would have consequences on python run_tests.py -u since it doesn't follow any particular order of compilation. Now say test a.py and b.py use a module m.py then if a.py gets compiled first then m.py will be different b.py and hence ASR output of b.py will not match. Something similar will happen if b.py gets compiled first. So how to deal with this situation? Because once a module is compiled then it changes the Symbol Table IDs of the rest of the program as well.