Skip to content

Commit 1aa9e36

Browse files
h4110w33nh4110w33n
committed
first commit
0 parents  commit 1aa9e36

File tree

1 file changed

+156
-0
lines changed

1 file changed

+156
-0
lines changed

‎README.md

+156
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
Python PEX Demo
2+
===============
3+
This is a demo repo to show how to build a single Python executable that contains a complete virtual environment. This process has helped me maintain an untainted "system" Python environment while being able to ship the latest modules. If you are a developer or sysadmin concerned those sorts of things, then this demo is for you.
4+
5+
What is PEX?
6+
------------
7+
From the overview of the PEX Project's Github:
8+
9+
> pex is a library for generating .pex (Python EXecutable) files which are executable Python environments in the spirit of virtualenvs. pex is an expansion upon the ideas outlined in [PEP 441](https://legacy.python.org/dev/peps/pep-0441/) and makes the deployment of Python applications as simple as cp. pex files may even include multiple platform-specific Python distributions, meaning that a single pex file can be portable across Linux and OS X.
10+
11+
#### Related links:
12+
* [PEX Github](https://github.com/pantsbuild/pex)
13+
* [PEP 441](https://legacy.python.org/dev/peps/pep-0441/)
14+
* [WTF is PEX?](https://www.youtube.com/watch?v=NmpnGhRwsu0)
15+
16+
What is the goal of this project?
17+
---------------------------------
18+
1. Demonstrate how to buidle a generic PEX file for executing arbitrary scripts (i.e. shippable virtualenv).
19+
2. Embed a basic Flask + Gunicorn API into a PEX file and launch it instead of a Python Shell.
20+
21+
0 Checkout this repo
22+
=====================
23+
Checkout this repo to a location of your preference and `cd` to it, and then we can get started. I also encourage initializing a new Python Virtualenv for this demo, but that is optional.
24+
25+
1 Creating a PEX executable
26+
===========================
27+
First thing you are going to need is the `pex` Python module. This can be easily installed via `pip` as shown below.
28+
29+
```sh
30+
pip install pex
31+
```
32+
33+
Now that you have PEX module, lets use is to build a basic Python executable. In this case, I'm going to import just the `numpy` module and save the resulting file to `numpy_example.pex` with the `-o` option.
34+
35+
```sh
36+
pex numpy -o numpy_example.pex
37+
```
38+
39+
If everything goes according to plan, we should now be able to launch the `numpy_example.pex` executable. Upon running the file directly, we should expect a ordinary Python interperter shell as show below.
40+
41+
```sh
42+
./numpy_example.pex
43+
Python 3.7.7 (default, Mar 10 2020, 15:43:33)
44+
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
45+
Type "help", "copyright", "credits" or "license" for more information.
46+
(InteractiveConsole)
47+
>>> import numpy
48+
>>> numpy.zeros((2,3))
49+
array([[0., 0., 0.],
50+
[0., 0., 0.]])
51+
>>>
52+
```
53+
54+
Also, like the normal Python interperter, we can tell it to execute an external script. Below is the result from running the `numpy_test.py` from the examples folder.
55+
56+
```sh
57+
./numpy_example.pex examples/numpy_test.py
58+
[[ 0 1 2 3 4]
59+
[ 5 6 7 8 9]
60+
[10 11 12 13 14]]
61+
```
62+
63+
Hopefully at this point some gears are turning. Could I have a whole mess of modules from a large, complicated project bundled into a single file? Yes! Yes you can, and the `-r` option is how. Below shows how to load this projects `requirements.txt` file in, including `xmltodict` which I exercise later on.
64+
65+
```sh
66+
pex -r requirements.txt -o xmltodict_example.pex
67+
```
68+
69+
Functionality is exactly the same as before, and the executable can either be executed directly or used to execute an aribtrary script.
70+
71+
```sh
72+
./xmltodict_example.pex examples/xmltodict_test.py
73+
OrderedDict([('key', OrderedDict([('subkey1', 'thing1'), ('subkey2', 'thing2')]))])
74+
```
75+
76+
2 Embedding more complicated apps
77+
=================================
78+
In the same vein as Golang's compiled executables, we can use PEX to bundle together modules with a wrapper to deliever a complete single-executable-app. The meat and potatoes of this repo consist of a fairly simple Flask application (called `babble`) that is served via the WSGI server Gunicorn.
79+
80+
Babble's structure is fairly important. All of the necessary machinery to load Gunicorn's config and start it are included in the `babble/__init__.py` file. Specifically the `launcher` function.
81+
82+
Below shows the process of integrating a locally developed module (the wrapper itself) as well as the use of the `-e` option for PEX that sets the entrypoint whenever executed.
83+
84+
```sh
85+
# Verify that all requirements are available
86+
pip install -r requirements.txt
87+
# Install the `babble` module included in this repo so PEX can address it
88+
python setup.py develop
89+
# Build the pex and name it `babble.pex`
90+
pex . -o babble.pex -e babble:launcher
91+
```
92+
93+
This process should yeild PEX file just a before; however, upon execution, we are greeded with the output from Gunicorn.
94+
95+
```
96+
$ ./babble.pex
97+
[2020-06-24 20:56:48 -0400] [22226] [INFO] Starting gunicorn 20.0.4
98+
[2020-06-24 20:56:48 -0400] [22226] [INFO] Listening at: http://127.0.0.1:8080 (22226)
99+
[2020-06-24 20:56:48 -0400] [22226] [INFO] Using worker: sync
100+
[2020-06-24 20:56:48 -0400] [22229] [INFO] Booting worker with pid: 22229
101+
[2020-06-24 20:56:48 -0400] [22230] [INFO] Booting worker with pid: 22230
102+
[2020-06-24 20:56:48 -0400] [22231] [INFO] Booting worker with pid: 22231
103+
```
104+
105+
Lets take at the component that make this tick.
106+
107+
###### babble/__init__.py
108+
This is the entry point used in the last `pex` command. Some special things worth noting:
109+
110+
1. This launcher contains the argument parsing and instantiation of the Gunicorn server. The Flask `app` is imported in the same file so it is ready to go as soon as Gunicorn is available.
111+
2. There is a tendancy to obscure operational parameters like config files or options when wraping up a module like this. Resist the urge. In the example below, the `-c` option was exposed to allow one to specify a Gunicorn config file. Otherwise, we set reasonable defaults.
112+
3. The launcher function could have been put into another file (i.e. not `__init__.py`). Best practice would has us put this in a seperate file. The important part is that it is addressable with the `-e` option.
113+
4. The Flask `app` object is imported from `babble/web_api.py` and the `StandaloneApplication` class is from `babble/web_server.py` just in case you want to review those as well.
114+
115+
```python
116+
def launcher(live_reload=False):
117+
logging.basicConfig(level=logging.DEBUG,
118+
format='%(asctime)s %(name)s %(levelname)s:%(message)s')
119+
120+
arg_parse = argparse.ArgumentParser(description="A super basic Flask + Gunicorn app")
121+
arg_parse.add_argument("-c", "--config-file", dest="config_file",
122+
help="Config File location", default=None)
123+
args = arg_parse.parse_args()
124+
125+
if not args.config_file:
126+
options = {
127+
'bind': '%s:%s' % ('127.0.0.1', '8080'),
128+
'workers': 3,
129+
'reload': live_reload,
130+
}
131+
else:
132+
options = config_importer(args.config_file)
133+
StandaloneApplication(app, options).run()
134+
```
135+
136+
Hopefully between this crash course, plus the supplied code, you'll know a little more about PEX. It might not be the best option for your app delievery needs, but having another arrow in the quiver never hurts.
137+
138+
FAQ
139+
===
140+
#### Q - What is the Flask App discussed in the examples even do?
141+
A - It is a super simple XML to JSON and JSON to XML converter. If you have it running still, you can test it with the following `curl` commands.
142+
143+
```
144+
# XML to JSON
145+
curl -i -H "Content-Type: application/xml" -X POST \
146+
-d '<?xml version="1.0" ?><person><name>john</name><age>20</age></person>' \
147+
http://127.0.0.1:8080/xml_to_json
148+
149+
# JSON to XML
150+
curl -i -H "Content-Type: application/json" -X POST \
151+
-d '{"userId":"1", "username": "fizz bizz"}' \
152+
http://127.0.0.1:8080/json_to_xml
153+
```
154+
155+
#### Q - Do I have to use this repo with PEX?
156+
A - Nope. If you just want to use this as a reference on how to embed a simple Flask app into a Gunicorn server, feel free. I did most of my development for the project in a normal Python environment before I wrote up the PEX portion.

0 commit comments

Comments
 (0)