Python is an interpreted language; When a program is run, the python interpreter is first parsing your code and checking for any syntax errors, then it is translating the source code into a series of bytecode instructions; these bytecode instructions are then run by the python interpreter. This text is explaining some of the features of the python bytecode.
This article looks at cpython, which is the reference implementation of the python interpreter. Most people are probably running cpython (that's what you get with
brew install python3 on the Mac). If you are talking about the latest and greatest versions of python, then use of the CPython interpreter is implicitly assumed. However there is a drawback, CPython doesn't have a just in time compiler right now, instead the interpreter is running the bytecode instructions directly.
There is a competing python runtime PyPy. This one does have a just in time compiler that translates the bytecode into machine code on the fly, while running your programm. PyPy is therefore several times faster than CPython in most benchmark tests. However it is normally playing catch up with CPython, and is normally a few minor versions behind the CPython release cycle.
This article is focusing on CPython.
The byte code deals with two entities, a memory store that keeps functions and data items, and a stack used for evaluating expression (the stack is maintained separately per each function object) The python interpreter works as a stack machine when it evaluates the bytecode instructions. This means that values are moved from a main memory store to the stack, where the expression is evaluated, then the result is moved back to the main memory store.
The purpose of some the bytecode instructions is to populate the stack, some example instructions:
- THE LOAD_CONST instructions takes a constant and pushes its value to the stack.
- The LOAD_FAST instruction takes a variable and pushes a reference to this variable to the stack
Other bytecode instructions serve the purpose of evaluating expressions. These instructions pop one or several values of a stack, perform some operation on the obtained values (like adding them) and push the result back to the stack. Some example instructions:
- BINARY_ADD pops two values off the stack, and pushes the sum of these values back to the stack.
- UNARY_NEGATE pops the top entry from the stack, and pushes the negated numeric value back to it.
There are instructions used to implement control flow
- JUMP_ABSOLUTE transfers control to a given bytecode instruction
- JUMP_IF_TRUE_OR_POP conditional jump if top of stack has True value, in this case the top element of the stack is left unchanged, if the top of the stack is False then pop the value off the stack.
A very important bytecode sequence is the function call sequence.
-
Here the lowest position on the stack must be a function object, this is put onto the stack by the LOAD_GLOBAL opcode,
-
Next on the stack come the arguments of the function call
-
The next instruction is a function call opcode CALL_FUNCTION; This opcode comes with a parameter that specifies the number of parameters / number of stack entries that will be passed to the function call; these parameters will be poped off the stack, the return value of the function call will be pushed onto the stack.
Here is a reference of the instructions, as part of the dis module from the python standard library. I was suprised to learn, that many bytecode instructions changed in minor releases of the runtime! If you are upgrading or downgrading the python interpreter, then you probably should also delete all __pycache__ folders, these folders hold the binary files that hold the compiled bytecode instructions, but you can't be sure that these will work after a version change!
You can examine the pyhon bytecode of a function by means of a dissassembler, as part of the python standard library you have the dis package, that can show you the bytecode of a python function.
I have written a disassembler that is producing a combined listing for a given python function, this means that you have a line of the python source code, followed by the python bytecode instructions that this source line translates into; I hope that this combined listing will make it much easier to comprehend the meaning of each lineof code and how it translates into the byte code instructions. I think that this will illustrate the concepts, that were explained in the previous section.
Let's look at an example:
(Note: there is one limitation, the tool can't be used, if running python code compiled from a string by means of the compile and exec built-in functions, here it is impossible to find the source code of a line, while running the program)
We will now learn about the python bytecode, while looking at disassembled example functions
The calc function receives the name of the operation and two argument numbers, it switches on the type of the requested operation and performs the requested arithmetic operation on the other two arguments. This is an example function that is evaluating some expression, and then returning a value back to the caller of the function.
Note some difference betwen the python bytecode and the mnemonics of an assembly language - on most CPU's you have a stack that is shared between function calls performed for a given operating system thread. However the python interpreter works differently, you have a Frame object per function, this frame object has a seperate stack used to evaluating the bytecode of that function. The frame objects are chained together and form a seperate stack ordered by the sequence of calling each function. Note that the python bytecode is therefore closely related to how the cpython interpreter is working.
Why should the python runtime maintain a separate stack, just for the purpose of evaluating an expression? One possible reason would be to simplify generators and asynchronous calls, here you need to switch often between co-routines, and it is easier to do so, if each activation record/frame has its own stack to begin with. You can learn more about the cooperative multitasking features of python in this lesson
Please examine the following example and bytecode listing; you can see the patterns exlained in the previous section: evaluation of an expression is loading the arguments to the evaluating stack, then performing an operation on the values in the stack, the result is moved back to main memory via a store instruction.
Source:
import pyasmtools
def calculator(op, num_one, num_two):
if op == 1:
return num_one + num_two
elif op == 2:
return num_one - num_two
elif op == 3:
return num_one * num_two
elif op == 4:
return num_one / num_two
else:
raise ValueError("Invalid operation")
print( "pyasmtools.prettydis(calculator, show_opcode_as_links=True):", pyasmtools.prettydis(calculator, show_opcode_as_links=True) )
Result:
> File path: /Users/michaelmo/mystuff/pyasmtools/calc.py > > calc.py:3 def calculator(op, num_one, num_two): > > calc.py:4 if op == 1: > > 4 0 LOAD_FAST 0 (op) > 2 LOAD_CONST 1 (1) > 4 COMPARE_OP 2 (==) > 6 POP_JUMP_IF_FALSE 16 > > calc.py:5 return num_one + num_two > > 5 8 LOAD_FAST 1 (num_one) > 10 LOAD_FAST 2 (num_two) > 12 BINARY_ADD > 14 RETURN_VALUE > > calc.py:6 elif op == 2: > > 6 >> 16 LOAD_FAST 0 (op) > 18 LOAD_CONST 2 (2) > 20 COMPARE_OP 2 (==) > 22 POP_JUMP_IF_FALSE 32 > > calc.py:7 return num_one - num_two > > 7 24 LOAD_FAST 1 (num_one) > 26 LOAD_FAST 2 (num_two) > 28 BINARY_SUBTRACT > 30 RETURN_VALUE > > calc.py:8 elif op == 3: > > 8 >> 32 LOAD_FAST 0 (op) > 34 LOAD_CONST 3 (3) > 36 COMPARE_OP 2 (==) > 38 POP_JUMP_IF_FALSE 48 > > calc.py:9 return num_one * num_two > > 9 40 LOAD_FAST 1 (num_one) > 42 LOAD_FAST 2 (num_two) > 44 BINARY_MULTIPLY > 46 RETURN_VALUE > > calc.py:10 elif op == 4: > > 10 >> 48 LOAD_FAST 0 (op) > 50 LOAD_CONST 4 (4) > 52 COMPARE_OP 2 (==) > 54 POP_JUMP_IF_FALSE 64 > > calc.py:11 return num_one / num_two > > 11 56 LOAD_FAST 1 (num_one) > 58 LOAD_FAST 2 (num_two) > 60 BINARY_TRUE_DIVIDE > 62 RETURN_VALUE > > calc.py:13 raise ValueError("Invalid operation") > > 13 >> 64 LOAD_GLOBAL 0 (ValueError) > 66 LOAD_CONST 5 ('Invalid operation') > 68 CALL_FUNCTION 1 > 70 RAISE_VARARGS 1 > 72 LOAD_CONST 0 (None) > 74 RETURN_VALUE > pyasmtools.prettydis(calculator, show_opcode_as_links=True): None
The next example has a recursive function with one positional argument, the function computes the factorial on the argument, recursively. Note that first on the stack is variable of type Function that is to be invoked. Next the function parameter is computed and pushed onto the stack, finally the CALL_FUNCTION opocde is perfoming the call, this instruction also has the number of arguments of the call.
Source:
import pyasmtools
def fac(arg_n=7):
if arg_n == 1:
return arg_n
return arg_n * fac(arg_n - 1)
print( "pyasmtools.prettydis(fac, show_opcode_as_links=True):", pyasmtools.prettydis(fac, show_opcode_as_links=True) )Result:
> File path: /Users/michaelmo/mystuff/pyasmtools/fac_rec.py > > fac_rec.py:3 def fac(arg_n = 7): > > fac_rec.py:4 if arg_n == 1: > > 4 0 LOAD_FAST 0 (arg_n) > 2 LOAD_CONST 1 (1) > 4 COMPARE_OP 2 (==) > 6 POP_JUMP_IF_FALSE 12 > > fac_rec.py:5 return arg_n > > 5 8 LOAD_FAST 0 (arg_n) > 10 RETURN_VALUE > > fac_rec.py:6 return arg_n * fac(arg_n - 1) > > 6 >> 12 LOAD_FAST 0 (arg_n) > 14 LOAD_GLOBAL 0 (fac) > 16 LOAD_FAST 0 (arg_n) > 18 LOAD_CONST 1 (1) > 20 BINARY_SUBTRACT > 22 CALL_FUNCTION 1 > 24 BINARY_MULTIPLY > 26 RETURN_VALUE > pyasmtools.prettydis(fac, show_opcode_as_links=True): None
Source:
import pyasmtools
def fac_iter(arg_n: int) -> int:
res = 1
for cur_n in range(1,arg_n):
res *= cur_n
return res
print( "pyasmtools.prettydis(fac, show_opcode_as_links=True):", pyasmtools.prettydis(fac_iter, show_opcode_as_links=True) )
Result:
> File path: /Users/michaelmo/mystuff/pyasmtools/fac_iter.py > > fac_iter.py:3 def fac_iter(arg_n: int) -> int: > > fac_iter.py:4 res = 1 > > 4 0 LOAD_CONST 1 (1) > 2 STORE_FAST 1 (res) > > fac_iter.py:5 for cur_n in range(1,arg_n): > > 5 4 LOAD_GLOBAL 0 (range) > 6 LOAD_CONST 1 (1) > 8 LOAD_FAST 0 (arg_n) > 10 CALL_FUNCTION 2 > 12 GET_ITER > >> 14 FOR_ITER 12 (to 28) > 16 STORE_FAST 2 (cur_n) > > fac_iter.py:6 res *= cur_n > > 6 18 LOAD_FAST 1 (res) > 20 LOAD_FAST 2 (cur_n) > 22 INPLACE_MULTIPLY > 24 STORE_FAST 1 (res) > 26 JUMP_ABSOLUTE 14 > > fac_iter.py:7 return res > > 7 >> 28 LOAD_FAST 1 (res) > 30 RETURN_VALUE > pyasmtools.prettydis(fac, show_opcode_as_links=True): None
Source:
import pyasmtools
class Hello:
def __init__(self, greeting):
self.greeting = greeting
def show(self):
print(self.greeting)
hello_obj = Hello("hello world")
hello_obj.show()
print( "pyasmtools.prettydis(hello_obj.show, show_opcode_as_links=True):", pyasmtools.prettydis(hello_obj.show, show_opcode_as_links=True) )
Result:
> hello world > File path: /Users/michaelmo/mystuff/pyasmtools/obj_call.py > > obj_call.py:7 def Hello.show(self): > > obj_call.py:8 print(self.greeting) > > 8 0 LOAD_GLOBAL 0 (print) > 2 LOAD_FAST 0 (self) > 4 LOAD_ATTR 1 (greeting) > 6 CALL_FUNCTION 1 > 8 POP_TOP > 10 LOAD_CONST 0 (None) > 12 RETURN_VALUE > pyasmtools.prettydis(hello_obj.show, show_opcode_as_links=True): None
Source:
#!/usr/bin/env python3
import pyasmtools
def compute_historgram(file_name):
with open(file_name,'r') as file:
text = file.read()
histo = {}
for ch in text:
if not ch in histo:
histo[ch] = 1
else:
histo[ch] += 1
for ch in histo.keys():
print("char:", repr(ch), "frequency:", histo[ch])
#compute_historgram(__file__)
print( "pyasmtools.prettydis(compute_historgram, show_opcode_as_links=True):", pyasmtools.prettydis(compute_historgram, show_opcode_as_links=True) )
Result:
> File path: /Users/michaelmo/mystuff/pyasmtools/histo.py > > histo.py:5 def compute_historgram(file_name): > > histo.py:6 with open(file_name,'r') as file: > > 6 0 LOAD_GLOBAL 0 (open) > 2 LOAD_FAST 0 (file_name) > 4 LOAD_CONST 1 ('r') > 6 CALL_FUNCTION 2 > 8 SETUP_WITH 108 (to 118) > 10 STORE_FAST 1 (file) > > histo.py:7 text = file.read() > > 7 12 LOAD_FAST 1 (file) > 14 LOAD_METHOD 1 (read) > 16 CALL_METHOD 0 > 18 STORE_FAST 2 (text) > > histo.py:9 histo = {} > > 9 20 BUILD_MAP 0 > 22 STORE_FAST 3 (histo) > > histo.py:10 for ch in text: > > 10 24 LOAD_FAST 2 (text) > 26 GET_ITER > >> 28 FOR_ITER 38 (to 68) > 30 STORE_FAST 4 (ch) > > histo.py:11 if not ch in histo: > > 11 32 LOAD_FAST 4 (ch) > 34 LOAD_FAST 3 (histo) > 36 CONTAINS_OP 1 > 38 POP_JUMP_IF_FALSE 50 > > histo.py:12 histo[ch] = 1 > > 12 40 LOAD_CONST 2 (1) > 42 LOAD_FAST 3 (histo) > 44 LOAD_FAST 4 (ch) > 46 STORE_SUBSCR > 48 JUMP_ABSOLUTE 28 > > histo.py:14 histo[ch] += 1 > > 14 >> 50 LOAD_FAST 3 (histo) > 52 LOAD_FAST 4 (ch) > 54 DUP_TOP_TWO > 56 BINARY_SUBSCR > 58 LOAD_CONST 2 (1) > 60 INPLACE_ADD > 62 ROT_THREE > 64 STORE_SUBSCR > 66 JUMP_ABSOLUTE 28 > > histo.py:16 for ch in histo.keys(): > > 16 >> 68 LOAD_FAST 3 (histo) > 70 LOAD_METHOD 2 (keys) > 72 CALL_METHOD 0 > 74 GET_ITER > >> 76 FOR_ITER 26 (to 104) > 78 STORE_FAST 4 (ch) > > histo.py:17 print("char:", repr(ch), "frequency:", histo[ch]) > > 17 80 LOAD_GLOBAL 3 (print) > 82 LOAD_CONST 3 ('char:') > 84 LOAD_GLOBAL 4 (repr) > 86 LOAD_FAST 4 (ch) > 88 CALL_FUNCTION 1 > 90 LOAD_CONST 4 ('frequency:') > 92 LOAD_FAST 3 (histo) > 94 LOAD_FAST 4 (ch) > 96 BINARY_SUBSCR > 98 CALL_FUNCTION 4 > 100 POP_TOP > 102 JUMP_ABSOLUTE 76 > >> 104 POP_BLOCK > 106 LOAD_CONST 0 (None) > 108 DUP_TOP > 110 DUP_TOP > 112 CALL_FUNCTION 3 > 114 POP_TOP > 116 JUMP_FORWARD 16 (to 134) > >> 118 WITH_EXCEPT_START > 120 POP_JUMP_IF_TRUE 124 > 122 RERAISE > >> 124 POP_TOP > 126 POP_TOP > 128 POP_TOP > 130 POP_EXCEPT > 132 POP_TOP > >> 134 LOAD_CONST 0 (None) > 136 RETURN_VALUE > pyasmtools.prettydis(compute_historgram, show_opcode_as_links=True): None
Source:
#!/usr/bin/env python3
import random
import pyasmtools
def shuffle(arr_size):
arr=[]
for num in range(1,arr_size+1):
arr.append(num)
for nun in range(0, arr_size):
idx = random.randint(1,arr_size)
tmp = arr[0]
arr[0] = arr[idx]
arr[idx] = tmp
return arr
#print(shuffle(7))
print( "pyasmtools.prettydis(shuffle, show_opcode_as_links=True):", pyasmtools.prettydis(shuffle, show_opcode_as_links=True) )
Result:
> File path: /Users/michaelmo/mystuff/pyasmtools/shuffle.py > > shuffle.py:7 def shuffle(arr_size): > > shuffle.py:8 arr=[] > > 8 0 BUILD_LIST 0 > 2 STORE_FAST 1 (arr) > > shuffle.py:9 for num in range(1,arr_size+1): > > 9 4 LOAD_GLOBAL 0 (range) > 6 LOAD_CONST 1 (1) > 8 LOAD_FAST 0 (arr_size) > 10 LOAD_CONST 1 (1) > 12 BINARY_ADD > 14 CALL_FUNCTION 2 > 16 GET_ITER > >> 18 FOR_ITER 14 (to 34) > 20 STORE_FAST 2 (num) > > shuffle.py:10 arr.append(num) > > 10 22 LOAD_FAST 1 (arr) > 24 LOAD_METHOD 1 (append) > 26 LOAD_FAST 2 (num) > 28 CALL_METHOD 1 > 30 POP_TOP > 32 JUMP_ABSOLUTE 18 > > shuffle.py:12 for nun in range(0, arr_size): > > 12 >> 34 LOAD_GLOBAL 0 (range) > 36 LOAD_CONST 2 (0) > 38 LOAD_FAST 0 (arr_size) > 40 CALL_FUNCTION 2 > 42 GET_ITER > >> 44 FOR_ITER 44 (to 90) > 46 STORE_FAST 3 (nun) > > shuffle.py:13 idx = random.randint(1,arr_size) > > 13 48 LOAD_GLOBAL 2 (random) > 50 LOAD_METHOD 3 (randint) > 52 LOAD_CONST 1 (1) > 54 LOAD_FAST 0 (arr_size) > 56 CALL_METHOD 2 > 58 STORE_FAST 4 (idx) > > shuffle.py:14 tmp = arr[0] > > 14 60 LOAD_FAST 1 (arr) > 62 LOAD_CONST 2 (0) > 64 BINARY_SUBSCR > 66 STORE_FAST 5 (tmp) > > shuffle.py:15 arr[0] = arr[idx] > > 15 68 LOAD_FAST 1 (arr) > 70 LOAD_FAST 4 (idx) > 72 BINARY_SUBSCR > 74 LOAD_FAST 1 (arr) > 76 LOAD_CONST 2 (0) > 78 STORE_SUBSCR > > shuffle.py:16 arr[idx] = tmp > > 16 80 LOAD_FAST 5 (tmp) > 82 LOAD_FAST 1 (arr) > 84 LOAD_FAST 4 (idx) > 86 STORE_SUBSCR > 88 JUMP_ABSOLUTE 44 > > shuffle.py:18 return arr > > 18 >> 90 LOAD_FAST 1 (arr) > 92 RETURN_VALUE > pyasmtools.prettydis(shuffle, show_opcode_as_links=True): None