Skip to content

Commit 3f5e402

Browse files
committed
[test][bionic] Force python3 and utf-8 encoding
bionic's /usr/bin/python is python2. Encoding and python2/python3 is.... complicated. Those tests already run on py3 which is the default for Ubuntu Focal and Fedora 34. This change symlinks /usr/local/bin/python to python3 within the Bionic container and changes all tools to use "#!/usr/bin/env python" shebang instead of "#!/usr/bin/python" which allows to use the default python set by the environment. To add insult to injury. Python 3.6 still needs a bit of help to default to UTF-8 which can be achieved with the env var `PYTHONIOENCODING` to `utf-8`. Since python 3.7, [UTF-8 mode is enabled by default when the locale is C or POSIX](https://docs.python.org/3/whatsnew/3.7.html#whatsnew37-pep540). Some more details about this problem can be found here: https://vstinner.github.io/posix-locale.html#misconfigured-locales-in-docker-images Illustration: ``` root@bionic:/# locale LANG= LANGUAGE= LC_CTYPE="POSIX" LC_NUMERIC="POSIX" LC_TIME="POSIX" LC_COLLATE="POSIX" LC_MONETARY="POSIX" LC_MESSAGES="POSIX" LC_PAPER="POSIX" LC_NAME="POSIX" LC_ADDRESS="POSIX" LC_TELEPHONE="POSIX" LC_MEASUREMENT="POSIX" LC_IDENTIFICATION="POSIX" LC_ALL= root@bionic:/# python2 Python 2.7.17 (default, Jul 1 2022, 15:56:32) [GCC 7.5.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> print("%s" % u'\xb7') Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeEncodeError: 'ascii' codec can't encode character u'\xb7' in position 0: ordinal not in range(128) >>> root@bionic:/# python3 Python 3.6.9 (default, Jun 29 2022, 11:45:57) [GCC 8.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> print("%s" % u'\xb7') Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeEncodeError: 'ascii' codec can't encode character '\xb7' in position 0: ordinal not in range(128) >>> root@bionic:/# PYTHONIOENCODING=utf-8 python3 Python 3.6.9 (default, Jun 29 2022, 11:45:57) [GCC 8.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> print("%s" % u'\xb7') · ``` On Focal: ``` root@focal:/# locale LANG= LANGUAGE= LC_CTYPE="POSIX" LC_NUMERIC="POSIX" LC_TIME="POSIX" LC_COLLATE="POSIX" LC_MONETARY="POSIX" LC_MESSAGES="POSIX" LC_PAPER="POSIX" LC_NAME="POSIX" LC_ADDRESS="POSIX" LC_TELEPHONE="POSIX" LC_MEASUREMENT="POSIX" LC_IDENTIFICATION="POSIX" LC_ALL= root@focal:/# python3 Python 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> print("%s" % u'\xb7') · ```
1 parent 1a3f8c2 commit 3f5e402

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

98 files changed

+101
-99
lines changed

docker/Dockerfile.tests

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,3 +72,4 @@ RUN wget -O ruby-install-0.7.0.tar.gz \
7272

7373
RUN ruby-install --system ruby 2.6.0 -- --enable-dtrace
7474
RUN if [ ! -f "/usr/bin/python" ]; then ln -s /bin/python3 /usr/bin/python; fi
75+
RUN if [ ! -f "/usr/local/bin/python" ]; then ln -s /usr/bin/python3 /usr/local/bin/python; fi

tests/wrapper.sh.in

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ kind=$1; shift
99
cmd=$1; shift
1010

1111
PYTHONPATH=@CMAKE_BINARY_DIR@/src/python/bcc-python3
12+
PYTHONIOENCODING=utf-8
1213
LD_LIBRARY_PATH=@CMAKE_BINARY_DIR@:@CMAKE_BINARY_DIR@/src/cc
1314

1415
ns=$name
@@ -32,15 +33,15 @@ function ns_run() {
3233
sudo ip netns exec $ns ethtool -K eth0 tx off
3334
sudo ip addr add dev $ns.out 172.16.1.1/24
3435
sudo ip link set $ns.out up
35-
sudo --preserve-env=PYTHON_TEST_LOGFILE env PYTHONPATH=$PYTHONPATH LD_LIBRARY_PATH=$LD_LIBRARY_PATH ip netns exec $ns $cmd "$@"
36+
sudo --preserve-env=PYTHON_TEST_LOGFILE env PYTHONIOENCODING=$PYTHONIOENCODING PYTHONPATH=$PYTHONPATH LD_LIBRARY_PATH=$LD_LIBRARY_PATH ip netns exec $ns $cmd "$@"
3637
return $?
3738
}
3839
function sudo_run() {
39-
sudo --preserve-env=PYTHON_TEST_LOGFILE env PYTHONPATH=$PYTHONPATH LD_LIBRARY_PATH=$LD_LIBRARY_PATH $cmd "$@"
40+
sudo --preserve-env=PYTHON_TEST_LOGFILE env PYTHONIOENCODING=$PYTHONIOENCODING PYTHONPATH=$PYTHONPATH LD_LIBRARY_PATH=$LD_LIBRARY_PATH $cmd "$@"
4041
return $?
4142
}
4243
function simple_run() {
43-
PYTHONPATH=$PYTHONPATH PYTHON_TEST_LOGFILE=$PYTHON_TEST_LOGFILE LD_LIBRARY_PATH=$LD_LIBRARY_PATH $cmd "$@"
44+
PYTHONIOENCODING=$PYTHONIOENCODING PYTHONPATH=$PYTHONPATH PYTHON_TEST_LOGFILE=$PYTHON_TEST_LOGFILE LD_LIBRARY_PATH=$LD_LIBRARY_PATH $cmd "$@"
4445
return $?
4546
}
4647

tools/argdist.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/usr/bin/python
1+
#!/usr/bin/env python
22
#
33
# argdist Trace a function and display a distribution of its
44
# parameter values as a histogram or frequency count.

tools/bashreadline.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/usr/bin/python
1+
#!/usr/bin/env python
22
#
33
# bashreadline Print entered bash commands from all running shells.
44
# For Linux, uses BCC, eBPF. Embedded C.

tools/bindsnoop.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/usr/bin/python
1+
#!/usr/bin/env python
22
#
33
# bindsnoop Trace IPv4 and IPv6 binds()s.
44
# For Linux, uses BCC, eBPF. Embedded C.

tools/biolatency.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/usr/bin/python
1+
#!/usr/bin/env python
22
# @lint-avoid-python-3-compatibility-imports
33
#
44
# biolatency Summarize block device I/O latency as a histogram.

tools/biolatpcts.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/usr/bin/python
1+
#!/usr/bin/env python
22
#
33
# biolatpcts.py Monitor IO latency distribution of a block device.
44
#

tools/biopattern.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/usr/bin/python
1+
#!/usr/bin/env python
22
# @lint-avoid-python-3-compatibility-imports
33
#
44
# biopattern - Identify random/sequential disk access patterns.

tools/biosnoop.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/usr/bin/python
1+
#!/usr/bin/env python
22
# @lint-avoid-python-3-compatibility-imports
33
#
44
# biosnoop Trace block device I/O and print details including issuing PID.

tools/biotop.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/usr/bin/python
1+
#!/usr/bin/env python
22
# @lint-avoid-python-3-compatibility-imports
33
#
44
# biotop block device (disk) I/O by process.

0 commit comments

Comments
 (0)