Skip to content

Commit

Permalink
Release v0.6.0 (#40)
Browse files Browse the repository at this point in the history
* Release v0.6.0

	- Major changes:
	  * Change verificarlo version to compile fuzzy images [v0.5.0 -> v0.6.0]
	  * Change base3 sklearn version [v0.23.2 -> v1.0.2]
	  * Change base2 scipy version [v1.5.4 -> v1.7.3]
	  * Change base1 numpy version [v1.19.1 -> v1.22.0]
	  * Fix illegal instruction due to cross-compiling
	    Can be run on any x86_64 architecture, no matter from where it has been compiled
	- Minor changes:
	  * Remove verificarlo and NumPy patching
	  * Simplify the NumPy and SciPy compilation
	  * Add sanity check for NumPy and SciPy
  • Loading branch information
yohanchatelain authored Jan 11, 2022
1 parent 03d41b0 commit 0cc650f
Show file tree
Hide file tree
Showing 11 changed files with 117 additions and 1,103 deletions.
34 changes: 17 additions & 17 deletions .github/workflows/build-fuzzy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,32 +18,32 @@ jobs:
include:
- NAME: Lapack + Libmath
DOCKER_FROM: ""
DOCKER_TO: fuzzy_v0.5.0-lapack
DOCKER_IMAGE: verificarlo/fuzzy:v0.5.0-lapack
DOCKER_TO: fuzzy_v0.6.0-lapack
DOCKER_IMAGE: verificarlo/fuzzy:v0.6.0-lapack
DOCKER_FILE: ./docker/base/Dockerfile.ubuntu20.04-0-lapack-libmath

- NAME: Python
DOCKER_FROM: fuzzy_v0.5.0-lapack
DOCKER_TO: fuzzy_v0.5.0-lapack-python3.8.5
DOCKER_IMAGE: verificarlo/fuzzy:v0.5.0-lapack-python3.8.5
DOCKER_FROM: fuzzy_v0.6.0-lapack
DOCKER_TO: fuzzy_v0.6.0-lapack-python3.8.5
DOCKER_IMAGE: verificarlo/fuzzy:v0.6.0-lapack-python3.8.5
DOCKER_FILE: ./docker/base/Dockerfile.ubuntu20.04-1-python

- NAME: Numpy
DOCKER_FROM: fuzzy_v0.5.0-lapack-python3.8.5
DOCKER_TO: fuzzy_v0.5.0-lapack-python3.8.5-numpy
DOCKER_IMAGE: verificarlo/fuzzy:v0.5.0-lapack-python3.8.5-numpy
DOCKER_FROM: fuzzy_v0.6.0-lapack-python3.8.5
DOCKER_TO: fuzzy_v0.6.0-lapack-python3.8.5-numpy
DOCKER_IMAGE: verificarlo/fuzzy:v0.6.0-lapack-python3.8.5-numpy
DOCKER_FILE: ./docker/base/Dockerfile.ubuntu20.04-2-numpy

- NAME: Scipy
DOCKER_FROM: fuzzy_v0.5.0-lapack-python3.8.5-numpy
DOCKER_TO: fuzzy_v0.5.0-lapack-python3.8.5-numpy-scipy
DOCKER_IMAGE: verificarlo/fuzzy:v0.5.0-lapack-python3.8.5-numpy-scipy
DOCKER_FROM: fuzzy_v0.6.0-lapack-python3.8.5-numpy
DOCKER_TO: fuzzy_v0.6.0-lapack-python3.8.5-numpy-scipy
DOCKER_IMAGE: verificarlo/fuzzy:v0.6.0-lapack-python3.8.5-numpy-scipy
DOCKER_FILE: ./docker/base/Dockerfile.ubuntu20.04-3-scipy

- NAME: Scikit-Learn
DOCKER_FROM: fuzzy_v0.5.0-lapack-python3.8.5-numpy-scipy
DOCKER_TO: fuzzy_v0.5.0-lapack-python3.8.5-numpy-scipy-sklearn
DOCKER_IMAGE: verificarlo/fuzzy:v0.5.0-lapack-python3.8.5-numpy-scipy-sklearn
DOCKER_FROM: fuzzy_v0.6.0-lapack-python3.8.5-numpy-scipy
DOCKER_TO: fuzzy_v0.6.0-lapack-python3.8.5-numpy-scipy-sklearn
DOCKER_IMAGE: verificarlo/fuzzy:v0.6.0-lapack-python3.8.5-numpy-scipy-sklearn
DOCKER_FILE: ./docker/base/Dockerfile.ubuntu20.04-4-sklearn

name: ${{ matrix.NAME }}
Expand Down Expand Up @@ -96,7 +96,7 @@ jobs:
docker build ./ -t ${{ matrix.DOCKER_IMAGE }} \
-f ${{ matrix.DOCKER_FILE }} \
--build-arg VERIFICARLO_VERSION=v0.5.0
--build-arg VERIFICARLO_VERSION=v0.6.0
docker save ${{ matrix.DOCKER_IMAGE }} -o /tmp/${{ matrix.DOCKER_TO }}.tar
Expand All @@ -117,9 +117,9 @@ jobs:
runs-on: ubuntu-20.04
env:
RUN_IF_CHANGED: docker
DOCKER_FROM: fuzzy_v0.5.0-lapack-python3.8.5-numpy-scipy-sklearn
DOCKER_FROM: fuzzy_v0.6.0-lapack-python3.8.5-numpy-scipy-sklearn
DOCKER_IMAGE: verificarlo/fuzzy
DOCKER_FROM_TAG: v0.5.0-lapack-python3.8.5-numpy-scipy-sklearn
DOCKER_FROM_TAG: v0.6.0-lapack-python3.8.5-numpy-scipy-sklearn

steps:
- name: Checkout repository
Expand Down
14 changes: 14 additions & 0 deletions CHANGELOG
Original file line number Diff line number Diff line change
@@ -1,3 +1,17 @@
# [v0.6.0] 2022/01/07

## Changed
* Change verificarlo version to compile fuzzy images [v0.5.0 -> v0.6.0]
* Change base3 sklearn version [v0.23.2 -> v1.0.2]
* Change base2 scipy version [v1.5.4 -> v1.7.3]
* Change base1 numpy version [v1.19.1 -> v1.22.0]

## Fixed
* Fix issue due to hardware specific instruction (AVX)
* Can be run on any x86_64 architecture now

## Deprecated
* Version v0.5.0 must not be used on architecture not supporting AVX-512
# [v0.5.0] 2021/06/18

## Changed
Expand Down
9 changes: 3 additions & 6 deletions docker/base/Dockerfile.ubuntu20.04-0-lapack-libmath
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@
ARG VERIFICARLO_VERSION=v0.5.0
ARG VERIFICARLO_VERSION=v0.6.0
FROM verificarlo/verificarlo:${VERIFICARLO_VERSION}

RUN cat /proc/cpuinfo | grep flags | uniq
RUN cat /proc/cpuinfo | grep "model name" | uniq

# Setup build dependencies
RUN apt-get update -qqq &&\
apt-get install -y --no-install-recommends -qqq \
Expand Down Expand Up @@ -51,8 +48,8 @@ RUN cd /opt/build/ &&\
mkdir build &&\
cd /opt/build/lapack-3.9.0/build &&\
cmake FC=verificarlo-f -DCMAKE_BUILD_TYPE=RELEASE -DCMAKE_C_COMPILER=verificarlo-c -DCMAKE_Fortran_COMPILER=verificarlo-f \
-DCBLAS=ON -DBUILD_SHARED_LIBS=ON -DCMAKE_C_FLAGS="--conservative -O0" -DCMAKE_Fortran_FLAGS="--conservative -O0" .. &&\
make &&\
-DCBLAS=ON -DBUILD_SHARED_LIBS=ON -DCMAKE_C_FLAGS="--conservative" -DCMAKE_Fortran_FLAGS="--conservative" .. &&\
make -j &&\
make install

# Remove temporary files
Expand Down
7 changes: 2 additions & 5 deletions docker/base/Dockerfile.ubuntu20.04-1-python
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@
ARG VERIFICARLO_VERSION=v0.5.0
ARG VERIFICARLO_VERSION=v0.6.0
FROM verificarlo/fuzzy:${VERIFICARLO_VERSION}-lapack

RUN cat /proc/cpuinfo | grep flags | uniq
RUN cat /proc/cpuinfo | grep "model name" | uniq

# Use IEEE mode for compiling with verificarlo
RUN echo "libinterflop_ieee.so" > $VFC_BACKENDS_FROM_FILE

Expand All @@ -15,7 +12,7 @@ RUN cd /opt/build/ && \
wget https://www.python.org/ftp/python/3.8.5/Python-3.8.5.tgz && \
tar xvf Python-3.8.5.tgz && \
cd Python-3.8.5 && \
CFLAGS="--exclude-file=/tmp/python-vfc-exclude.txt --conservative -O0" ./configure --with-ensurepip=install &&\
CFLAGS="--conservative --exclude-file=/tmp/python-vfc-exclude.txt" ./configure --with-ensurepip=install &&\
make -j &&\
make install &&\
wget https://bootstrap.pypa.io/get-pip.py &&\
Expand Down
34 changes: 14 additions & 20 deletions docker/base/Dockerfile.ubuntu20.04-2-numpy
Original file line number Diff line number Diff line change
@@ -1,36 +1,30 @@
ARG VERIFICARLO_VERSION=v0.5.0
ARG VERIFICARLO_VERSION=v0.6.0
FROM verificarlo/fuzzy:${VERIFICARLO_VERSION}-lapack-python3.8.5

RUN cat /proc/cpuinfo | grep flags | uniq
RUN cat /proc/cpuinfo | grep "model name" | uniq

# Use IEEE mode for compiling with verificarlo
RUN echo "libinterflop_ieee.so" > $VFC_BACKENDS_FROM_FILE

# Copy verificarlo's exclusion file for Python 3
COPY docker/resources/numpy-verificarlo.patch /tmp/numpy-verificarlo.patch
COPY docker/resources/numpy-vfc-exclude.txt /tmp/numpy-vfc-exclude.txt

RUN git config --global user.email "[email protected]" &&\
git config --global user.name "Anonymous Patcher"
COPY docker/resources/numpy-sanity-check.py /tmp/numpy-sanity-check.py

# Build numpy from sources and link with the local BLAS and LAPACK
RUN python3 -m pip install cython
RUN cd /opt/build &&\
git clone -b v1.19.1 --depth=1 https://github.com/numpy/numpy.git &&\
cd /opt/build/numpy &&\
git checkout v1.19.1 &&\
git am /tmp/numpy-verificarlo.patch &&\
OPT="--exclude-file=/tmp/numpy-vfc-exclude.txt -Wunused-command-line-argument --conservative -O0" \
FOPT="--exclude-file=/tmp/numpy-vfc-exclude.txt -Wunused-command-line-argument --conservative -O0" \
NPY_BLAS_ORDER=BLAS NPY_LAPACK_ORDER=LAPACK \
python3 setup.py config --compiler=verificarlo --fcompiler=verificarlof build_clib \
--compiler=verificarlo --fcompiler=verificarlof build_ext \
--compiler=verificarlo --fcompiler=verificarlof build -j 4 install
# Numpy links libraries with x86_64-linux-gnu-gcc

RUN ln -sf $(which verificarlo) $(which x86_64-linux-gnu-gcc)

RUN cd /opt/build/ &&\
wget https://github.com/numpy/numpy/releases/download/v1.22.0/numpy-1.22.0.tar.gz &&\
tar xvf numpy-1.22.0.tar.gz &&\
cd numpy-1.22.0 &&\
CC="verificarlo-c" FC="verificarlo-f" CXX="verificarlo-c++" CFGLAGS="--conservative --exclude-file=/tmp/numpy-vfc-exclude.txt -Wunused-command-line-argument" \
python3 setup.py build -j $(nproc) --disable-optimization install


# Smoked test for vectorization.
# We should run the full numpy test suit in next versions
RUN python3 -c "import numpy as np; x = np.array(range(4),dtype=np.float64); z=[i**2 for i in range(4)]; print(z, x**2); assert((x**2==z).all())"
RUN python3 /tmp/numpy-sanity-check.py

# Remove temporary files
RUN rm -rf /opt/build/*
Expand Down
36 changes: 24 additions & 12 deletions docker/base/Dockerfile.ubuntu20.04-3-scipy
Original file line number Diff line number Diff line change
@@ -1,22 +1,34 @@
ARG VERIFICARLO_VERSION=v0.5.0
ARG VERIFICARLO_VERSION=v0.6.0
FROM verificarlo/fuzzy:${VERIFICARLO_VERSION}-lapack-python3.8.5-numpy

RUN cat /proc/cpuinfo | grep flags | uniq
RUN cat /proc/cpuinfo | grep "model name" | uniq

# Load backend IEEE
RUN echo "libinterflop_ieee.so" > $VFC_BACKENDS_FROM_FILE

RUN pip3 install joblib
# Copy sanity check for SciPy
COPY docker/resources/numpy-sanity-check.py /tmp/scipy-sanity-check.py

# Install SciPy dependencies
RUN pip3 install joblib pythran pybind11
# Replace gfortran with verificarlo-f
RUN cp /usr/local/bin/verificarlo-f /usr/bin/gfortran
# Replace gcc with verificarlo-c
RUN ln -sf $(which verificarlo-c) $(which x86_64-linux-gnu-gcc)

# Remove debug flag which makes flag crash
RUN sed -i "294s/.*/ llvm_options = llvm_options.replace(\"'-g'\",'') if \"'-g'\" in llvm_options else llvm_options/" $(which verificarlo)

RUN cd /opt/build/ &&\
git clone https://github.com/scipy/scipy.git &&\
cd /opt/build/scipy &&\
git checkout v1.5.4 &&\
CFLAGS="--conservative -O0" FFLAGS="--conservative -O0" NPY_NUM_BUILD_JOBS=4 \
python3 setup.py config --compiler=verificarlo --fcompiler=verificarlof build_clib \
--compiler=verificarlo --fcompiler=verificarlof build_ext \
--compiler=verificarlo --fcompiler=verificarlof build -j 4 install
wget https://github.com/scipy/scipy/releases/download/v1.7.3/scipy-1.7.3.tar.gz &&\
tar xvf scipy-1.7.3.tar.gz &&\
cd scipy-1.7.3 &&\
CC="verificarlo-c" FC="verificarlo-f" CXX="verificarlo-c++" CFGLAGS="--conservative -Wunused-command-line-argument" \
python3 setup.py build --disable-optimization -j $(nproc) install

# Restore original verificarlo
RUN sed -i "294s/ llvm_options = llvm_options.replace(\"'-g'\",'') if \"'-g'\" in llvm_options else llvm_options//" $(which verificarlo)

# Sanity check
RUN python3 /tmp/scipy-sanity-check.py

# Remove temporary files
RUN rm -rf /opt/build/*
Expand Down
10 changes: 4 additions & 6 deletions docker/base/Dockerfile.ubuntu20.04-4-sklearn
Original file line number Diff line number Diff line change
@@ -1,17 +1,15 @@
ARG VERIFICARLO_VERSION=v0.5.0
ARG VERIFICARLO_VERSION=v0.6.0
FROM verificarlo/fuzzy:${VERIFICARLO_VERSION}-lapack-python3.8.5-numpy-scipy

RUN cat /proc/cpuinfo | grep flags | uniq
RUN cat /proc/cpuinfo | grep "model name" | uniq

# Load backend IEEE
RUN echo "libinterflop_ieee.so" > $VFC_BACKENDS_FROM_FILE

RUN cd /opt/build &&\
git clone https://github.com/scikit-learn/scikit-learn.git &&\
cd scikit-learn &&\
git checkout 0.23.2 &&\
CFLAGS="--conservative -O0" python3 setup.py build -j 4 install
git checkout 1.0.2 &&\
CC="verificarlo-c" FC="verificarlo-f" CXX="verificarlo-c++" CFGLAGS="--conservative -Wunused-command-line-argument"\
python3 setup.py build -j 4 install

# Remove temporary files
RUN rm -rf /opt/build/*
Expand Down
27 changes: 27 additions & 0 deletions docker/resources/numpy-sanity-check.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
#!/usr/bin/env python3

import numpy as np


def test_instrumentation():
"""[Test numpy is instrumented properly]"""
res = len(set([np.sum([0.1] * 1000) for i in range(100)]))
assert res != 1


def test_vecto():
"""[Test vectorization is handled properly]"""
x = np.array(range(4), dtype=np.float64)
z = [i ** 2 for i in range(4)]
assert (x ** 2 == z).all()


def test_arange():
"""[Test that PyArray_Arange is not instrumented]"""
sizes = all([len(np.linspace(0, 1, 10)) == 10 for i in range(100)])
assert sizes


if "__main__" == __name__:
test_vecto()
test_arange()
Loading

0 comments on commit 0cc650f

Please sign in to comment.