CMake error when building v0.87.7 in CentOS 7

[shadowleaves]

CMake seems to complain about missing fortran library for OpenBLAS but I have both gfortran and libgfortran installed at /usr/bin and usr/lib64 respectively. setting GFORTRAN_LIB=/usr/lib64 does not help either.

-- Appending /tmp/Elemental/include for Elemental's source includes
-- Appending /tmp/Elemental/build/include for Elemental's binary includes
CMake Warning at CMakeLists.txt:286 (message):
Build mode not specified, defaulting to Release build.

-- CXX11_COMPILER_FLAGS=-std=gnu++11
-- Using restrict keyword.
-- Will parse MPI header /usr/local/include/mpi.h
-- Appending /usr/local/include for MPI headers
-- Using prespecified OpenMP_C_FLAGS=-fopenmp
-- Using prespecified OpenMP_CXX_FLAGS=-fopenmp
-- Valgrind Prefix:
-- Could NOT find VALGRIND (missing: VALGRIND_INCLUDE_DIR VALGRIND_PROGRAM)
-- A library with BLAS API found.
-- A library with LAPACK API found.
CMake Error at cmake/external_projects/ElMath/OpenBLAS.cmake:43 (message):
Could not find gfortran library; please consider setting the GFORTRAN_LIB
variable.
Call Stack (most recent call first):
cmake/external_projects/ElMath.cmake:251 (include)
CMakeLists.txt:434 (include)

-- Configuring incomplete, errors occurred!
See also "/tmp/Elemental/build/CMakeFiles/CMakeOutput.log".
See also "/tmp/Elemental/build/CMakeFiles/CMakeError.log".

[shadowleaves]

looks like this issue has something to do with the location of libgfortran under gcc folder in different versions of linux. on Ubuntu 14.04, where I was able to succesfully build Elemental 0.87.7, the location is as below:

/usr/lib/gcc/x86_64-linux-gnu/4.8/libgfortran.a
/usr/lib/gcc/x86_64-linux-gnu/4.8/libgfortran.so
/usr/lib/gcc/x86_64-linux-gnu/4.8/libgfortran.spec
/usr/lib/gcc/x86_64-linux-gnu/4.8/libgfortranbegin.a

whereas in CentOS 7, the location is at:
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/libgfortran.so
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/libgfortran.spec
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/libgfortranbegin.a

for CentOS 7 installation, the error can be avoided by adding symbolic links in /usr/lib/gcc/ so that /usr/lib/gcc/x86_64-redhat-linux/4.8.5/ (or whatever latest 4.8 version installed) will be mapped to /usr/lib/gcc/x86_64-linux-gnu/4.8/.

I suggest adding proper mapping (e.g., /usr/lib/gcc/x86_64-redhat-linux/4.8.5/) to the /Elemental/blob/master/cmake/external_projects/ElMath/OpenBLAS.cmake file (line 20-36) to make it work on CentOS (and probably related redhat systems).

[shadowleaves]

(following-up)
although cmake/make compilation and installation can go through via the method described above, most of the python code in the examples/interface directory will fail on some openBLAS-related errors. for example:

$ python GLM.py 
Process 0 caught error message:
Argument 13 had an illegal value

Traceback (most recent call last):
  File "GLM.py", line 90, in <module>
    print('GLM time: {} seconds'.format(endGLM-startGLM))
ValueError: On entry to DSTEVR parameter number 13 had an illegal value

another one:

$ python TV.py 
lambda = 1.0
Process 0 caught error message:
Argument 13 had an illegal value

Traceback (most recent call last):
  File "TV.py", line 51, in <module>
    print('TV time: {}'.format(endTV-startTV))
ValueError: On entry to DSTEVR parameter number 13 had an illegal value

@poulson any idea what might be causing this?

[poulson]

I can reply in more detail later, but this sounds like a mismatch between 32bit and 64bit BLAS interfaces. There is a flag for building El with a 64bit BLAS API that needs to be set. It is unfortunately nontrivial to determine this automatically.

[shadowleaves]

thanks. would you please elaborate on the flag?
also I have found that re-building and installing gcc-4.9.4 from source will fix the above gfortran_lib issue without the need of adding symbolic links. however the openBLAS-related error still persists.

[poulson]

The flag is EL_USE_64BIT_BLAS_INTS: https://github.com/elemental/Elemental/blob/master/CMakeLists.txt#L131

[shadowleaves]

Thanks. I tried to re-config with cmake:
cmake .. -DEL_USE_64BIT_BLAS_INTS=TRUE

and see the following warnings:

CMake Warning at cmake/external_projects/ElMath/OpenBLAS.cmake:82 (message):
  OpenBLAS was found as /usr/lib64/libopenblas64.so, but BLAS support was not
  detected
Call Stack (most recent call first):
  cmake/external_projects/ElMath.cmake:251 (include)
  CMakeLists.txt:434 (include)

CMake Warning at cmake/external_projects/ElMath/OpenBLAS.cmake:87 (message):
  OpenBLAS was found as /usr/lib64/libopenblas64.so, but LAPACK support was
  not detected
Call Stack (most recent call first):
  cmake/external_projects/ElMath.cmake:251 (include)
  CMakeLists.txt:434 (include)

which is somewhat expected given the 64-bit integer setting usually not switched on with regular OpenBLAS installations. and so Elemental's cmake will grab openBLAS source from github and build by itself with 64-bit integer support.

but then when I do sudo make in the build directory, it runs into another error:

[  0%] [  0%] Built target El_config
[  3%] Built target ElSuiteSparse
Performing install step for 'project_openblas'
gmake[3]: warning: jobserver unavailable: using -j1.  Add `+' to parent make rule.
gmake[3]: warning: jobserver unavailable: using -j1.  Add `+' to parent make rule.
Generating openblas_config.h in /usr/local/include
Generating f77blas.h in /usr/local/include
Generating cblas.h in /usr/local/include
Copying LAPACKE header files to /usr/local/include
Copying the static library to /usr/local/lib
install: cannot stat ‘libopenblas_haswell-r0.2.15.a’: No such file or directory
gmake[4]: *** [install] Error 1
gmake[3]: *** [install] Error 2
make[2]: *** [download/OpenBLAS/build/stamp/project_openblas-install] Error 2
make[1]: *** [CMakeFiles/project_openblas.dir/all] Error 2
make: *** [all] Error 2

[poulson]

The issue using the old version is likely due to an undefined reference due to a missing library; you can see what caused it by looking in CMakeFiles/CMakeError.log for undefined references related to BLAS and LAPACK.

The latter appears to be an issue with the OpenBLAS Makefile, but I'm a bit surprised since Elemental should pull down a particular fixed tag, so I would have expected to have run into this before.

[shadowleaves]

Thanks. I ended up with building and installing OpenBLAS separately with make INTERFACE64=1 option and then build Elemental without the EL_USE_64BIT_BLAS_INTS switch. I was able to build it without errors and the resulting libraries can pass most of the tests in examples/interface folder.

however when using Elemental from CVXPY, i still end up with the following error:
Intel MKL ERROR: Parameter 13 was incorrect on entry to DSTEVR.

I assume this has something to do with CVXPY's usage of OpenBLAS? maybe it needs to be compiled with INTERFACE64=1 as well?

[poulson]

I'm glad to hear you fixed the issue (albeit in a roundabout way). That issue again sounds like a 64-bit issue; I'm not sure what CVXPY's API choice is though.