Compiling R from source with MKL using icc

References

• an r-devel thread
• Another R-devel thread
• John's script on Hyak
• Intel's instructions for building R with MKL
• Another set of instructions from Intel for building R with MKL
• Intel's link advisor
• Static MKL with R https://martynplummer.files.wordpress.com/2014/10/r-intel-fedora.pdf

Documenting things that didn't work (R 3.6.2, mkl 2021.3.0, on centos 7.6.1810)

All the following was tested on this image: https://github.com/kaufman-lab/build_containers/pkgs/container/mkl_centos7

• Intel link advisor recommends -DMKL_ILP64 for Cflags and -i8 for fortran flags, but these cause a compile error ifort and icc -std=gnu99 "disagree on int and double"

1. The following (using code advised for static linking) compiles but sessionInfo() displays default blas (not MKL).

   source /opt/intel/oneapi/setvars.sh
   
   export CC="icc"
   export CXX="icpc"
   export F77="ifort"
   export FC="ifort"
   export AR="xiar"
   export LD="xild"
   
   export CFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export CXXFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export FFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export FCFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export LDFLAGS="-qopenmp"
   
   MKL_LIB_PATH="${MKLROOT}/lib/intel64/"
   MKL=" -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_ilp64.a ${MKLROOT}/lib/intel64/libmkl_intel_thread.a ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group -liomp5 -lpthread -lm -ldl"
   
   
   ./configure --prefix=/home/staff/myoung3/Rinst --enable-R-static-lib --with-blas="$MKL" --with-lapack --with-x=no --enable-memory-profiling --disable-BLAS-shlib
   make
   make install
   

   from the log

    ifort -c -I/opt/intel/oneapi/mkl/2021.3.0/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8  conftest.F >&5
   conftest.F(3): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: ( : % [ . = =>
      choke me
   
   

2. The following (using code advised via link advisor for dynamic linking) compiles but sessionInfo() displays default blas (not MKL).

   source /opt/intel/oneapi/setvars.sh
   
   export CC="icc"
   export CXX="icpc"
   export F77="ifort"
   export FC="ifort"
   export AR="xiar"
   export LD="xild"
   
   export CFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export CXXFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export FFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export FCFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export LDFLAGS="-qopenmp"
   
   
   
   MKL="-L${MKLROOT}/lib/intel64 -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl"
   
   
   ./configure --prefix=/home/staff/myoung3/Rinst --enable-R-shlib --with-blas="$MKL" --with-lapack --with-x=no --enable-memory-profiling
   make
   make install
   
   
   
   

• R-dev shows success with -lmkl_intel_lp64 rather than -lmkl_intel_ilp64 compare discussions with lmkl_intel_lp64 vs with lmkl_intel_ilp64

3. switching out ilp64 for lp64 causes MKL to get compile successfully, but and it's as fast as reference BLAS (and faster than a poorly optimized MKL installation on bare hardware on the same machine), but it doesn't scale with MKL_NUM_THREADs:

   rm -rf ~/Rinst
   rm -rf R-3.6.2/
     tar zxf R-3.6.2.tar.gz
   cd R-3.6.2/
     mkdir ~/Rinst
   
   
   source /opt/intel/oneapi/setvars.sh
   
   export CC="icc"
   export CXX="icpc"
   export F77="ifort"
   export FC="ifort"
   export AR="xiar"
   export LD="xild"
   
   export CFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export CXXFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export FFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export FCFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export LDFLAGS="-qopenmp"
   
   
   
   MKL="-L${MKLROOT}/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl"
   
   
   ./configure --prefix=/home/staff/myoung3/Rinst --enable-R-shlib --with-blas="$MKL" --with-lapack --with-x=no --enable-memory-profiling --disable-java
   make
   make install
   

   Note you may encounter this error when trying to start R:

   R
   /home/staff/myoung3/Rinst/lib64/R/bin/exec/R: error while loading shared libraries: libiomp5.so: cannot open shared object file: No such file or directory
   

   which can be solved by running source /opt/intel/oneapi/setvars.sh first. Here's the result from sessionInfo(): BLAS/LAPACK: /opt/intel/oneapi/mkl/2021.3.0/lib/intel64/libmkl_intel_lp64.so.1 Weirdly, performance seems to dependent on whether the singularty image has been restarted (or setvars.sh has been sourced again). Immediately after building, this is the performance (bad, worse than reference blas)

   system.time(crossprod(matrix(1:1e9, ncol=1000))[1])
      user  system elapsed
    31.384   5.480  36.895
   > Sys.setenv(OMP_NUM_THREADS=16)
   >
   > system.time(crossprod(matrix(1:1e9, ncol=1000))[1])
      user  system elapsed
    31.672   5.336  37.023
   >
   >  Sys.setenv(MKL_THREADING_LAYER="GNU")
   > system.time(crossprod(matrix(1:1e9, ncol=1000))[1])
      user  system elapsed
    30.848   5.520  36.383
   > quit()
   

   Then after starting a new image and sourcing setvars.sh, this is the performance (comparable to reference blas):

    > system.time(crossprod(matrix(1:1e9, ncol=1000))[1])
       user  system elapsed
     97.228   5.740  15.149
    > Sys.setenv(OMP_NUM_THREADS=16)
    >
    > system.time(crossprod(matrix(1:1e9, ncol=1000))[1])
   
     Sys.setenv(MKL_THREADING_LAYER="GNU")
    system.time(crossprod(matrix(1:1e9, ncol=1000))[1])
       user  system elapsed
     89.304   5.600  13.906
    >
    >  Sys.setenv(MKL_THREADING_LAYER="GNU")
    > system.time(crossprod(matrix(1:1e9, ncol=1000))[1])
       user  system elapsed
     77.676   5.620  13.345
    >
   

4. This doesn't compile, (based on this: https://stat.ethz.ch/pipermail/r-devel/2015-September/071717.html)

   rm -rf ~/Rinst
   rm -rf R-3.6.2/
     tar zxf R-3.6.2.tar.gz
   cd R-3.6.2/
     mkdir ~/Rinst
   
   source /opt/intel/oneapi/setvars.sh
   
   fast="-ip -O3 -qopt-mem-layout-trans=3 -march=skylake-avx512"
   export CC="icc"
   export CFLAGS="$fast -wd188 -fp-model precise"
   export F77="ifort"
   export FFLAGS="$fast -fp-model precise"
   export CXX="icpc"
   export CXXFLAGS="$fast -fp-model precise"
   export FC="ifort"
   export FCFLAGS="$fast -fp-model precise"
   
   ./configure --prefix=/home/staff/myoung3/Rinst --with-blas='-mkl=parallel' --with-lapack --enable-BLAS-shlib --with-x=no --enable-memory-profiling
   make
   make install
   

5. switching out the mkl call to what was used above causes this to compile successfully, but it's actually slower than above and still not actually doing multithreaded calculations:

   rm -rf ~/Rinst
   rm -rf R-3.6.2/
     tar zxf R-3.6.2.tar.gz
   cd R-3.6.2/
     mkdir ~/Rinst
   
   
   
   source /opt/intel/oneapi/setvars.sh
   
   fast="-ip -O3 -qopt-mem-layout-trans=3 -march=skylake-avx512"
   export CC="icc"
   export CFLAGS="$fast -wd188 -fp-model precise"
   export F77="ifort"
   export FFLAGS="$fast -fp-model precise"
   export CXX="icpc"
   export CXXFLAGS="$fast -fp-model precise"
   export FC="ifort"
   export FCFLAGS="$fast -fp-model precise"
   
   MKL="-L${MKLROOT}/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl"
   
   
   ./configure --prefix=/home/staff/myoung3/Rinst --enable-R-shlib --with-blas="$MKL" --with-lapack --with-x=no --enable-memory-profiling
   make
   make install
   
   

   Performance (unlike above in #3, this is the same after restarting the image and resourcing setvars.sh)

   Sys.setenv(OMP_NUM_THREADS=1)
   system.time(crossprod(matrix(1:1e9, ncol=1000))[1])
   Sys.setenv(OMP_NUM_THREADS=16)
   
   system.time(crossprod(matrix(1:1e9, ncol=1000))[1])
   
    Sys.setenv(MKL_THREADING_LAYER="GNU")
   system.time(crossprod(matrix(1:1e9, ncol=1000))[1])
   

   I suspect this is slower because it only uses -ip rather than -ipo as in 3. ipo has been reported to cause compilation issues, but subsequently reported to work now (see r-devel references above). But I haven't tested this difference specifically and there are other differences compared to 3.

6. Use static BLAS (the previous attempts have been a weird mix of static specified in config (since it's the default and sh isn't specified) but with output generated from the intel link advisor corresponding to dynamic linkage. Additionally, and more imporatantly, use libmkl_gnu_thread instead of libmkl_intel_thread since this is what's used in the official R installation and Administration guide. This compiles successfully, but sessionInfo() shows. Note I'm also adding -qopenmp here to the MKL flags.

   Matrix products: default
   BLAS:   /home/staff/myoung3/Rinst/lib64/R/lib/libR.so
   LAPACK: /home/staff/myoung3/Rinst/lib64/R/modules/lapack.so
   

   I'm not sure what this is supposed look like when BLAS has been linked statically, so this may be ok? In any case the performance bad, even after restarting singularity

   rm -rf ~/Rinst
   rm -rf R-3.6.2/
     tar zxf R-3.6.2.tar.gz
   cd R-3.6.2/
     mkdir ~/Rinst
   
   
   source /opt/intel/oneapi/setvars.sh
   
   export CC="icc"
   export CXX="icpc"
   export F77="ifort"
   export FC="ifort"
   export AR="xiar"
   export LD="xild"
   
   export CFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export CXXFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export FFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export FCFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export LDFLAGS="-qopenmp"
   
   
   MKL=" -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a ${MKLROOT}/lib/intel64/libmkl_gnu_thread.a ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group -liomp5 -lpthread -lm -ldl -qopenmp"
   
   
   ./configure --prefix=/home/staff/myoung3/Rinst --enable-R-shlib --with-blas="$MKL" --with-lapack --disable-BLAS-shlib --with-x=no --enable-memory-profiling --disable-java
   make
   make install
   
   

7. rerun pretty much the same code but link MKL dynamically. also turn off memory profiling. Compiles correctly. sessionInfo shows BLAS/LAPACK: /opt/intel/oneapi/mkl/2021.3.0/lib/intel64/libmkl_intel_lp64.so.1, but performance is bad and doesn't scale with MKL_NUM_THREADS.

   rm -rf ~/Rinst
   rm -rf R-3.6.2/
   tar zxf R-3.6.2.tar.gz
   cd R-3.6.2/
   mkdir ~/Rinst
   
   
   source /opt/intel/oneapi/setvars.sh
   
   export CC="icc"
   export CXX="icpc"
   export F77="ifort"
   export FC="ifort"
   export AR="xiar"
   export LD="xild"
   
   export CFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export CXXFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export FFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export FCFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export LDFLAGS="-qopenmp"
   
   MKL="-L${MKLROOT}/lib/intel64 -lmkl_intel_lp64 -lmkl_gnu_thread -lmkl_core -lpthread -lm -ldl -qopenmp"
   
   ./configure --prefix=/home/staff/myoung3/Rinst --enable-R-shlib --with-blas="$MKL" --with-lapack --enable-BLAS-shlib --with-x=no --disable-java
   make
   make install
   
   

• Potentially relevant: https://github.com/spack/spack/issues/8642. Does this mean mkl_gf_lp64 needs to be specified instead?

  to use (a sequential version of) the MKL with R when building it with GCC (and GFortran), it seems necessary to link to three shared MKL libraries: mkl_gf_lp64, mkl_core and mkl_sequential (see https://cran.r-project.org/doc/manuals/r-release/R-admin.html#MKL and https://software.intel.com/en-us/articles/using-intel-mkl-with-r para. B). The intel-mkl package in Spack does expose the latter two, but not the first (mkl_gf_lp64). Instead, it only provides mkl_intel_lp64 (or mkl_intel_ilp64 when requesting the ilp64 variant). The difference seems to be relevant when GFortran is involved in the build process, see https://software.intel.com/en-us/forums/intel-math-kernel-library/topic/558132. Unfortunately, this does not raise any immediate (fatal) errors (only some hints) during compile time, but the resulting R installation is not linking to the MKL, falling back to the internal BLAS/LAPACK implementation instead:

8. switch to lmkl_gf_lp64. This compiles but the sessionInfo shows default blas being used, not mkl.

   rm -rf ~/Rinst
   rm -rf R-3.6.2/
   tar zxf R-3.6.2.tar.gz
   cd R-3.6.2/
   mkdir ~/Rinst
   
   
   source /opt/intel/oneapi/setvars.sh
   
   export CC="icc"
   export CXX="icpc"
   export F77="ifort"
   export FC="ifort"
   export AR="xiar"
   export LD="xild"
   
   export CFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export CXXFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export FFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export FCFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export LDFLAGS="-qopenmp"
   
   MKL="-L${MKLROOT}/lib/intel64 -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core -lpthread -lm -ldl -qopenmp"
   
   ./configure --prefix=/home/staff/myoung3/Rinst --enable-R-shlib --with-blas="$MKL" --with-lapack --enable-BLAS-shlib --with-x=no --disable-java
   make
   make install
   
   

9. Drop other flags in MKL to match R-admin guide. again this compiles but MKL isn't being used according to sessionInfo

   rm -rf ~/Rinst
   rm -rf R-3.6.2/
   tar zxf R-3.6.2.tar.gz
   cd R-3.6.2/
   mkdir ~/Rinst
   
   
   source /opt/intel/oneapi/setvars.sh
   
   export CC="icc"
   export CXX="icpc"
   export F77="ifort"
   export FC="ifort"
   export AR="xiar"
   export LD="xild"
   
   export CFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export CXXFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export FFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export FCFLAGS="-I${MKLROOT}/include -fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=8"
   export LDFLAGS="-qopenmp"
   
   MKL="-L${MKLROOT}/lib/intel64 -lmkl_gf_lp64 -lmkl_core -lmkl_gnu_thread -dl -qopenmp"
   
   ./configure --prefix=/home/staff/myoung3/Rinst --enable-R-shlib --with-blas="$MKL" --with-lapack --enable-BLAS-shlib --with-x=no --disable-java
   make
   make install
   
   

• mkl breaking between 7 and 8 (switching to lmkl_gf_lp64 from lmkl_intel_lp64) is a key finding. Why is this happening? Double check that's the only change.

• Also it may be worth trying to at least get mkl sequential working.

• also, read carefully this: https://github.com/eddelbuettel/mkl4deb/issues/2#issuecomment-382530940

  If no environment variable is set the default configuration is C LP64 + Intel OpenMP threading layer. So at run-time MKL will load libiomp5.so and libmkl_intel_thread.so. If your application or its dependencies use GNU OpenMP (e.g. some parts of it is built with gcc and -fopenmp flags) then Linux dynamic linker will load GNU OpenMP RT as well. So at that moment your application will have 2 OpenMP run-times. That typically leads to numerical errors or even crashes.

  which suggests whenever people link libmkl_rt without doing anything else, it's really using liomp5 and libmkl_intel_thread. An alterative approach might be to try to use those or just use libmkl_rt (and specify MKL_THREADING_LAYER=GNU).

10. This works, and scales with number of threads. You need to run source /opt/intel/oneapi/setvars.sh to get R to load without error. The key is that you need to set threads before starting R. Number of threads cannot be modified once R is running, even with Sys.setenv() export OMP_NUM_THREADS=1 for single-threaded and e.g export OMP_NUM_THREADS=16 for a specific number. If it's unset, a reasonable number of threads will be used as the default if it's unset. This suggets that #3 above was probably actually working correctly.

    rm -rf ~/Rinst
    rm -rf R-3.6.2/
    tar zxf R-3.6.2.tar.gz
    cd R-3.6.2/
    mkdir ~/Rinst
    
    
    source /opt/intel/oneapi/setvars.sh
    
    export CC="icc"
    export CXX="icpc"
    export F77="ifort"
    export FC="ifort"
    export AR="xiar"
    export LD="xild"
    
    export CFLAGS="-fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=20"
    export CXXFLAGS="-fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=20"
    export FFLAGS="-fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=20"
    export FCFLAGS="-fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=20"
    export LDFLAGS="-qopenmp"
    
    MKL="-lmkl_rt -lpthread"
    
    ./configure --prefix=/home/staff/myoung3/Rinst --enable-R-shlib --with-blas="$MKL" --with-lapack --enable-BLAS-shlib --with-x=no --disable-java
    make -j20
    make install
    

11. Attempt to link statically. this works, and scales with OMP_NUM_THREADS. But if I attempt to run R without first sourcing setvars.sh I get this error:

    Singularity> ./Rinst/bin/R
    /home/staff/myoung3/Rinst/lib64/R/bin/exec/R: error while loading shared libraries: libiomp5.so: cannot open shared object file: No such file or directory
    

    It would be nice to have this fully static so I can delete the intel installation if that's possible...

    rm -rf ~/Rinst
    rm -rf R-3.6.2/
    tar zxf R-3.6.2.tar.gz
    cd R-3.6.2/
    mkdir ~/Rinst
    
    
    source /opt/intel/oneapi/setvars.sh
    
    export CC="icc"
    export CXX="icpc"
    export F77="ifort"
    export FC="ifort"
    export AR="xiar"
    export LD="xild"
    
    export CFLAGS="-fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=20"
    export CXXFLAGS="-fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=20"
    export FFLAGS="-fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=20"
    export FCFLAGS="-fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=20"
    export LDFLAGS="-qopenmp"
    
    MKL="-lmkl_rt -lpthread"
    
    ./configure --prefix=/home/staff/myoung3/Rinst --enable-R-shlib --with-blas="$MKL" --with-lapack --disable-BLAS-shlib --with-x=no --disable-java
    make -j20
    make install
    
    

12. Adding -liomp5 to MKL call doesn't fix the issue. At least MKL still works with threading here:

    rm -rf ~/Rinst
    rm -rf R-3.6.2/
    tar zxf R-3.6.2.tar.gz
    cd R-3.6.2/
    mkdir ~/Rinst
    
    
    source /opt/intel/oneapi/setvars.sh
    
    export CC="icc"
    export CXX="icpc"
    export F77="ifort"
    export FC="ifort"
    export AR="xiar"
    export LD="xild"
    
    export CFLAGS="-fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=20"
    export CXXFLAGS="-fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=20"
    export FFLAGS="-fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=20"
    export FCFLAGS="-fPIC -qopenmp -O3 -ipo -xHost -multiple-processes=20"
    export LDFLAGS="-qopenmp"
    
    MKL="-lmkl_rt -lpthread -liomp5"
    
    ./configure --prefix=/home/staff/myoung3/Rinst --enable-R-shlib --with-blas="$MKL" --with-lapack --disable-BLAS-shlib --with-x=no --disable-java
    make -j20
    make install
    
    

• This post suggests that libiomp5 is a library part of the icc compiler. since i'm compiling with icc rather than gcc, i need to figure out a way to link this statically.

  Intel MKL-DNN v0.x had an option to be built against Intel MKL (which comes with libiomp5.so -- Intel OpenMP runtime). If a user chooses to do so, the make install command would copy Intel MKL-DNN library, Intel MKL library, and libiomp5.so.

  Starting with Intel MKL-DNN v1.0 we removed that (optional) dependency on Intel MKL by porting all necessary functions from Intel MKL to Intel MKL-DNN. The OpenMP runtime is assumed to be the compiler's one. So if you built Intel MKL-DNN with GCC, the library would depend on libgomp.so and you don't need libiomp5.so at all.

  Could you please elaborate on why you are looking for libiomp5.so?

  Where can I find libiomp5 library?

  Just in case, you can find it in Intel MKL small libraries we shipped for Intel MKL-DNN v0.x releases. For instance, here:

  mklml_lnx_2019.0.5.20190502.tgz mklml_win_2019.0.5.20190502.zip

• Docs suggest I can link omp statically with -qopenmp-link=static. I probably need to use -static-intel as well (perhaps this is why attempt #6 didn't work).

• Some code I just found on building R with MKL linked statically. (https://martynplummer.files.wordpress.com/2014/10/r-intel-fedora.pdf)

   MKL_LIB_PATH=${MKLROOT}/lib/intel64
   MKL="-Wl,--start-group ${MKL_LIB_PATH}/libmkl_gf_lp64.a \
   ${MKL_LIB_PATH}/libmkl_intel_thread.a \
   ${MKL_LIB_PATH}/libmkl_core.a \
   -Wl,--end-group -liomp5 -lpthread -ldl -lm"
   CC="icc -std=c99" \
   CXX="icpc" \
   CFLAGS="-g -O3 -fopenmp" \
   CXXFLAGS="-g -O3 -fopenmp" \
   FFLAGS="-g -O2 -fopenmp" \
   FCFLAGS="-g -O2 -fopenmp" \
   FLIBS="-lgfortran" \
   LDFLAGS="-liomp5 -lpthread -static-intel -no-intel-extensions" \
   ../R/configure --with-blas="$MKL" --enable-BLAS-shlib
  

13. using static-intel and -qopenm-link=static didn't work...still getting lipiom5.so error

    rm -rf ~/Rinst
    rm -rf R-3.6.2/
    tar zxf R-3.6.2.tar.gz
    cd R-3.6.2/
    mkdir ~/Rinst
    
    
    source /opt/intel/oneapi/setvars.sh
    
    export CC="icc"
    export CXX="icpc"
    export F77="ifort"
    export FC="ifort"
    export AR="xiar"
    export LD="xild"
    
    export CFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export CXXFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export FFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export FCFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export LDFLAGS="-liomp5 -lpthread -qopenmp-link=static -static-intel -no-intel-extensions" 
    
    MKL="-lmkl_rt -lpthread -liomp5 -qopenmp-link=static"
    
    ./configure --prefix=/home/staff/myoung3/Rinst --enable-R-shlib --with-blas="$MKL" --with-lapack --disable-BLAS-shlib --with-x=no --disable-java
    make -j20
    make install
    
    

14. Dropped all the liomp5 links. different error message. progress

/home/staff/myoung3/Rinst/lib64/R/bin/exec/R: error while loading shared libraries: libmkl_rt.so.1: cannot open shared object file: No such file or directory

```
rm -rf ~/Rinst
rm -rf R-3.6.2/
tar zxf R-3.6.2.tar.gz
cd R-3.6.2/
mkdir ~/Rinst


source /opt/intel/oneapi/setvars.sh

export CC="icc"
export CXX="icpc"
export F77="ifort"
export FC="ifort"
export AR="xiar"
export LD="xild"

export CFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
export CXXFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
export FFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
export FCFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
export LDFLAGS="-lpthread -qopenmp-link=static -static-intel -no-intel-extensions" 

MKL="-lmkl_rt -lpthread -qopenmp-link=static"

./configure --prefix=/home/staff/myoung3/Rinst --enable-R-shlib --with-blas="$MKL" --with-lapack --disable-BLAS-shlib --with-x=no --disable-java
make -j20
make install

```

15. Drop lmkl_rt. Different error message. Try to specify libraries without using mkl_rt, which doesn't seem like it can be linked statically two issues: first, this library is missing

    /home/staff/myoung3/Rinst/lib64/R/bin/exec/R: error while loading shared libraries: libifport.so.5: cannot open shared object file: No such file or directory
    

    secondly and more problematic, after running source /opt/intel/oneapi/setvars.sh, sessionInfo() shows reference BLAS.

    rm -rf ~/Rinst
    rm -rf R-3.6.2/
    tar zxf R-3.6.2.tar.gz
    cd R-3.6.2/
    mkdir ~/Rinst
    
    
    source /opt/intel/oneapi/setvars.sh
    
    export CC="icc"
    export CXX="icpc"
    export F77="ifort"
    export FC="ifort"
    export AR="xiar"
    export LD="xild"
    
    export CFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export CXXFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export FFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export FCFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export LDFLAGS="-lpthread -qopenmp-link=static -static-intel -no-intel-extensions" 
    
    MKL=" -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a ${MKLROOT}/lib/intel64/libmkl_intel_thread.a ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm -ldl -qopenmp-link=static"
    
    ./configure --prefix=/home/staff/myoung3/Rinst --enable-R-shlib --with-blas="$MKL" --with-lapack --disable-BLAS-shlib --with-x=no --disable-java
    make -j20
    make install
    
    

16. This didn't work either. compiled correctly but no MKL

    rm -rf ~/Rinst
    rm -rf R-3.6.2/
    tar zxf R-3.6.2.tar.gz
    cd R-3.6.2/
    mkdir ~/Rinst
    
    
    source /opt/intel/oneapi/setvars.sh
    
    export CC="icc"
    export CXX="icpc"
    export F77="ifort"
    export FC="ifort"
    export AR="xiar"
    export LD="xild"
    
    export CFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export CXXFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export FFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export FCFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export LDFLAGS="-lpthread -qopenmp-link=static -static-intel -no-intel-extensions" 
    
    MKL="-L${MKLROOT}/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -qopenmp-link=static -lpthread -lm -ldl"
    
    ./configure --prefix=/home/staff/myoung3/Rinst --enable-R-shlib --with-blas="$MKL" --with-lapack --disable-BLAS-shlib --with-x=no --disable-java
    make -j20
    make install
    
    

17. link libifport--this doesn't work. I still get the same error about libifport.so.5

    rm -rf ~/Rinst
    rm -rf R-3.6.2/
      tar zxf R-3.6.2.tar.gz
    cd R-3.6.2/
      mkdir ~/Rinst
    
    
    source /opt/intel/oneapi/setvars.sh
    
    export CC="icc"
    export CXX="icpc"
    export F77="ifort"
    export FC="ifort"
    export AR="xiar"
    export LD="xild"
    
    export CFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export CXXFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export FFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export FCFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export LDFLAGS="-lpthread -qopenmp-link=static -static-intel -no-intel-extensions" 
    
    
    MKL=" -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a ${MKLROOT}/lib/intel64/libmkl_intel_thread.a ${MKLROOT}/lib/intel64/libmkl_core.a /opt/intel/oneapi/compiler/2021.3.0/linux/compiler/lib/intel64_lin/libifport.a -Wl,--end-group -lpthread -lm -ldl -qopenmp-link=static"
    
    ./configure --prefix=/home/staff/myoung3/Rinst --enable-R-shlib --with-blas="$MKL" --with-lapack --disable-BLAS-shlib --with-x=no --disable-java
    make -j20
    make install
    

• note:

  Statically linking to MKL and using the option --enable-BLAS-shlib pulls the MKL BLAS functions into a dynamic library libRblas.so which is available to all R packages, including

future packages installed by the user.

This suggests I probably shouldn't be specifying disable-BLAS-shlib

17. still getting reference blas when specifying enable-BLAS-shlib

    rm -rf ~/Rinst
    rm -rf R-3.6.2/
      tar zxf R-3.6.2.tar.gz
    cd R-3.6.2/
      mkdir ~/Rinst
    
    
    source /opt/intel/oneapi/setvars.sh
    
    export CC="icc"
    export CXX="icpc"
    export F77="ifort"
    export FC="ifort"
    export AR="xiar"
    export LD="xild"
    
    export CFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export CXXFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export FFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export FCFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export LDFLAGS="-lpthread -qopenmp-link=static -static-intel -no-intel-extensions" 
    
    
    MKL=" -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a ${MKLROOT}/lib/intel64/libmkl_intel_thread.a ${MKLROOT}/lib/intel64/libmkl_core.a  -Wl,--end-group -lpthread -lm -ldl -qopenmp-link=static"
    
    ./configure --prefix=/home/staff/myoung3/Rinst --enable-R-shlib --with-blas="$MKL" --with-lapack --enable-BLAS-shlib --with-x=no --disable-java
    make -j20
    make install
    

18. Based on some vague error messages in the log, try linking libiomp5.a explicitly (even though -qopenmp.link=static should be doing this). This seemed to clean up some error messages but didn't fix the problem--still reference blas in sessionInfo()

    rm -rf ~/Rinst
    rm -rf R-3.6.2/
      tar zxf R-3.6.2.tar.gz
    cd R-3.6.2/
      mkdir ~/Rinst
    
    
    source /opt/intel/oneapi/setvars.sh
    
    export CC="icc"
    export CXX="icpc"
    export F77="ifort"
    export FC="ifort"
    export AR="xiar"
    export LD="xild"
    
    export CFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export CXXFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export FFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export FCFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export LDFLAGS="-lpthread -qopenmp-link=static -static-intel -no-intel-extensions" 
    
    
    MKL=" -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a ${MKLROOT}/lib/intel64/libmkl_intel_thread.a ${MKLROOT}/lib/intel64/libmkl_core.a /opt/intel/oneapi/compiler/2021.3.0/linux/compiler/lib/intel64_lin/libiomp5.a -Wl,--end-group -lpthread -lm -ldl -qopenmp-link=static"
    
    ./configure --prefix=/home/staff/myoung3/Rinst --enable-R-shlib --with-blas="$MKL" --with-lapack --enable-BLAS-shlib --with-x=no --disable-java
    make -j20
    make install
    

19. based on some errors in the log, specify icc for OBJC

    rm -rf ~/Rinst
    rm -rf R-3.6.2/
      tar zxf R-3.6.2.tar.gz
    cd R-3.6.2/
      mkdir ~/Rinst
    
    
    source /opt/intel/oneapi/setvars.sh
    
    export CC="icc -std=c99"
    export CXX="icpc"
    export OBJC="icc -std=c99"
    export F77="ifort"
    export FC="ifort"
    export AR="xiar"
    export LD="xild"
    
    export CFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export CXXFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export OBJCFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export FFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export FCFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export LDFLAGS="-lpthread -qopenmp-link=static -static-intel -no-intel-extensions" 
    
    
    MKL=" -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a ${MKLROOT}/lib/intel64/libmkl_intel_thread.a ${MKLROOT}/lib/intel64/libmkl_core.a /opt/intel/oneapi/compiler/2021.3.0/linux/compiler/lib/intel64_lin/libiomp5.a -Wl,--end-group -lpthread -lm -ldl -qopenmp-link=static"
    
    ./configure --prefix=/home/staff/myoung3/Rinst --enable-R-shlib --with-blas="$MKL" --with-lapack --enable-BLAS-shlib --with-x=no --disable-java
    make -j20
    make install
    

20. specify openmp flags. also turn off recommmended packages to speed up testing time

    rm -rf ~/Rinst
    rm -rf R-3.6.2/
      tar zxf R-3.6.2.tar.gz
    cd R-3.6.2/
      mkdir ~/Rinst
    
    
    source /opt/intel/oneapi/setvars.sh
    
    export CC="icc -std=c99"
    export CXX="icpc"
    export OBJC="icc -std=c99"
    export F77="ifort"
    export FC="ifort"
    export AR="xiar"
    export LD="xild"
    
    export CFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export CXXFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export OBJCFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export FFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export FCFLAGS="-fPIC -qopenmp-link=static -O3 -ipo -xHost -multiple-processes=20"
    export LDFLAGS="-lpthread -qopenmp-link=static -static-intel -no-intel-extensions" 
    export R_OPENMP_CFLAGS="-qopenmp-link=static"
    export SHLIB_OPENMP_CFLAGS="-qopenmp-link=static"
    export SHLIB_OPENMP_CXXFLAGS="-qopenmp-link=static"
    export SHLIB_OPENMP_FFLAGS="-qopenmp-link=static"
    
    
    MKL=" -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a ${MKLROOT}/lib/intel64/libmkl_intel_thread.a ${MKLROOT}/lib/intel64/libmkl_core.a /opt/intel/oneapi/compiler/2021.3.0/linux/compiler/lib/intel64_lin/libiomp5.a -Wl,--end-group -lpthread -lm -ldl -qopenmp-link=static"
    
    ./configure --prefix=/home/staff/myoung3/Rinst --enable-R-shlib --with-blas="$MKL" --with-lapack --enable-BLAS-shlib --with-x=no --disable-java --without-recommended-packages 
    make -j20
    make install
    

Still can't figure out static linking, so we'll need to go with dynamic.