[FIXED] sklearn can't find lapack in new conda environment

Issue

Question:

Why can't sklearn detect its liblapack dependency and how do I fix this?

Background:

I the following conda environment (torch-nightly) with scikit-learn installed (note that the liblapack dependency also exists):

(torch-nightly) au@a2 ~ % conda list |grep "scikit-learn"
scikit-learn              1.1.1            py38hc377ac9_0  
(torch-nightly) au@a2 ~ % conda list |grep "liblapack"
liblapack                 3.9.0           15_osxarm64_openblas    conda-forge

Problem:

I get the following error when attempting to import sklearn:

(torch-nightly) au@a2 ~ % python -c "import sklearn"                                                              
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/au/miniconda3/envs/torch-nightly/lib/python3.8/site-packages/sklearn/__init__.py", line 82, in <module>
    from .base import clone
  File "/Users/au/miniconda3/envs/torch-nightly/lib/python3.8/site-packages/sklearn/base.py", line 17, in <module>
    from .utils import _IS_32BIT
  File "/Users/au/miniconda3/envs/torch-nightly/lib/python3.8/site-packages/sklearn/utils/__init__.py", line 22, in <module>
    from scipy.sparse import issparse
  File "/Users/au/miniconda3/envs/torch-nightly/lib/python3.8/site-packages/scipy/sparse/__init__.py", line 283, in <module>
    from . import csgraph
  File "/Users/au/miniconda3/envs/torch-nightly/lib/python3.8/site-packages/scipy/sparse/csgraph/__init__.py", line 182, in <module>
    from ._laplacian import laplacian
  File "/Users/au/miniconda3/envs/torch-nightly/lib/python3.8/site-packages/scipy/sparse/csgraph/_laplacian.py", line 7, in <module>
    from scipy.sparse.linalg import LinearOperator
  File "/Users/au/miniconda3/envs/torch-nightly/lib/python3.8/site-packages/scipy/sparse/linalg/__init__.py", line 120, in <module>
    from ._isolve import *
  File "/Users/au/miniconda3/envs/torch-nightly/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/__init__.py", line 4, in <module>
    from .iterative import *
  File "/Users/au/miniconda3/envs/torch-nightly/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/iterative.py", line 9, in <module>
    from . import _iterative
ImportError: dlopen(/Users/au/miniconda3/envs/torch-nightly/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/_iterative.cpython-38-darwin.so, 0x0002): Library not loaded: '@rpath/liblapack.3.dylib'
  Referenced from: '/Users/au/miniconda3/envs/torch-nightly/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/_iterative.cpython-38-darwin.so'
  Reason: tried: '/Users/au/miniconda3/envs/torch-nightly/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/../../../../../../liblapack.3.dylib' (no such file), '/Users/au/miniconda3/envs/torch-nightly/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/../../../../../../liblapack.3.dylib' (no such file), '/Users/au/miniconda3/envs/torch-nightly/bin/../lib/liblapack.3.dylib' (no such file), '/Users/au/miniconda3/envs/torch-nightly/bin/../lib/liblapack.3.dylib' (no such file), '/usr/local/lib/liblapack.3.dylib' (no such file), '/usr/lib/liblapack.3.dylib' (no such file)

Research on Problem:

I confirmed that the "missing" library (liblapack.3.dylib) actually exists at the location shown in the error:

(torch-nightly) au@a2 ~ % ls miniconda3/envs/torch-nightly/lib/liblapack.3.dylib
miniconda3/envs/torch-nightly/lib/liblapack.3.dylib

Solution

NOTE: This answer is for macOS users only, and involves copying dilyb(s) that may not be compatible all versions of sklearn. I highly recommend trying to re-install the packages first. If you continue to fail and have reached a point where you are willing to try anything, the following steps may resolve the issue.

I was able to resolve this by manually copying over a copy of the necessary dylib(s) out of a homebrew installation. First: I ran brew install lapack [NOTE: For me, this installed version (3.11), you may end up with a different version after running the command] Then I identified all the relevant files in of /opt/homebrew/Cellar/lapack/3.11/lib/*:

liblapacke.dylib
libblas.3.11.0.dylib
libblas.3.dylib
libblas.dylib
liblapack.3.11.0.dylib
liblapack.3.dylib
liblapack.dylib
liblapacke.3.11.0.dylib
liblapacke.3.dylib

Identify a search path that I all my python packages would search. I'm using miniconda for my python environment management, for me it was: ~/miniconda3/envs/maxEnt/lib/.

Step 1: Delete all existing files with a matching name in this lib folder.

Step 2: Copy the files referenced above.

Step 3: Update all the symlinks (aliases) in these files [.dylib points to 3.11.0.dylib; .3.dylib points to 3.11.0.dylib].

Answered By - Francesco A S Mikulis-Borsoi