[FIXED] Tensorflow 2.4.1 and R 4.1.1 conflict in Anaconda

Issue

Is it possible to install both the latest versions of Tensorflow (2.4.1 as of 8/2021) and R (4.1.1) in Anaconda?

I'm trying to create a Docker image using the Jupyter Spark base image that has both of these. A minimal dockerfile to do this is

FROM jupyter/pyspark-notebook:notebook-6.4.2

USER root

ENV R_LIBS_USER $SPARK_HOME/R/lib
RUN fix-permissions $R_LIBS_USER

USER $NB_UID

RUN mamba install --quiet --yes \
    'tensorflow=2.4.1' \
    'r-base=4.1.1'

Running this with docker build . -f "minimal.dockerfile" -t minimal:latest results in an error:

 => [internal] load build definition from minimal.dockerfile                       0.1s 
 => => transferring dockerfile: 426B                                               0.0s 
 => [internal] load .dockerignore                                                  0.0s 
 => => transferring context: 2B                                                    0.0s 
 => [internal] load metadata for docker.io/jupyter/pyspark-notebook:notebook-6.4.  3.0s 
 => [auth] jupyter/pyspark-notebook:pull token for registry-1.docker.io            0.0s 
 => [1/3] FROM docker.io/jupyter/pyspark-notebook:notebook-6.4.2@sha256:a9a92d02f  0.0s 
 => CACHED [2/3] RUN fix-permissions /usr/local/spark/R/lib                        0.0s 
 => ERROR [3/3] RUN mamba install --quiet --yes     'tensorflow=2.4.1'     'r-ba  33.8s 
------
 > [3/3] RUN mamba install --quiet --yes     'tensorflow=2.4.1'     'r-base=4.1.1':     
#7 33.19 Encountered problems while solving:
#7 33.19   - package arrow-cpp-5.0.0-py39he92c077_3_cpu requires libprotobuf >=3.16.0,<3.17.0a0, but none of the providers can be installed
#7 33.19 
------
executor failed running [/bin/bash -o pipefail -c mamba install --quiet --yes     'tensorflow=2.4.1'     'r-base=4.1.1']: exit code: 1

If I run just mamba install tensorflow=2.4.1, I can see that it downgrades arrow-cpp and libprotobuf, among others. Is there a way to override this behaviour?

Other things I've tried:

• If I run mamba install tensorflow followed by mamba install r-base, I get the same error as above
• If I run mamba install r-base followed by mamba install tensorflow, I end up with R 3.6, not 4.1.1

Solution

Don't try to install everything in a single environment. Create a Python environment and create an R environment, and make sure they each have their respective kernel packages (ipykernel, r-irkernel) so that they can be used in Jupyter.

Answered By - merv