[FIXED] Pickle File not found while loading

Issue

I have saved a classifier model with pickle using the following code-

import pickle
with open('skm.pickle', 'wb') as fid:
    pickle.dump(clf, fid) 

This is the score.py file where I am loading the pickle model and the same file is called during deployment. Also note that, all these files (code, pickle file and related files) are in the same directory.

%%writefile sklearnscore.py

import json
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
from sklearn.ensemble import RandomForestClassifier
import pickle

# Initialize the deployment environment
def init():
    # read in the model file
    from sklearn.pipeline import Pipeline
    global obj
    with open('./skm.pickle', 'rb') as f:
        obj = pickle.load(f)

This is getting stored in some directory var/azureml-app

I am registering the model using- model = Model.register(ws, model_name="utility15", model_path="./skm.pickle")

And the deployment code is-

service_name = 'my-custom-env-service-4'
sklearn_env = Environment.from_conda_specification(name='sklearn-env', file_path='Sklearn.yaml')

inference_config = InferenceConfig(entry_script='sklearnscore.py', environment=sklearn_env)

aci_config = AciWebservice.deploy_configuration(cpu_cores=1, memory_gb=4,tags={'Createdby':'xyz'})

service = Model.deploy(workspace=ws,
                        name=service_name,
                        models=[model],
                        inference_config=inference_config,
                        deployment_config=aci_config,
                        overwrite=True)
service.wait_for_deployment(show_output=True)

After running this deployment code, this is the error I am getting-

2022-05-24 09:58:08,198 | root | INFO | Starting up app insights client
logging socket was found. logging is available.
logging socket was found. logging is available.
2022-05-24 09:58:08,199 | root | INFO | Starting up request id generator
2022-05-24 09:58:08,199 | root | INFO | Starting up app insight hooks
2022-05-24 09:58:08,199 | root | INFO | Invoking user's init function
2022-05-24 09:58:08,201 | root | ERROR | User's init function failed
2022-05-24 09:58:08,203 | root | ERROR | Encountered Exception Traceback (most recent call last):
  File "/var/azureml-server/aml_blueprint.py", line 191, in register
    main.init()
  File "/var/azureml-app/sklearnscore.py", line 11, in init
    with open('/mnt/batch/tasks/shared/LS_root/mounts/clusters/lowcomputeamber/code/Users/1964821/skm.pickle', 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/batch/tasks/shared/LS_root/mounts/clusters/lowcomputeamber/code/Users/1964821/skm.pickle'

2022-05-24 09:58:08,204 | root | INFO | Waiting for logs to be sent to Application Insights before exit.
2022-05-24 09:58:08,205 | root | INFO | Waiting 30 seconds for upload.
Worker exiting (pid: 103)
Shutting down: Master
Reason: Worker failed to boot.
2022-05-24T09:58:38,514243200+00:00 - gunicorn/finish 3 0
2022-05-24T09:58:38,516894700+00:00 - Exit code 3 is not normal. Killing image.

From this error, I can see that sklearnscore.py is at /var/azureml-app/sklearnscore.py and so, it might be expecting the pickle model to be there. But how do I save my pickle model on the path var/azureml-app since the path is not accesible?

Solution

I figured out the way to solve this error.

Instead of loading the model as

 with open('./skm.pickle', 'rb') as f:
        obj = pickle.load(f)

You need to load the model as

 model_path = os.path.join(os.getenv('AZUREML_MODEL_DIR'), 'skm.pickle')
 model = joblib.load(model_path)

The AZUREML_MODEL_DIR will take the environment directory of the model where it is stored.

This will solve the error.

Answered By - PeakyBlinder