[FIXED] Sklearn: Calibrate a multi-label classification with CalibratedClassifierCV

Issue

I have built a number of sklearn classifier models to perform multi-label classification and I would like to calibrate their predict_proba outputs so that I can obtain confidence scores. I would also like to use metrics such as sklearn.metrics.recall_score to evaluate them.

I have 4 labels to predict and the true labels are multi-hot encoded (e.g. [0, 1, 1, 1]). As a result, CalibratedClassifierCV does not directly accept my data:

clf = tree.DecisionTreeClassifier(max_depth=15)
clf = clf.fit(train_X, train_Y)
calibrated_clf = CalibratedClassifierCV(clf, cv="prefit", method="sigmoid")
calibrated_clf.fit(dev_X, dev_Y)

This would return an error:

ValueError: classes [[0 1]
 [0 1]
 [0 1]
 [0 1]] mismatch with the labels [0 1 2 3] found in the data

Thus, I tried to wrap it in a OneVsRestClassifier:

clf = OneVsRestClassifier(tree.DecisionTreeClassifier(max_depth=15), n_jobs=4)
clf = clf.fit(train_X, train_Y)
calibrated_clf = CalibratedClassifierCV(clf, cv="prefit", method="sigmoid")
calibrated_clf.fit(dev_X, dev_Y)

Note that MultiOutputClassifier and ClassifierChain do not work even though they possibly suit my problem better.

It works, but the predict output of the calibrated classifier is multi-class instead of multi-label because of its implementation. There are four classes ([0 1 2 3]) but if there is no need to put a label, it still predicts a 0.

Upon further inspection by means of calibration curves, it turns out the base estimator wrapped inside the calibrated classifier is not calibrated at all. That is, (calibrated_clf.calibrated_classifiers_)[0].base_estimator returns the same clf as before calibration.

I would like to observe the performance of my (calibrated) models doing deterministic (predict) and probabilistic (predict_proba) predictions. How should I design my model/wrap things in other containers to get both calibrated probabilities for each label and comprehensible label predictions?

Solution

In your example, you're using a DecisionTreeClassifier which by default support targets of dimension (n, m) where m > 1.

However if you want to have as result the marginal probability of each class then use the OneVsRestClassifier.

Notice that CalibratedClassifierCV expects target to be 1d so the "trick" is to extend it to support Multilabel Classification with MultiOutputClassifier.

Full Example

from sklearn.datasets import make_multilabel_classification
from sklearn.tree import DecisionTreeClassifier
from sklearn.multioutput import MultiOutputClassifier
from sklearn.model_selection import train_test_split, StratifiedKFold

import numpy as np

# Generate a sample multilabel target
X, y = make_multilabel_classification(n_classes=4, random_state=0)

y
>>> 
array([[1, 0, 1, 0],
       [0, 0, 0, 0],
       [1, 0, 1, 0],
       ...
       [0, 0, 0, 0],
       [0, 1, 1, 1],
       [1, 1, 0, 1]])

# Split in train/test
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.9, random_state=42
)

# Splits Stratify target variable
cv = StratifiedKFold(n_splits=2)

# Decision tree support by default multiclass target or use OneVsRest if marginal probabilities
clf = OneVsRestClassifier(DecisionTreeClassifier(max_depth=10))

# Calibrate estimator probabilities
calibrated_clf = CalibratedClassifierCV(base_estimator=clf, cv=cv)

# calibrated_clf target is one dimensional, extend classifier to multi-target classification.
multioutput_clf = MultiOutputClassifier(calibrated_clf).fit(X_train, y_train)

# Check predict 
multioutput_clf.predict(X_test[-5:])
>>> 
array([[0, 0, 1, 1],
       [0, 0, 0, 1],
       [0, 0, 0, 1],
       [0, 0, 0, 1],
       [0, 0, 0, 1]])

# Check predict_proba
multioutput_clf.predict_proba(X_test[-5:])
>>>
[array([[0.78333315, 0.21666685],
        [0.78333315, 0.21666685],
        [0.78333315, 0.21666685],
        [0.78333315, 0.21666685],
        [0.78333315, 0.21666685]]),
 array([[0.59166537, 0.40833463],
        [0.59166537, 0.40833463],
        [0.40833361, 0.59166639],
        [0.59166537, 0.40833463],
        [0.59166537, 0.40833463]]),
 array([[0.61666922, 0.38333078],
        [0.61666427, 0.38333573],
        [0.80000098, 0.19999902],
        [0.61666427, 0.38333573],
        [0.61666427, 0.38333573]]),
 array([[0.26874774, 0.73125226],
        [0.26874774, 0.73125226],
        [0.45208444, 0.54791556],
        [0.26874774, 0.73125226],
        [0.26874774, 0.73125226]])]

Notice that the result from predict_proba is a list with 4 arrays, each array is the probability to belong to the class i. For example, inside the first sample of the first array is the probability that first sample belongs to class 1 and so on.

Regarding the calibration curves, scikit-learn provides examples to plot probability path for two dimension and three dimension targets.

Answered By - Miguel Trejo