[FIXED] Difference between roc_auc_score and cross_val_score(scoring=roc_auc)

Issue

I'm learning machine learning and am not so clear about this. I saw similar post in Stack Overflow but I may need your help little bit more to understand.

Code 1

kfold = model_selection.KFold(n_splits=10, random_state=7, shuffle=True)
lrCV = LogisticRegression(class_weight='balanced')
scoring = 'roc_auc'
lr_results = model_selection.cross_val_score(lrCV, X_train, y_train, cv=kfold, scoring=scoring)

Result

array([0.91374269, 0.70209059, 0.89164087, 0.8021978 , 0.85077519,
   0.80888889, 0.79338843, 0.76446281, 0.84803002, 0.74506579])

Code 2

lrmodel = LogisticRegression(class_weight='balanced')
lrmodel.fit(X_train, y_train)
lr_auc = roc_auc_score(y_test, lrmodel.predict(X_test))

Result

Logistic Regression AUC = 0.67

Obviously Code 2 is a lot lower than Code 1. What is causing this difference? I'm new to ML and self-learning. Please let me know little bit more about this.

Solution

First of all, both models are calculating the AUC exactly the same way. Both instances use the sklearn roc_auc_score.

Code 1: https://scikit-learn.org/stable/modules/model_evaluation.html

Code 2: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html

For Code 1, if you press the link next to roc_auc, you will end up on the same page as for Code 2.

I think the difference is simply due to the fact that for Code 1, you are using K-fold CV with shuffling. While for Code 2, you are using a train-test split. I don't know any specifics about your data at hand but since there is such a large discrepancy, what is your sample size? If your sample size is quite small, you could be witnessing a bias due to your training-test splits.

Answered By - Fred