Open
Description
Describe the bug
Uncertain if this is a bug or counter-intuitive expected behavior.
Under certain circumstances the ROC AUC calculated for SVC with the sigmoid kernel will not agree depending on if you use predict_proba or decision_function. In fact, they will be nearly 1-other_method_auc.
This was noticed when comparing ROC AUC calculated using roc_auc_score with predictions from predict_proba(X)[:, 1] to using the scorer from get_scorer('roc_auc') which appears to be calling roc_auc_score with scores from decision_function.
Steps/Code to Reproduce
import numpy as np
from sklearn.svm import SVC
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score, get_scorer
from sklearn.model_selection import train_test_split
n_samples = 100
n_features = 100
random_state = 123
rng = np.random.default_rng(random_state)
X = rng.normal(loc=0.0, scale=1.0, size=(n_samples, n_features))
y = rng.integers(0, 2, size=n_samples)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=random_state)
svc_params = {
"kernel": "sigmoid",
"probability": True,
"random_state":random_state,
}
pipeline = Pipeline([
('scaler', StandardScaler()),
('svc', SVC(**svc_params))
])
pipeline.fit(X_train, y_train)
y_proba = pipeline.predict_proba(X_test)[:, 1]
y_dec = pipeline.decision_function(X_test)
roc_auc_proba = roc_auc_score(y_test, y_proba)
roc_auc_dec = roc_auc_score(y_test, y_dec)
auc_scorer = get_scorer('roc_auc')
scorer_auc = auc_scorer(pipeline, X_test, y_test)
print(f"AUC (roc_auc_score from predict_proba) = {roc_auc_proba:.4f}")
print(f"AUC (roc_auc_score from decision_function) = {roc_auc_dec:.4f}")
print(f"AUC (get_scorer) = {scorer_auc:.4f}")Expected Results
The measures of ROC AUC agree
Actual Results
AUC (roc_auc_score from predict_proba) = 0.5833
AUC (roc_auc_score from decision_function) = 0.4295
AUC (get_scorer) = 0.4295Versions
System:
python: 3.11.5
Python dependencies:
sklearn: 1.7.dev0
pip: 25.0.1
setuptools: 65.5.0
numpy: 1.26.4
scipy: 1.15.2
Cython: 3.0.12
pandas: 2.2.3
matplotlib: 3.10.1
joblib: 1.2.0
threadpoolctl: 3.1.0
Built with OpenMP: True