I must be missing something, our SGDRegressor doesn't have a C param.

BaseSGDClassifier has an attibute C. This PR exposes this attribute to SGDClassifier (same for regressors).

I don't really like that. Proposition: I revert the exposition of C and write in the deprecation note, using somewhat private API:

clf = SGDClassifier(
    penalty=None,
    alpha=1.0,
    eta0=1.0,
    learning_rate="pa1",
    loss="hinge",
)
clf.C = 1.0  # THIS IS USING PRIVATE API

See c24c95f. I can revert if you dislike it.

If we're deprecating these estimators and telling users to use SGDRegressor and SGDClassifier instead, it makes sense to actually have a public way of doing that. So I'm in favor of exposing C in both SGD estimators, document them, and then here simply create and equivalent estimator using them.

How about leaving them private for the time being?

I would like to have a dedicated discussion whether to

• expose C in SGD; xor
• remove C and all passive aggressive things altogether

I think this PR is the right place to have that discussion. I'd like others to weigh in about this point.

maybe @ogrisel @amueller @GaelVaroquaux @adam2392 would have an opinion?

I'm not super familiar with passive aggressive vs SGD but a loose read suggests passive aggressive allows more online updating. Though i thought SGD easily allows online training too, so why would someone use one over the other?

Is it true that the only parameter that makes these different is "C" step size parameter?

Pragmatically: if no one uses "C", then i kinda agree might be simplest to just remove it? If there's something I'm missing tho, lmk.

PassiveAggressiveClassifier was first implemented in #1259 in 2012. I have never seen it used anywhere and I think our SGD... offer enough for online learning (partial_fit). So I would be fine with removing it completely, i.e. also removing C in SGD. Unless someone steps in for keeping it.

Having read through the code and partially papers, I could also live with making learning_rate="pa1" and "pa2" as well as C public in SGD.... In that case I would rename C to something like pa_C.

I am partially in favor of just simplification and deprecating those parameters / the PAClf altogether, unless someone has an actual use case for them.