[FIXED] How do you show cost function per iteration in scikit-learn?

Issue

I've been running some linear/logistic regression models recently, and I wanted to know how you can output the cost function for each iteration. One of the parameters in sci-kit LinearRegression is 'maxiter', but in reality you need to see cost vs iteration to find out what this value really needs to be i.e. is the benefit worth the computational time to run more iterations etc

I'm sure I'm missing something but I would have thought there was a method that outputted this information?

Thanks in advance!

Solution

One has to understand if there is any iteration (implying computing a cost function) or an analytical exact solution, when fitting any estimator.

Linear Regression

In fact, Linear Regression - ie Minimization of the Ordinary Least Square - is not an algorithm but a minimization problem that can be solved using different techniques. And those techniques

Not getting into the details of the statistical part described here :

There are at least three methods used in practice for computing least-squares solutions: the normal equations, QR decomposition, and singular value decomposition.

As far as I got into the details of the codes, it seems that the computational time is involved by getting the analytical exact solution, not iterating over the cost function. But I bet they depend on your system being under-, well- or over-determined, as well as the language and library you are using.

Logistic Regression

As Linear Regression, Logistic Regression is a minimization problem that can be solved using different techniques that, for scikit-learn, are : newton-cg, lbfgs, liblinear and sag. As you mentionned, sklearn.linear_model.LogisticRegression includes the max_iter argument, meaning it includes iterations*. Those are controled either because the updated argument doesn't change anymore - up to a certain epsilon value - or because it reached the maximum number of iterations.

*As mentionned in the doc, it includes iterations only for some of the solvers

Useful only for the newton-cg, sag and lbfgs solvers. Maximum number of iterations taken for the solvers to converge.

In fact, each solver involves its own implementation, such as here for the liblinear solver.

I would recommand to use the verbose argument, maybe equal to 2 or 3 to get the maximum value. Depending on the solver, it might print the cost function error. However, I don't understand how you are planning to use this information.

Another solution might be to code your own solver and print the cost function at each iteration.

---

Curiosity kills cat but I checked the source code of scikit which involves many more.

1. First, sklearn.linear_model.LinearRegression use a fit to train its parameters.

2. Then, in the source code of fit, they use the Ordinary Least Square of Numpy (source).

3. Finally, Numpy's Least Square function uses the functionscipy.linalg.lapack.dgelsd, a wrapper to the LAPACK (Linear Algebra PACKage) function DGELSD written in Fortran (source).

That is to say that getting into the error calculation, if any, is not easy for scikit-learn developers. However, for the various using of LinearRegression and many more I had, the trade-off between cost-function and iteration time is well-adressed.

Answered By - Igor OA