[FIXED] Error while using of scipy.optimize.fmin_bfgs

Issue

My code look like this:

theta0 = np.array([0,0,0])
print(X.shape)
print(Y.shape)
print(theta0.shape)

(100, 2)
(100,)
(3,)

def logistic_func(theta, x):
    m, n = x.shape
    x = np.hstack((np.ones(x.shape[0]).reshape(x.shape[0], 1), x))
    s = np.dot(theta, x.T)
    s = np.where(np.abs(np.sum(theta*x, axis=1)) > 18, s, np.sign(s)*18)
    f = 1/(1+np.exp(-s))
    return f

def log_likelihood(theta, x, y, model):
    result = np.sum(y*np.log(model) + (1-y)*np.log(1-model))
    return result

def negative_log_likelihood(theta, x, y, model):
    return (-1)*log_likelihood(theta, x, y, model)

def log_likelihood_derivative(dtheta, x, y, model):    
    result = np.array(y - model(dtheta, x))
    x = np.hstack((np.ones(x.shape[0]).reshape(x.shape[0], 1), x))
    result = np.reshape(result, (-1, 1))
    result = np.sum(result*x, axis=0) 
    assert result.shape == theta.shape
    return result

def negative_log_likelihood_derivative(theta, x, y, model):
    return -log_likelihood_derivative(theta, x, y, model)

I am not able to resolve the problem by myself.

When I type:

model = logistic_func
theta_opt = so.fmin_bfgs(negative_log_likelihood, theta0, 
                         fprime=negative_log_likelihood_derivative,
                         args=(X, Y, model), disp=True)

I get:

TypeError: loop of ufunc does not support argument 0 of type function which has no callable log method

Solution

From the docs:

Parameters

f callable f(x,*args)
     Objective function to be minimized.

x0 ndarray
    Initial guess.

fprime callable f'(x,*args), optional
    Gradient of f.

Callable means that a function must be passed as a parameter. You, however, pass a return value of that function, negative_log_likelihood_derivative(theta0, X, Y, model) (note the round brackets that are used to call the function!). Instead you need to pass the function itself:

theta_opt = so.fmin_bfgs(negative_log_likelihood_derivative, theta0, 
                         fprime=negative_log_likelihood_derivative, 
                         args=(X,Y), disp=True)

Side notes:

1. You import the function by name; why do you call it from the so namespace?
2. You have the same functions given for f and fprime; it's usually incorrect unless f = np.exp(x)

Answered By - Andrey Sobolev