[FIXED] Neural network always predicts the same class

Issue

I'm trying to implement a neural network that classifies images into one of the two discrete categories. The problem is, however, that it currently always predicts 0 for any input and I'm not really sure why.

Here's my feature extraction method:

def extract(file):
    # Resize and subtract mean pixel
    img = cv2.resize(cv2.imread(file), (224, 224)).astype(np.float32)
    img[:, :, 0] -= 103.939
    img[:, :, 1] -= 116.779
    img[:, :, 2] -= 123.68
    # Normalize features
    img = (img.flatten() - np.mean(img)) / np.std(img)

    return np.array([img])

Here's my gradient descent routine:

def fit(x, y, t1, t2):
    """Training routine"""
    ils = x.shape[1] if len(x.shape) > 1 else 1
    labels = len(set(y))

    if t1 is None or t2 is None:
        t1 = randweights(ils, 10)
        t2 = randweights(10, labels)

    params = np.concatenate([t1.reshape(-1), t2.reshape(-1)])
    res = grad(params, ils, 10, labels, x, y)
    params -= 0.1 * res

    return unpack(params, ils, 10, labels)

Here are my forward and back(gradient) propagations:

def forward(x, theta1, theta2):
    """Forward propagation"""

    m = x.shape[0]

    # Forward prop
    a1 = np.vstack((np.ones([1, m]), x.T))
    z2 = np.dot(theta1, a1)

    a2 = np.vstack((np.ones([1, m]), sigmoid(z2)))
    a3 = sigmoid(np.dot(theta2, a2))

    return (a1, a2, a3, z2, m)

def grad(params, ils, hls, labels, x, Y, lmbda=0.01):
    """Compute gradient for hypothesis Theta"""

    theta1, theta2 = unpack(params, ils, hls, labels)

    a1, a2, a3, z2, m = forward(x, theta1, theta2)
    d3 = a3 - Y.T
    print('Current error: {}'.format(np.mean(np.abs(d3))))

    d2 = np.dot(theta2.T, d3) * (np.vstack([np.ones([1, m]), sigmoid_prime(z2)]))
    d3 = d3.T
    d2 = d2[1:, :].T

    t1_grad = np.dot(d2.T, a1.T)
    t2_grad = np.dot(d3.T, a2.T)

    theta1[0] = np.zeros([1, theta1.shape[1]])
    theta2[0] = np.zeros([1, theta2.shape[1]])

    t1_grad = t1_grad + (lmbda / m) * theta1
    t2_grad = t2_grad + (lmbda / m) * theta2

    return np.concatenate([t1_grad.reshape(-1), t2_grad.reshape(-1)])

And here's my prediction function:

def predict(theta1, theta2, x):
    """Predict output using learned weights"""
    m = x.shape[0]

    h1 = sigmoid(np.hstack((np.ones([m, 1]), x)).dot(theta1.T))
    h2 = sigmoid(np.hstack((np.ones([m, 1]), h1)).dot(theta2.T))

    return h2.argmax(axis=1)

I can see that the error rate is gradually decreasing with each iteration, generally converging somewhere around 1.26e-05.

What I've tried so far:

1. PCA
2. Different datasets (Iris from sklearn and handwritten numbers from Coursera ML course, achieving about 95% accuracy on both). However, both of those were processed in a batch, so I can assume that my general implementation is correct, but there is something wrong with either how I extract features, or how I train the classifier.
3. Tried sklearn's SGDClassifier and it didn't perform much better, giving me a ~50% accuracy. So something wrong with the features, then?

Edit: An average output of h2 looks like the following:

[0.5004899   0.45264441]
[0.50048522  0.47439413]
[0.50049019  0.46557124]
[0.50049261  0.45297816]

So, very similar sigmoid outputs for all validation examples.

Solution

After a week and a half of research I think I understand what the issue is. There is nothing wrong with the code itself. The only two issues that prevent my implementation from classifying successfully are time spent learning and proper selection of learning rate / regularization parameters.

I've had the learning routine running for some tome now, and it's pushing 75% accuracy already, though there is still plenty of space for improvement.

Answered By - Yurii Dolhikh