[FIXED] Put input in a tensorflow neural network

Issue

I precoded a neural network using tensorflow with the MNIST dataset, I have also created an input system with pygame, however I do not know how to implement my data to the neural network that I have created.

screenshot of my work:

my two files of code:

import tensorflow as tf
import matplotlib.pyplot as plt
import numpy as np

mnist=tf.keras.datasets.mnist

(x_train, y_train), (x_test, y_test) = mnist.load_data()

x_train=tf.keras.utils.normalize(x_train, axis=1)
x_test=tf.keras.utils.normalize(x_test, axis=1)

model=tf.keras.models.Sequential()
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(128, activation=tf.nn.relu))
model.add(tf.keras.layers.Dense(128, activation=tf.nn.relu))
model.add(tf.keras.layers.Dense(10, activation=tf.nn.softmax))

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])
model.fit(x_train, y_train, epochs=3)

val_loss, val_acc = model.evaluate(x_test, y_test)
print(val_loss, val_acc)

model.save("num_reader.model")
new_model=tf.keras.models.load_model('num_reader.model')
predictions=new_model.predict([x_test])

import pygame
import sys
from main import *

class Screen:
    def __init__(self):
        pygame.init()
        #self.screen=pygame.display.set_mode((28,28),pygame.FULLSCREEN)
        self.screen=pygame.display.set_mode((280,280))
        self.array=[]
        self.setArr()
        self.bg_color=(250, 250,250)
        self.ok=False
        self.full=[]

    def runGame(self):
        self.screen.fill(self.bg_color)
        while True:
            pygame.display.flip()
            self._check_events()
            self.draw()

    def _check_events(self):
        for event in pygame.event.get():
            if event.type==pygame.QUIT:
                sys.exit()
            if event.type==pygame.KEYDOWN:
                if event.key==pygame.K_ESCAPE:
                    sys.exit()
                if event.key==pygame.K_d:
                    self.decode()
                    print(len(self.full))
                if event.key==pygame.K_c:
                    self.setArr()
                    self.screen.fill(self.bg_color)
            if event.type==pygame.MOUSEBUTTONDOWN:
                #print("mouseDown")
                self.ok=True
            elif event.type==pygame.MOUSEBUTTONUP:
                self.ok=False

    def setArr(self):
        self.shortArr=[]
        for y in range(28):
            self.shortArr.append(0)
        for x in range(28):
            self.array.append(self.shortArr)

    def draw(self):
        if self.ok==True:
            x,y=pygame.mouse.get_pos()
            x=round(x/10)*10
            y=round(y/10)*10

            #print(x,y)
            #print(self.array[int(x)//10][int(y)//10])
            self.array[int(x)//10][int(y)//10]=1

            pygame.draw.rect(self.screen, (0,0,0), pygame.Rect(x, y, 10, 10))
            #print("draw")

    def decode(self):
        self.full=[]
        for x in range(28):
            for y in range(28):
                self.full.append(self.array[x][y])

if __name__ == '__main__':
      Sc=Screen()
      Sc.runGame()

Solution

I believe you want to edit the game's runGame function to stop the game (without stopping the script) when the user is done drawing and the image has been decoded (assuming the user presses 'd' when they have drawn everything they want to - correct me if I'm wrong). You will then want to get that finished array, convert it to a NumPy array and pass it to your neural network's predict function.

runGame which now stops the game when the array is finished:

def runGame(self):
    self.screen.fill(self.bg_color)
    while True:
        pygame.display.flip()
        self._check_events()
        self.draw()
        if self.full != []:
            break

Modified main code, which gets the array from the game, converts it to a numpy array, normalizes it, and calls your neural network's predict function on it:

if __name__ == '__main__':
    Sc=Screen()
    Sc.runGame()
    result = nn.new_model.predict(tf.keras.utils.normalize(np.array(Sc.full), axis=1))
    print(result)

You will also need to add this to the top of your code:

import main as nn

This imports your main neural network script into your screen.py script, allowing you to call the function. This only works if both scripts are in the same directory, and that main.py stays named that. If you change the file name, change the import too.

NOTE: All of the changes above are made to screen.py.

DISCLAIMER: I am not entirely sure of the shape of input your neural network is looking for, but this should work (you flatten everything first, which means it should all end up the same either way). If there are errors though, let me know.

Answered By - Finn E