[FIXED] StellarGraph PaddedGraphGenerator - how to provide specific training, validation and test sets

Issue

I'm trying to train a basic Graph Neural Network using the StellarGraph library, in particular starting from the example provided in [0].

The example works fine, but now I would like to repeat the same exercize removing the N-Fold Crossvalidation and providing specific training, validation and test sets. I'm trying to do so with the following code:

# One hot encoding
graph_training_set_labels_encoded = pd.get_dummies(graphs_training_set_labels, drop_first=True)
graph_validation_set_labels_encoded = pd.get_dummies(graphs_validation_set_labels, drop_first=True)

graphs = graphs_training_set + graphs_validation_set

# Graph generator preparation
generator = PaddedGraphGenerator(graphs=graphs)

train_gen = generator.flow([x for x in range(0, len(graphs_training_set))],
                           targets=graph_training_set_labels_encoded,
                           batch_size=batch_size)

valid_gen = generator.flow([x for x in range(len(graphs_training_set),
                                            len(graphs_training_set) + len(graphs_validation_set))],
                          targets=graph_validation_set_labels_encoded,
                          batch_size=batch_size)

# Stopping criterium
es = EarlyStopping(monitor="val_loss",
                   min_delta=0,
                   patience=20,
                   restore_best_weights=True)

# Model definition
gc_model = GCNSupervisedGraphClassification(layer_sizes=[64, 64],
                                            activations=["relu", "relu"],
                                            generator=generator,
                                            dropout=dropout_value)

x_inp, x_out = gc_model.in_out_tensors()
predictions = Dense(units=32, activation="relu")(x_out)
predictions = Dense(units=16, activation="relu")(predictions)
predictions = Dense(units=1, activation="sigmoid")(predictions)

# Creating Keras model and preparing it for training
model = Model(inputs=x_inp, outputs=predictions)
model.compile(optimizer=Adam(adam_value), loss=binary_crossentropy, metrics=["acc"])

# GNN Training
history = model.fit(train_gen, epochs=num_epochs, validation_data=valid_gen, verbose=0, callbacks=[es])

# Calculate performance on the validation data
test_metrics = model.evaluate(valid_gen, verbose=0)
valid_acc = test_metrics[model.metrics_names.index("acc")]

print(f"Test Accuracy model = {valid_acc}")

Where graphs_training_set and graphs_validation_set are lists of StellarDiGraphs.

I am able to run this piece of code, but it provides NaN as result. What could be the problem?

Since it is the first time I am using StellarGraph and in particular PaddedGraphGenerator. I think my mistake rely on the usage of that generator, but providing training set and validation set in different manner didn't produce better results.

Thank you in advance.

UPDATE Fixed I typo in the code, as pointed out here (thanks to george123).

[0] https://stellargraph.readthedocs.io/en/stable/demos/graph-classification/gcn-supervised-graph-classification.html

Solution

I found a solution digging in the StellarGraph documentation for PaddedGraphGenerator and GCN Neural Network Class GCNSupervisedGraphClassification. Furthermore, I have found a similar question on StellarGraph Issue Tracker which also points out to the solution.

# Graph generator preparation
generator = PaddedGraphGenerator(graphs=graphs)

train_gen = generator.flow([x for x in range(0, num_graphs_for_training)],
                            targets=training_graphs_labels,
                            batch_size=35)
valid_gen = generator.flow([x for x in range(num_graphs_for_training, num_graphs_for_training + num_graphs_for_validation)],
                            targets=validation_graphs_labels,
                            batch_size=35)

# Stopping criterium
es = EarlyStopping(monitor="val_loss",
                    min_delta=0.001,
                    patience=10,
                    restore_best_weights=True)

# Model definition
gc_model = GCNSupervisedGraphClassification(layer_sizes=[64, 64],
                                            activations=["relu", "relu"],
                                            generator=generator,
                                            dropout=dropout_value)

x_inp, x_out = gc_model.in_out_tensors()
predictions = Dense(units=32, activation="relu")(x_out)
predictions = Dense(units=16, activation="relu")(predictions)
predictions = Dense(units=1, activation="sigmoid")(predictions)

# Let's create the Keras model and prepare it for training
model = Model(inputs=x_inp, outputs=predictions)
model.compile(optimizer=Adam(adam_value), loss=binary_crossentropy, metrics=["acc"])

# GNN Training
history = model.fit(train_gen, epochs=num_epochs, validation_data=valid_gen, verbose=1, callbacks=[es])

# Evaluate performance on the validation data
valid_metrics = model.evaluate(valid_gen, verbose=0)
valid_acc = valid_metrics[model.metrics_names.index("acc")]

# Define test set indices temporary vars
index_begin_test_set = num_graphs_for_training + num_graphs_for_validation
index_end_test_set = index_begin_test_set + num_graphs_for_testing

test_set_indices = [x for x in range(index_begin_test_set, index_end_test_set)]

# Evaluate performance on test set
generator_for_test_set = PaddedGraphGenerator(graphs=graphs)
test_gen = generator_for_test_set.flow(test_set_indices)
result = model.predict(test_gen)

Answered By - hwktmp