[FIXED] Tensorflow Keras: Can `Conv2d` layers now accept multispectral images meaning bands greater than 3

Issue

I have some 8-band satellite images and wanted to do some image segmentation with Tensorflow and Keras. I tried to do this a couple of years ago, but saw that TF and Keras could not handle images with bands greater than 3. However, I am seeing more blog posts about deep learning with multiband images.

In looking at the Keras documentation, it does not specifically list any problems with accepting multiband images. And I found this code which seems to make it work:

def unet_model(n_classes=5, im_sz=320, n_channels=8, n_filters_start=32, growth_factor=2, upconv=True,
               class_weights=[0.2, 0.3, 0.1, 0.1, 0.3]):
    droprate=0.25
    n_filters = n_filters_start
    inputs = Input((im_sz, im_sz, n_channels))
    #inputs = BatchNormalization()(inputs)
    conv1 = Conv2D(n_filters, (3, 3), activation='relu', padding='same')(inputs)
    conv1 = Conv2D(n_filters, (3, 3), activation='relu', padding='same')(conv1)
    pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)
pool1 = Dropout(droprate)(pool1)

So just wanted to clarify, can tf.keras.Conv2d layers and other layers accept 8 or more band images now? Are there any pitfalls of using multiband images--like needing some transformations on the data before processing. Are there any limitations on using multispectral images?

Solution

Yes the can accept more 8 channels. Both Keras and TensorFlow layers. The main problem with images that have more than 3 channels is that majority of readily accessible pre-trained models were trained on standard imagenet dimensions, like [299,299,3] In this case it will require considerable ammout of work to fine tune such model to your data. As a solution to this you can insert a special 'resizing' convolutional layer which will reshape it to 3 layers.

inputs = tf.keras.layers.Input(shape=(320,320,8)) 
resize = tf.keras.layers.Conv2D(filters=3, kernel_size)(inputs)

This however, can cause some data loss so needs to be used with caution.

Answered By - Sharky