[FIXED] Split features, preprocess some of them, then join them back together. (hangs forever)

Issue

I'm trying to feed the all features (except the first one) to some layers (nn.Linear + nn.LeakyReLU), get the output, then reassemble the initial data structure and feed it to the last layers. But the training process just hangs forever and I don't get any output.

To be clear, the code works fine without this, but I'm trying to improve the results by preprocessing some of the features before feeding them (with the unprocessed first feature) to the last layer.

Any help would be much appreciated.

Here is my code:

def forward(self, x):
        # save the residual for the skip connection
        res = x[:, :, 0:self.skip]

        xSignal = np.zeros((len(x),len(x[0]),1))
        xParams = np.zeros((len(x),len(x[0]),len(x[0][0])-1))

        # separate data
        for b in range(len(x)):
            for c in range(len(x[b])):
                for d in range(len(x[b][c])):
                    if d == 0:
                        xSignal[b][c][d] = x[b][c][d]
                    else:
                        xParams[b][c][d-1] = x[b][c][d]

        # pass parameters through first network
        xParams = torch.from_numpy(xParams).cuda().float()
        xParams = self.paramsLinear(xParams)
        xParams = self.paramsLeakyRelu(xParams)

        # make new array with output and the signal
        xConcat = np.zeros((len(x),len(x[0]),len(x[0][0])))
        for b in range(len(x)):
            for c in range(len(x[b])):
                for d in range(len(x[b][c])):
                    if d == 0:
                        xConcat[b][c][d] = xSignal[b][c][d]
                    else:
                        xConcat[b][c][d] = xParams[b][c][d-1]

        # convert to tensor
        xConcat = torch.from_numpy(xConcat).cuda().float()

        # pass it through the recurrent part
        xConcat, self.hidden = self.rec(xConcat, self.hidden)
        # then the linear part and return
        return self.lin(xConcat) + res```

Solution

Well, as it turns out, slicing is WAY faster and easier than iterating. And I also used torch.cat function to put everything back in one tensor.

    def forward(self, x):
        # save the residual for the skip connection
        res = x[:, :, 0:self.skip]

        # split features
        xSignal = x[:, :, 0:1]
        xParams = x[:, :, 1:]

        # pass only some features through first layers
        xParams = self.paramsLinear(xParams)
        xParams = self.paramsLeakyRelu(xParams)

        # put everything back together
        x = torch.cat((xSignal, xParams), 2)

        # pass it through the last layers
        x, self.hidden = self.rec(x, self.hidden)
        # then the linear part and return
        return self.lin(x) + res

It's now training as expected :)

Answered By - Vítor Manfredini