[FIXED] Tensorflow gpu not able to train my Xception model

Issue

I am training an Xception model with tensorflow-gpu. I am getting this error:

FailedPreconditionError: 2 root error(s) found. (0) Failed precondition: Error while reading resource variable block14_sepconv2_bn_5/moving_variance from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/block14_sepconv2_bn_5/moving_variance/N10tensorflow3VarE does not exist. [[{{node FusedBatchNormV3/ReadVariableOp_1}}]] [[block9_sepconv3_bn_5/cond/else/_9661/OptionalFromValue_3/_1548]] (1) Failed precondition: Error while reading resource variable block14_sepconv2_bn_5/moving_variance from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/block14_sepconv2_bn_5/moving_variance/N10tensorflow3VarE does not exist. [[{{node FusedBatchNormV3/ReadVariableOp_1}}]] 0 successful operations. 0 derived errors ignored.

When I reload, it gives error on a different conv layer. I have not used a gpu before so I do not know how it works. Thanks for the help!

Solution

The problem has been sorted. Though I am not exactly sure what happened, I think it was a memory issue. Below lines of code have helped me a lot to manipulate my memory usage:

from tensorflow.compat.v1 import ConfigProto from tensorflow.compat.v1 import InteractiveSession

config = ConfigProto() config.gpu_options.allow_growth = True config.gpu_options.per_process_gpu_memory_fraction = 0.5 session = InteractiveSession(config=config)

Answered By - sonam agarwal