Dedicated GPU memory doesn't reset

Programming
, , ,
1

Problem Statement

I am having an issue with training machine learning models with TensorFlow-GPU in a Jupyter notebook. After completing the training, my dedicated GPU memory is still at 85% usage, and I am unable to run the same model or a modified version without receiving an error (Failed to get convolution algorithm. This is probably because cuDNN failed to initialize). To free up the GPU memory, I have to quit Anaconda Prompt and relaunch Jupyter notebook. Is anyone else experiencing the same issue? Does anyone know how to clear the GPU memory?

Failed to get convolution algorithm. This is probably because cuDNN failed to initialize
2

To clear the GPU memory in TensorFlow, you can use the following code snippet:

from numba import cuda
cuda.select_device(0)
cuda.close()

This code will release GPU memory used by TensorFlow. You can run this code snippet after completing the training to free up the GPU memory.