Memory leak evaluating CNN model for text clasification

Question

I've been doing some adaptation to code in this blog about CNN for text clasification: http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/

Everything works fine! But when I try to use the model trained to predict new instances it consumes all memory available. It seems that it's not liberating any memory when evaluates and load all the model again and again. As far as I know memory should be liberated after every sess.run command.

Here is the part of the code I'm working with:

with graph.as_default():

session_conf = tf.ConfigProto(
  allow_soft_placement=FLAGS.allow_soft_placement,
  log_device_placement=FLAGS.log_device_placement)
sess = tf.Session(config=session_conf)
with sess.as_default():

    # Load the saved meta graph and restore variables
    saver = tf.train.import_meta_graph("{}.meta".format(checkpoint_file))
    saver.restore(sess, checkpoint_file)

    # Get the placeholders from the graph by name
    input_x = graph.get_operation_by_name("input_x").outputs[0]
    # input_y = graph.get_operation_by_name("input_y").outputs[0]
    dropout_keep_prob = graph.get_operation_by_name("dropout_keep_prob").outputs[0]

    # Tensors we want to evaluate
    predictions = graph.get_operation_by_name("output/predictions").outputs[0]

    # Add a vector for probas
    probas =graph.get_operation_by_name("output/scores").outputs[0]

    # Generate batches for one epoch
    print("\nGenerating Bathces...\n")
    gc.collect()
    #mem0 = proc.get_memory_info().rss
    batches = data_helpers.batch_iter(list(x_test), FLAGS.batch_size, 1, shuffle=False)
    #mem1 = proc.get_memory_info().rss

    print("\nBatches done...\n")
    #pd = lambda x2, x1: 100.0 * (x2 - x1) / mem0
    #print "Allocation: %0.2f%%" % pd(mem1, mem0)
    # Collect the predictions here
    all_predictions = []

    all_probas = []

    for x_test_batch in batches:
        #Calculate probability of prediction been good
        gc.collect()
        batch_probas = sess.run(tf.reduce_max(tf.nn.softmax(probas),1), {input_x: x_test_batch, dropout_keep_prob: 1.0})
        batch_predictions = sess.run(predictions, {input_x: x_test_batch, dropout_keep_prob: 1.0})
        all_predictions = np.concatenate([all_predictions, batch_predictions])
        all_probas = np.concatenate([all_probas, batch_probas])
        # Add summary ops to collect data
        with tf.name_scope("eval") as scope:
            p_h = tf.histogram_summary("eval/probas", batch_probas)
            summary= sess.run(p_h)
            eval_summary_writer.add_summary(summary)

Any help will be much appreciated

Cheers

Answer 1

Your training loop creates new TensorFlow operations (tf.reduce_max(), tf.nn.softmax() and tf.histogram_summary()) in each iteration, which will lead to more memory being consumed over time. TensorFlow is most efficient when you run the same graph many times, because it can amortize the cost of optimizing the graph over multiple executions. Therefore, to get the best performance, you should revise your program so that you create each of these operations once, before the for x_test_batch in batches: loop, and then re-use the same operations in each iteration.