Run Tensorflow object detecion with Nvidia GTX650 Ti (2GB)?

Question

Is there a way to let run the object detection with a 2GB graphic card? Have 24GB DD3 Ram on the main board, can't I use that too with the GPU?

I did try to add session_config.gpu_options.allow_growth=True in the trainer.py but that don't help. It Seems that the graphic card doesn't have enough memory.

CardInfos:

0, name: GeForce GTX 650, pci bus id: 0000:01:00.0)
[name: "/cpu:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 4876955943962853047
, name: "/gpu:0"
device_type: "GPU"
memory_limit: 1375862784
locality {
  bus_id: 1
}
incarnation: 4236842880144430162
physical_device_desc: "device: 0, name: GeForce GTX 650, pci bus id: 0000:01:00.0"
]

train.py output:

Limit:                   219414528
InUse:                   192361216
MaxInUse:                192483072
NumAllocs:                    6030
MaxAllocSize:              6131712

2017-09-13 13:47:13.429510: W tensorflow/core/common_runtime/bfc_allocator.cc:277] ****************************************************************************************____________
2017-09-13 13:47:13.481829: W tensorflow/core/framework/op_kernel.cc:1192] Internal: Dst tensor is not initialized.
     [[Node: prefetch_queue_Dequeue/_5471 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_5476_prefetch_queue_Dequeue", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InternalError'>, Dst tensor is not initialized.
     [[Node: prefetch_queue_Dequeue/_5471 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_5476_prefetch_queue_Dequeue", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]
2017-09-13 13:47:13.955327: W tensorflow/core/framework/op_kernel.cc:1192] Internal: Dst tensor is not initialized.
     [[Node: prefetch_queue_Dequeue/_299 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_3432_prefetch_queue_Dequeue", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]
2017-09-13 13:47:13.956056: W tensorflow/core/framework/op_kernel.cc:1192] Internal: Dst tensor is not initialized.
     [[Node: prefetch_queue_Dequeue/_299 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_3432_prefetch_queue_Dequeue", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1327, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1306, in _run_fn
    status, run_metadata)
  File "/usr/lib/python3.5/contextlib.py", line 66, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.
     [[Node: prefetch_queue_Dequeue/_5471 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_5476_prefetch_queue_Dequeue", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train.py", line 198, in <module>
    tf.app.run()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "train.py", line 194, in main
    worker_job_name, is_chief, FLAGS.train_dir)
  File "/home/dee/Documents/projects/tensor/models/object_detection/trainer.py", line 297, in train
    saver=saver)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/slim/python/slim/learning.py", line 755, in train
    sess, train_op, global_step, train_step_kwargs)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/slim/python/slim/learning.py", line 488, in train_step
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 895, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1124, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1321, in _do_run
    options, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1340, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.

Answer 1

Indeed the Dst tensor is not initialized message indicates that your GPU runs out of memory. You can try to decrease the batch size to a minimum and also decrease the resolution of the images that you are feeding into your model. Also try to use the SSD Mobilenet Model, because it is very lightweight.

To answer the second part of your question: I have always thought that modern GPUs will run into a hybrid mode where the drivers/GPU start streaming resources from system RAM over the PCIe bus to make up for the "missing" VRAM. Since system RAM is 3-5X slower than GDDR5 with much higher latency, running out of VRAM would translate into a significant performance loss. However I faced the same issue on my GTX 1060 with 6GB VRAM, where the CUDA Process crashed because it ran out of GPU.