I have implemented a method which I am using for preprocessing of examples during my training.
I was able to boil it down to one particular function. If I am using it during training, I am seeing this in my log:
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into /mariana/models/.model=ed6f9dfedc5dbba0_shards=9e8a89eceebc4a45/checkpoints/model.ckpt.
2019-12-12 11:36:07.608652: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at strided_slice_op.cc:106 : Invalid argument: slice index 0 of dimension 1 out of bounds.
2019-12-12 11:36:07.608709: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at strided_slice_op.cc:106 : Invalid argument: slice index 0 of dimension 1 out of bounds.
...
2019-12-12 11:36:08.105489: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at strided_slice_op.cc:106 : Invalid argument: slice index 0 of dimension 1 out of bounds.
2019-12-12 11:36:08.105494: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at strided_slice_op.cc:106 : Invalid argument: slice index 0 of dimension 1 out of bounds.
2019-12-12 11:36:08.105494: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at strided_slice_op.cc:106 : Invalid argument: slice index 0 of dimension 1 out of bounds.
2019-12-12 11:36:08.105494: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at strided_slice_op.cc:106 : Invalid argument: slice index 0 of dimension 1 out of bounds.
2019-12-12 11:36:08.105728: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at strided_slice_op.cc:106 : Invalid argument: slice index 0 of dimension 1 out of bounds.
2019-12-12 11:36:08.105754: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at strided_slice_op.cc:106 : Invalid argument: slice index 0 of dimension 1 out of bounds.
I cannot reproduce this issue in my unit tests and I cannot reproduce this in another project, where I use the same function without errors.
Since I do not get a stack trace, I have no clue what the issue is or where to look for errors.
Is there a way to get more information from tensorflow for errors like this?