Fail to convert tensorflow model to onnx in nvidia NGC tensorflow container

Question

I follow instructions in triton-inference-server/tutorials to convert a tensorflow model to onnx with the purpose of testing the triton inference server.

However, the conversion fails inside of ngc tensorflow container. The command sequence is:

wget https://www.dropbox.com/s/r2ingd0l3zt8hxs/frozen_east_text_detection.tar.gz
tar -xvf frozen_east_text_detection.tar.gz
docker run -it --gpus all -v ${PWD}:/workspace nvcr.io/nvidia/tensorflow:22.01-tf2-py3

After pip install -U tf2onnx inside of the container, then run:

root@2ce770cd6b4e:/workspace# python -m tf2onnx.convert --input frozen_east_text_detection.pb --inputs "input_images:0" --outputs "feature_fusion/Conv_7/Sigmoid:0","feature_fusion/concat_3:0" --output detect
ion.onnx  

This results to pass1 convert failed for name: "resnet_v1_50/conv1/Conv2D" and the detail message is:

/usr/lib/python3.8/runpy.py:127: RuntimeWarning: 'tf2onnx.convert' found in sys.modules after import of package 'tf2onnx', but prior to execution of 'tf2onnx.convert'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))
WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tf2onnx/tf_loader.py:302: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.convert_variables_to_constants`
2023-11-19 10:20:14,199 - WARNING - From /usr/local/lib/python3.8/dist-packages/tf2onnx/tf_loader.py:302: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.convert_variables_to_constants`
WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/convert_to_constants.py:929: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
2023-11-19 10:20:14,199 - WARNING - From /usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/convert_to_constants.py:929: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
2023-11-19 10:20:14,876 - INFO - Using tensorflow=2.7.0, onnx=1.15.0, tf2onnx=1.15.1/37820d
2023-11-19 10:20:14,876 - INFO - Using opset <onnx, 15>
2023-11-19 10:20:17,650 - ERROR - pass1 convert failed for name: "resnet_v1_50/conv1/Conv2D"
op: "Conv2D"
input: "resnet_v1_50/Pad"
input: "resnet_v1_50/conv1/weights/read"
attr {
  key: "T"
  value {
    type: DT_FLOAT
  }
}
attr {
  key: "data_format"
  value {
    s: "NHWC"
  }
}
attr {
  key: "dilations"
  value {
    list {
      i: 1
      i: 1
      i: 1
      i: 1
    }
  }
}
attr {
  key: "explicit_paddings"
  value {
    list {
    }
  }
}
attr {
  key: "padding"
  value {
    s: "VALID"
  }
}
attr {
  key: "strides"
  value {
    list {
      i: 1
      i: 2
      i: 2
      i: 1
    }
  }
}
attr {
  key: "use_cudnn_on_gpu"
  value {
    b: true
  }
}
, ex=Could not infer attribute `explicit_paddings` type from empty iterator
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.8/dist-packages/tf2onnx/convert.py", line 714, in <module>
    main()
  File "/usr/local/lib/python3.8/dist-packages/tf2onnx/convert.py", line 273, in main
    model_proto, _ = _convert_common(
  File "/usr/local/lib/python3.8/dist-packages/tf2onnx/convert.py", line 168, in _convert_common
    g = process_tf_graph(tf_graph, const_node_values=const_node_values,
  File "/usr/local/lib/python3.8/dist-packages/tf2onnx/tfonnx.py", line 459, in process_tf_graph
    main_g, subgraphs = graphs_from_tf(tf_graph, input_names, output_names, shape_override, const_node_values,
  File "/usr/local/lib/python3.8/dist-packages/tf2onnx/tfonnx.py", line 474, in graphs_from_tf
    ordered_func = resolve_functions(tf_graph)
  File "/usr/local/lib/python3.8/dist-packages/tf2onnx/tf_loader.py", line 764, in resolve_functions
    _, _, _, _, _, functions = tflist_to_onnx(tf_graph, {})
  File "/usr/local/lib/python3.8/dist-packages/tf2onnx/tf_utils.py", line 462, in tflist_to_onnx
    onnx_node = helper.make_node(node_type, input_names, output_names, name=node.name, **attr)
  File "/usr/local/lib/python3.8/dist-packages/onnx/helper.py", line 164, in make_node
    node.attribute.extend(
  File "/usr/local/lib/python3.8/dist-packages/onnx/helper.py", line 165, in <genexpr>
    make_attribute(key, value)
  File "/usr/local/lib/python3.8/dist-packages/onnx/helper.py", line 876, in make_attribute
    raise ValueError(
ValueError: Could not infer attribute `explicit_paddings` type from empty iterator

Answer 1

Looks like an issue with ONNX 1.15 as documented here: https://github.com/onnx/tensorflow-onnx/issues/2262

Workaround is to run pip install onnx==1.14.1. Worked for me.