ValueError: Comments are not supported by the python backend

Question

The ijson module has a documented option allow_comments=True, but when I include it, an error message is produced:

ValueError: Comments are not supported by the python backend

Below is a transcript using the file test.py:

import ijson
for o in ijson.items(open(0), 'item'):
     print(o)

Please note that I have no problem with a similar documented option, multiple_values=True.

Transcript

$ python3 --version
Python 3.10.9
$ python3 test.py <<< [1,2]
1
2
# Now change the call to: ijson.items(open(0), 'item', allow_comments=True)
$ python3 test.py <<< [1,2]
Traceback (most recent call last):
  File "/Users/user/test.py", line 5, in <module>
    for o in ijson.items(open(0), 'item', allow_comments=True):
  File "/usr/local/lib/python3.10/site-packages/ijson/utils.py", line 51, in coros2gen
    f = chain(events, *coro_pipeline)
  File "/usr/local/lib/python3.10/site-packages/ijson/utils.py", line 29, in chain
    f = coro_func(f, *coro_args, **coro_kwargs)
  File "/usr/local/lib/python3.10/site-packages/ijson/backends/python.py", line 284, in basic_parse_basecoro
    raise ValueError("Comments are not supported by the python backend")
ValueError: Comments are not supported by the python backend
$

Answer 1

Take a look at the Backends section of the documentation, which says:

Ijson provides several implementations of the actual parsing in the form of backends located in ijson/backends:

yajl2_c: a C extension using YAJL 2.x. This is the fastest, but might require a compiler and the YAJL development files to be present when installing this package. Binary wheel distributions exist for major platforms/architectures to spare users from having to compile the package.

yajl2_cffi: wrapper around YAJL 2.x using CFFI.

yajl2: wrapper around YAJL 2.x using ctypes, for when you can’t use CFFI for some reason.

yajl: deprecated YAJL 1.x + ctypes wrapper, for even older systems.

python: pure Python parser, good to use with PyPy

And later on in the FAQ it says:

Q: Are there any differences between the backends?

... The python backend doesn’t support allow_comments=True It also internally works with str objects, not bytes, but this is an internal detail that users shouldn’t need to worry about, and might change in the future.

If you want support for allow_comments=True, you need to be using one of the yajl based backends. According to the docs:

Importing the top level library as import ijson uses the first available backend in the same order of the list above, and its name is recorded under ijson.backend. If the IJSON_BACKEND environment variable is set its value takes precedence and is used to select the default backend.

You'll need the necessary libraries, etc, installed on your system in order for this to work.