How to solve circular reference detected

Question

I need to convert json into toml format.

When I run the following code, it returns with an error.

import toml

toml_config = toml.dumps(data)

where data is:

{"general":{"log_level":{"value":"4","editable":"true","description":"debug=5, info=4, warning=3, error=2, fatal=1, panic=0"},"log_to_syslog":{"value":"false","editable":"true","description":"When set to true, log messages are being written to syslog."}},"postgresql":{"dsn":"true","postgres://localhost/chirpstack_ns?sslmode\n\n# Automatically apply database migrations.\n#\n# It is possible to apply the database-migrations by hand\n# (see https://github.com/brocaar/chirpstack-network-server/tree/master/migrations)\n# or let ChirpStack Application Server migrate to the latest state automatically, by using\n# this setting. Make sure that you always make a backup when upgrading ChirpStack\n# Application Server and / or applying migrations.\nautomigratemax_open_connections":0,"max_idle_connections":2},"redis":{"servers":["localhost:6379"],"password":"","database":0,"cluster":"false","master_name":"","pool_size":0},"network_server":{"net_id":"000000","deduplication_delay":"200ms","device_session_ttl":"744h0m0s","get_downlink_data_delay":{"value":"100ms","editable":"true"},"band":{"name":"EU868","uplink_dwell_time_400ms":"false","downlink_dwell_time_400ms":"false","uplink_max_eirp":-1,"repeater_compatible":"false"},"network_settings":{"installation_margin":10,"rx_window":0,"rx1_delay":1,"rx1_dr_offset":0,"rx2_dr":-1,"rx2_frequency":-1,"rx2_prefer_on_rx1_dr_lt":0,"rx2_prefer_on_link_budget":"false","gateway_prefer_min_margin":10,"downlink_tx_power":-1,"disable_mac_commands":"false","disable_adr":"false","max_mac_command_error_count":3,"enabled_uplink_channels":[],"class_b":{"ping_slot_dr":0,"ping_slot_frequency":0},"rejoin_request":{"enabled":"false","max_count_n":0,"max_time_n":0}},"scheduler":{"scheduler_interval":"1s","class_c":{"downlink_lock_duration":"2s","multicast_gateway_delay":"2s"}},"api":{"bind":"0.0.0.0:8000","ca_cert":"","tls_cert":"","tls_key":""},"gateway":{"ca_cert":"","ca_key":"","client_cert_lifetime":"8760h0m0s","backend":{"type":"mqtt","multi_downlink_feature":"hybrid","mqtt":{"event_topic":"gateway/+/event/+","command_topic_template":"gateway/{{ .GatewayID }}/command/{{ .CommandType }}","server":"tcp://localhost:1883","username":"","password":"","max_reconnect_interval":"1m0s","qos":0,"clean_session":"true","client_id":"","ca_cert":"","tls_cert":"","tls_key":""},"amqp":{"url":"amqp://guest:guest@localhost:5672","event_queue_name":"gateway-events","event_routing_key":"gateway.*.event.*","command_routing_key_template":"gateway.{{ .GatewayID }}.command.{{ .CommandType }}"},"gcp_pub_sub":{"credentials_file":"","project_id":"","uplink_topic_name":"","downlink_topic_name":"","uplink_retention_duration":"24h0m0s"},"azure_iot_hub":{"events_connection_string":{"value":"","editable":"true","description":"This connection string must point to the Service Bus Queue to which the IoT Hub is forwarding the (uplink) gateway events."},"commands_connection_string":{"value":"","editable":"true","description":"This connection string must point to the IoT Hub and is used by ChirpStack Network Server for sending commands to the gateways."}}}}},"monitoring":{"bind":"","prometheus_endpoint":"false","prometheus_api_timing_histogram":"false","healthcheck_endpoint":"false"},"join_server":{"resolve_join_eui":"false","resolve_domain_suffix":".joineuis.lora-alliance.org","default":{"server":"http://localhost:8003","ca_cert":"","tls_cert":"","tls_key":""}},"network_controller":{"server":"","ca_cert":"","tls_cert":"","tls_key":""}}

The error I get while running above code is:

Traceback (most recent call last):
  File "<pyshell#38>", line 1, in <module>
    toml_string = toml.dumps(data)
  File "/usr/local/lib/python3.8/dist-packages/toml/encoder.py", line 67, in dumps
    raise ValueError("Circular reference detected")
ValueError: Circular reference detected

After digging in, I found that after deleting a key-value pair, the code works fine. The key-value pair is:

"type": "mqtt"

The location of this key-value pair is data["network_server"]["gateway"]["backend"].

I am not able to understand this situation. I tried to change the key-value pair string but still the same problem. Only removing this key-value pair solves the problem. But I need this pair.

Any help will be highly appreciated. Thanks in advance.

Answer 1

I tried to reproduce your problem, because I planned to switch from json config to toml config too. I have installed toml version 0.10.1.

There is a small problem in the data (# comments after the "postgresql" key), but that section can be deleted entirely.

I could minimize the example to:

data = { 
        "network_server":{
                "downlink_data_delay":{},
                "gateway":{"key3":{"key4":{}}
                }   
        }   
}

import toml
print(toml.dumps(data)) # ValueError: Circular reference detected

But the weird thing is the error goes away when you keep the structure, but slightly rename the keys!!

Just delete the first 'n' from the "network_server" and you'll get na output!

That must be a bug. Very sad.

---

Update: After looking at the source, the bug is quite obvious.

retval = ""
if encoder is None:
    encoder = TomlEncoder(o.__class__)
addtoretval, sections = encoder.dump_sections(o, "")
retval += addtoretval
outer_objs = [id(o)]
while sections:
    section_ids = [id(section) for section in sections]
    for outer_obj in outer_objs:
        if outer_obj in section_ids:
            raise ValueError("Circular reference detected")
    outer_objs += section_ids
    newsections = encoder.get_empty_table()
    for section in sections:
        addtoretval, addtosections = encoder.dump_sections(
            sections[section], section)

        if addtoretval or (not addtoretval and not addtosections):
            if retval and retval[-2:] != "\n\n":
                retval += "\n"
            retval += "[" + section + "]\n"
            if addtoretval:
                retval += addtoretval
        for s in addtosections:
            newsections[section + "." + s] = addtosections[s]
    sections = newsections
return retval

Or isn't it? OK, it tracks the ids of sections, but the sections are not preserved between loop iterations. When ids get reused, the program is confused.

Quick & dirty fix: patch the toml/encoder.py file, in the dumps function:

1. add allsections=[] before the while loop

2. add one line here:

        allsections += sections   # <---
        sections = newsections
    return retval
   

The purpose is to prevent the garbage collection of unused sections, in order to keep their ids in use.

Feel free to report the issue to the author at github.