How to upgrade Hono from file based device registry to mongodb based?

Question

I try to upgrade Hono from 1.2.3 to 1.4.3 version. But in 1.2.3 version, I store my data at file, now I want to use mongodb based device registry. But I found that the structure between mongodb collection and file is different. Like below:

1.2.3 tenants.json

[ {
  "enabled" : true,
  "tenant-id" : "test"
}, {
  "enabled" : true,
  "trusted-ca" : [ {
    "subject-dn" : "CN=loadtest CA,OU=Hono,O=Bosch,L=NJ,C=CN",
    "public-key" : "MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAxN+0lLMoRvMyiINcP4COYFbx6X5tPzNJ/kg57dFAEf6dPVtLuO4wBI44X6BHmefOskLsAOW0UGFztXwfYxYgVgEFdDWWGqEXFjMPXfDR48ace7DsxqLFl98dOIp0VEB0HWf5sQi6CtMb6r0RWatvw2Otx2GZuaxCix/rJ96UnkoiqUt6aaCwfLRXEYTeNJOqyValzhgJ0R5vUH/QqO4lTIejTS3uCep08l2I7YcZvpSTshETKVH847ByxYE3uOSgtnYRUTmOtHh44r3g8vlL8bJfkmrps3atXpkvKxo+xj2TAszjRSEfjH2zfGWrHl46zdz5hr7gBJSHh9LegXc/V843YSbPcN/7fVJr5qX9fNSvVPTGLqoL4dGuYzWoLxjeC9lYihizRUrcdy+ApS9COlwRL4VHPE8quid3QK12+DcwK4k1nrBbxi9UshlQEW7KMHkSl8wNnwKLGhKR/d9KJ71C7blsVfmC+JNA+WPFhzSa+kYM3WvGf/SnwpZoErW009izgWFgHCY19jZFjy0qgMWUoLdXzUkSEkhP5JUhPl+PMxcfGe46BSIZm38yBJI85KN0pui/990BvusnrKYs6WaP1zx+/KmWwBvlbBVcingnRHUQ4n/lGeps5+HscjbTwMXOzk1gwmzyqvZbhCyYiTZM4uC6GAsNKD69kyzfvh8CAwEAAQ==",
    "algorithm" : "RSA",
    "not-before" : "2020-06-16T23:42:42Z",
    "not-after" : "2021-06-16T23:42:42Z",
    "auto-provisioning-enabled" : false
  } ],
  "tenant-id" : "loadtest"
} ]

1.4.3 tenants collection

{
        "_id" : "5fbca60323059c55e8bf7174",
        "version" : "466f5076-6e4f-4c10-a4c7-1434b1725b7e",
        "updatedOn" : "2020-11-24T06:19:47Z",
        "tenant-id" : "DEFAULT_TENANT",
        "tenant" : {
                "enabled" : true,
                "trusted-ca" : [
                        {
                                "subject-dn" : "CN=DEFAULT_TENANT_CA,OU=Hono,O=Eclipse IoT,L=Ottawa,C=CA",
                                "public-key" : "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAElkwCSPlO563eQb6ONdULAISm2XngGGSoAAz+I1s8zkS9guPUpNKoxeczLtKlObelHqBgIZtRXdrPRgXidGOnmQ==",
                                "algorithm" : "EC",
                                "not-before" : "2019-09-18T08:35:40Z",
                                "not-after" : "2020-09-17T08:35:40Z",
                                "auto-provisioning-enabled" : false
                        }
                ]
        }
}
{
        "_id" : "5fbca60423059c55e8bf7175",
        "version" : "c41fe230-88ae-4701-bbd6-66ea4b068271",
        "updatedOn" : "2020-11-24T06:19:48Z",
        "tenant-id" : "HTTP_TENANT",
        "tenant" : {
                "enabled" : true,
                "adapters" : [
                        {
                                "type" : "hono-http",
                                "enabled" : true,
                                "device-authentication-required" : true
                        },
                        {
                                "type" : "hono-mqtt",
                                "enabled" : false,
                                "device-authentication-required" : true
                        },
                        {
                                "type" : "hono-kura",
                                "enabled" : false,
                                "device-authentication-required" : true
                        },
                        {
                                "type" : "hono-coap",
                                "enabled" : false,
                                "device-authentication-required" : true
                        }
                ]
        }
}

other two collections devices/credentials are also have different structure.

So how can I upgrade my Hono instance?

Thanks in advance.

Answer 1

Update
This solution has been added to the hono-extras github repository. When you have an update or fix, this might be the proper place to share this.

We also made this transition and wrote a small python script: https://gist.github.com/BobClaerhout/4dd1d2558e3210954419e2746232bbf9.
I must warn you, this was sufficient for us but might not be complete. However, this can give you a headstart.

You can follow following guide to move your data:

• Export from file-based registry:

kubectl cp hono-service-device-registry-0:/var/lib/hono/device-registry .

• Transform all collections:

./file2mongo.py tenants device-registry/tenants.json > tenants-converted.json
./file2mongo.py devices device-registry/devices.json > devices-converted.json
./file2mongo.py credentials device-registry/credentials.json > credentials-converted.json

• Import in MongoDB (see https://github.com/txn2/kubefwd)

sudo -E kubefwd svc

mongoimport --host hono-monogodb-server:27017 -u device-registry@HONO -p hono-secret -d honodb -c tenants --jsonArray --file tenants-converted.json
mongoimport --host hono-monogodb-server:27017 -u device-registry@HONO -p hono-secret -d honodb -c devices --jsonArray --file devices-converted.json
mongoimport --host hono-monogodb-server:27017 -u device-registry@HONO -p hono-secret -d honodb -c credentials --jsonArray --file credentials-converted.json