Receive incremental ZFS stream of clone and overwrite origin

Question

tldr; I'm trying to receive a ZFS stream, that has been created as replicate (-R) from a cloned filesystem. Using zfs recv -o origin=[clone-origin] just gives cannot receive: local origin for clone [...] does not exist.

Precondition

I have a SmartOS zone ZFS filesystem, which is cloned from a particular image. (IMAGE-uuid and ZONE-uuid have been replaced for better readability)

$ zfs list -r -o name,origin zones/[ZONE]
NAME          ORIGIN              
zones/[ZONE]  zones/[IMAGE]@[ZONE]

The zone filesystem has serveral snapshots:

$ zfs list -r -t all -o name, origin zones/[ZONE]
NAME                  ORIGIN              
zones/[ZONE]          zones/[IMAGE]@[ZONE]
zones/[ZONE]@[SNAP0]  -
zones/[ZONE]@[SNAP1]  -
zones/[ZONE]@[SNAP2]  -
[...]

Regarding the base image, SmartOS (better vmadm) creates a snapshot of the image for the newly created zone. The zone root is created as clone based on this snapshot (here with guid 11194422825011190557).

$ zfs list -r -o name,origin,guid zones/[IMAGE]
NAME                        ORIGIN  GUID
zones/[IMAGE]               -       5616748063181666458
zones/[IMAGE]@[OTHER-ZONE]  -       11174377117517693115
zones/[IMAGE]@[OTHER-ZONE]  -       5587104570997150836
zones/[IMAGE]@[OTHER-ZONE]  -       535244446308996462
zones/[IMAGE]@[OTHER-ZONE]  -       12527420623439849960
zones/[IMAGE]@[ZONE]        -       11194422825011190557
zones/[IMAGE]@[OTHER-ZONE]  -       18143527942366063753
zones/[IMAGE]@[OTHER-ZONE]  -       15066902894708043304
zones/[IMAGE]@[OTHER-ZONE]  -       16574922393629090803
zones/[IMAGE]@[OTHER-ZONE]  -       818178725388359655
zones/[IMAGE]@[OTHER-ZONE]  -       11867824093224114226
zones/[IMAGE]@[OTHER-ZONE]  -       9357513766021831186

Backup

To create a backup of my zone root, I created a snapshot and a replicate stream.

zfs snapshot zones/[ZONE]@[DATE]
zfs send -R zones/[ZONE]@[DATE] > [ZONE]_[DATE].zfs

Inspecting it with zstreamdump shows the expected origin. It is in hex but 0x9b5a943fae511b1d is 11194422825011190557:

$ zstreamdump < [ZONE]_[DATE].zfs
BEGIN record
        hdrtype = 2
        features = 4
        magic = 2f5bacbac
        creation_time = 0
        type = 0
        flags = 0x0
        toguid = 0
        fromguid = 0
        toname = zones/[ZONE]@[DATE]
nvlist version: 0
        tosnap = [DATE]
        fss = (embedded nvlist)
        nvlist version: 0
                0xf19ec8c66f3ca037 = (embedded nvlist)
                nvlist version: 0
                        name = zones/[ZONE]
                        parentfromsnap = 0x0
                        origin = 0x9b5a943fae511b1d
                        props = (embedded nvlist)
                        nvlist version: 0
                                devices = 0x0
                                compression = 0x2
                                quota = 0x500000000
                        (end props)
[...]

Restore

To recover a desaster, I recreate the zone using vmadm create with a backup of the vm description (the ZONE-uuid is preserved). vmadm pulls the image and creates the respective zfs filesystem zones/[IMAGE] with a snapshot, as clone origin for the recreated zone filesystem zones/[ZONE].

So the structure is the same as before the crash:

$ zfs list -r -o name,origin zones/[ZONE]
NAME          ORIGIN              
zones/[ZONE]  zones/[IMAGE]@[ZONE]

However the guid of the image-snapshot (created by vmadm), is different - as expected. The stream expects 0x9b5a943fae511b1d (or 11194422825011190557), but it actually is 12464070312561851369:

: zfs list -r -o name,guid zones/[IMAGE]
NAME                  GUID
zones/[IMAGE]         5616748063181666458
[...]
zones/[IMAGE]@[ZONE]  12464070312561851369
[...]

That's where - I thought - the -o origin= parameter of zfs recv comes in.

Problem

Restoring the actual data by receiving the zfs stream, ends up with an error:

$ zfs recv -vF zones/[ZONE] < [ZONE]_[DATE].zfs
cannot receive: local origin for clone zones/[ZONE]@[SNAP0] does not exist

(where SNAP0 is the first snapshot of the backed up filesystem, see "Precondition" above)

This is expected, since the guid changed. So I forced the origin to the image snapshot with the new guid (12464070312561851369), but the error remains the same:

$ zfs recv -vF -o origin=zones/[IMAGE]@[ZONE] zones/[ZONE] < [ZONE]_[DATE].zfs
cannot receive: local origin for clone zones/[ZONE]@[SNAP0] does not exist

Question

Is my interpretation of the -o origin=-parameter correct?

Why doesn't work it as expected?

If this is the wrong way, how can I create a backup and restore a zfs filesystem that is cloned?

Thanks a lot for reading and helping!

Answer 1

It seems you stumbled on a ZFS bug that only now is getting a bit of attention.

If you can change how the stream is created

The -R flag tries to preserve all kinds of relations that may often not be of relevance, such as parent clones etc. There is no handy alternative that would only "send all incrementals up until this one". Instead, you have to do two passes. This is not specific to vdadm, so for ZFS in general, the logic is as follows:

zfs send zones/[ZONE]@[EARLIESTDATE] > [ZONE]_[EARLIESTDATE].zfs
zfs send -I zones/[ZONE]@[EARLIESTDATE] zones/[ZONE]@[DATE] > [ZONE]_[EARLIESTDATE]-[DATE].zfs
zfs recv -vF zones/[ZONE] < [ZONE]_[EARLIESTDATE].zfs
zfs recv -vF zones/[ZONE] < [ZONE]_[EARLIESTDATE]-[DATE].zfs

After this, only -I passes are needed between the latest backed up snapshot and the newest one at the source.

If you have to restore the already created stream

One proposed solution is to use a modified variant of zfs described here:

https://github.com/openzfs/zfs/issues/10135

Please ensure you know how this affects your dataset, though. Then your command would be

FQ_OVERRIDE_GTND=1 .zfs recv vF -o origin=zones/[IMAGE]@[ZONE] zones/[ZONE] < [ZONE]_[DATE].zfs

Another bug report of the same is here:

https://github.com/openzfs/zfs/issues/10935