lxc-snapshot fails with rc 1 for no apparent reason

Question

At my side lxc-snapshot fails, but does not tell anything.

• Tried to Google for it, not found anything. All is for lxc snapshot (note the missing dash) which is something completely different.
• Looked into manual, etc. nothing found.
• strace is not telling anything either, it talks to the anonymous command socket, gets a response and fails

$ lxc-start b2
$ lxc-attach b2
# 

So container is running. But snapshot does not work:

$ lxc-snapshot b2; echo $?
1
$ lxc-snapshot b2 -L
No snapshots

Problem exist on Debian Buster:

$ lxc-snapshot --version
3.0.3
$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 10 (buster)
Release:    10
Codename:   buster

Problem exist on Ubuntu, too:

$ lxc-snapshot --version
4.0.2
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.1 LTS
Release:        20.04
Codename:       focal

Container configuration:

lxc.include = /usr/share/lxc/config/common.conf
lxc.include = /usr/share/lxc/config/userns.conf
lxc.arch = linux64
lxc.uts.name = b2
lxc.rootfs.path = dir:/home/tino/.local/share/lxc/b2/rootfs
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:d2:0e:02
lxc.idmap = u 0 638752 65536
lxc.idmap = g 0 638752 65536

What am I missing here? The questions are:

• How to find/enable diagnostics?
• Where is this documented?
• What do I (perhaps) need in Container's config?

You can point to lxc, lxd or docker for diagnostic purpose. But please note that this question is not about them, it's precisely about the low level container commands like lxc-start and lxc-snapshot (note the dash).

Answer 1

The manual states:

   -l, --logpriority=LEVEL
          Set log priority to LEVEL. The default log priority is ERROR.  Possible  values
          are : FATAL, CRIT, WARN, ERROR, NOTICE, INFO, DEBUG.

          Note  that  this option is setting the priority of the events log in the alter‐
          nate log file. It do not have effect on the ERROR events log on stderr.

This apparently is wrong. Logging to stderr is affected by -l and it isn't ERROR by default:

$ lxc-start b2
$ lxc-snapshot -lERROR b2 2>/dev/null; echo $?
1
$ lxc-snapshot -lERROR b2
lxc-snapshot: b2: lxccontainer.c: do_lxcapi_snapshot: 4190 Snapshot of directory-backed container requested
lxc-snapshot: b2: lxccontainer.c: do_lxcapi_snapshot: 4191 Making a copy-clone.  If you do want snapshots, then
lxc-snapshot: b2: lxccontainer.c: do_lxcapi_snapshot: 4192 please create overlay clone first, snapshot that
lxc-snapshot: b2: lxccontainer.c: do_lxcapi_snapshot: 4193 and keep the original container pristine
lxc-snapshot: b2: lxccontainer.c: do_lxcapi_clone: 3809 error: Original container (b2) is running. Use --allowrunning if you want to force a snapshot of the running container.
lxc-snapshot: b2: lxccontainer.c: do_lxcapi_snapshot: 4199 Failed to clone of /home/tino/.local/share/lxc:b2
lxc-snapshot: b2: tools/lxc_snapshot.c: do_snapshot: 199 Error creating a snapshot

Hence, lxc-snapshot only works on stopped containers and you only see errors if you enable logging. On a stopped container it works:

$ lxc-stop b2
$ lxc-snapshot -n b2 -L
No snapshots
$ lxc-snapshot -n b2
$ lxc-snapshot -n b2 -L
snap0 (/home/tino/.local/share/lxc/b2/snaps) 2020:11:25 12:22:03
$ ls -al ~/.local/share/lxc/b2/snaps/snap0/
total 20
drwxrwx---  3 638752 tino   4096 Nov 25 12:22 .
drwxr-xr-x  3 tino   tino   4096 Nov 25 12:20 ..
-rw-rw----  1 tino   tino   1171 Nov 25 12:22 config
drwxr-xr-x 21 638752 638752 4096 Nov 24 20:01 rootfs
-rw-rw-r--  1 tino   tino     19 Nov 25 12:22 ts
$ cat ~/.local/share/lxc/b2/snaps/snap0/ts; echo
2020:11:25 12:22:03

Notes:

• --allowrunning is not mentioned in the man page either
• lxc-snapshot appears to mostly wrap rsync (in this case).

This solution was found by looking up the implementation of the snapshot function on GitHub and digging deeper and deeper and deeper, getting completely lost and finally giving up on this path. Then I wrote above Question, and, while doing that, I luckily stumbled over the solution myself. Don't Panic, sometimes the solution seems to be just to ask the right way.