Segmentation fault when executing binary in runc container

Question

I'm having a particular issue with a runc container. I have a bash script (running as root) that's executing the following:

$ setcap cap_net_bind_service=+ep ${PACKAGE_DIR}/bin/my-binary
$ chpst -u vcap:vcap "${PACKAGE_DIR}/bin/my-binary" --config "${JOB_DIR}/config/config.json"

every time, when it runs at container start it fails with this output:

 5116 Segmentation fault      chpst -u vcap:vcap "${PACKAGE_DIR}/bin/my-binary" --config "${JOB_DIR}/config/config.json"

However:

• When I ssh or shell into the container and run the command by hand, it will work.
• If I remove the chpst -u vcap:vcap (run as root) it will also work.
• If I remove the setcap cap_net_bind_servic... declaration, it will also work.
• When I change chown my-binary to vcap:vcap, it will work.

But none of these are valid options for the real implementation. I've even edited the source code of the binary to simply sleep for 2 hours - which had no effect, so I don't think the actual source code of the binary is the cause. Is there something I'm missing?

My runc version is:

runc version 1.0.0-rc5+dev
commit: 69663f0bd4b60df09991c08812a60108003fa340
spec: 1.0.0

Answer 1

The missing piece of the puzzle was these bit of logs here:

[  197.027144] bosh-dns[29239] vsyscall attempted with vsyscall=none ip:ffffffffff600000 cs:33 sp:7ffeeb9ea458 ax:ffffffffff600000 si:0 di:7ffeeb9ea460
[  197.028504] bosh-dns[29239]: segfault at ffffffffff600000 ip ffffffffff600000 sp 00007ffeeb9ea458 error 15
[  197.028772] bosh-dns[29233] vsyscall attempted with vsyscall=none ip:ffffffffff600000 cs:33 sp:7ffea26a3028 ax:ffffffffff600000 si:0 di:7ffea26a3030
[  197.028774] bosh-dns[29233]: segfault at ffffffffff600000 ip ffffffffff600000 sp 00007ffea26a3028 error 15

I don't know why but the fix for my problem was to add vsyscall=emulate to the kernel command line arguments that I was using.