v4l2 VIDIOC_STREAMON: invalid argument when specifying pixel format

Question

I am using v4l2 to read frames from the camera.

The complete boiler plate is at https://www.kernel.org/doc/html/v4.9/media/uapi/v4l/capture.c.html

But the problem occurs when I change the pixel format:

struct v4l2_format fmt;
// This works (line 497 in boilerplate)
if (-1 == xioctl(fd, VIDIOC_G_FMT, &fmt)) {
  exit(EXIT_FAILURE);
}
// but when I add the following, it fails (line 485 in boilerplate):
fmt.fmt.pix.pixelformat = v4l2_fourcc('R', 'G', 'B', '4');

if (-1 == xioctl(fd, VIDIOC_S_FMT, &fmt)) {
  exit(EXIT_FAILURE);
}

When I run the code with setting the pixel format, it does not exit on the second ioctl, but errors later when starting stream:

enum v4l2_buf_type type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
if (-1 == xioctl(fd, VIDIOC_STREAMON, &type)) {
  fprintf(stderr, "VIDIOC_STREAMON error: %d, %s\n", errno, strerror(errno));
  exit(EXIT_FAILURE);
} 

now I get the error message "VIDIOC_STREAMON: error 22, Invalid argument"

The format is supported by the camera however:

v4l2-ctl -d /dev/video0 --list-formats
# ...
[10]: 'RGB4' (32-bit A/XRGB 8-8-8-8)
# ...

I am completely lost on why this happens and have been searching for hours.

The format returned by ioctl VIDIOC_G_FMT is 144869129, which translates to pBAA, which, to my knowledge, is not a valid fourcc pixel format?

hardware: running on Raspberry Pi 4 camera: Raspberry Pi Camera Module v3