I'm currently working on a linux kernel module for a school project, that involves exposing the kernels hashtable implementation to user space.
To achieve this, I've also created an user space API that communicates with the LKM via a Netlink socket.
I do have it working now, but I ran into a bump that left me quite confused and I really can't wrap my head around. And after reading through all documentation, which really didn't help my understanding of the problem, and "going down the rabbit hole" and looking through the source code for Netlink, I figured I'd ask the question here to see if anyone might know what, and why, this is happening.
So, to isolate the problem, I created a small test program that runs the generic Netlink user-space and kernel-space communication example. Through this I will show 3 small variations of the user space program, which all have different behavior, and it is THAT behavior I am wondering about.
So first is the kernel module, which will be the same for all 3 variations:
#include <linux/module.h>
#include <net/sock.h>
#include <linux/netlink.h>
#include <linux/skbuff.h>
#define NETLINK_USER 31
struct sock *nl_sk = NULL;
static void hello_nl_recv_msg(struct sk_buff *skb){
struct nlmsghdr *nlh;
int pid;
struct sk_buff *skb_out;
int msg_size;
char *msg = "Hello from kernel";
int res;
printk(KERN_INFO "Entering: %s\n", __FUNCTION__);
msg_size = strlen(msg);
nlh = (struct nlmsghdr *)skb->data;
printk(KERN_INFO "Netlink received msg payload:%s\n", (char *)nlmsg_data(nlh));
pid = nlh->nlmsg_pid; //pid of sending process
skb_out = nlmsg_new(msg_size, 0);
if (!skb_out) {
printk(KERN_ERR "Failed to allocate new skb\n");
return;
}
nlh = nlmsg_put(skb_out, 0, 0, NLMSG_DONE, msg_size, 0);
NETLINK_CB(skb_out).dst_group = 0; // not in mcast group
strncpy(nlmsg_data(nlh), msg, msg_size);
res = nlmsg_unicast(nl_sk, skb_out, pid);
if (res < 0)
printk(KERN_INFO "Error while sending bak to user\n");
}
static int __init hello_init(void){
struct netlink_kernel_cfg cfg = {
.input = hello_nl_recv_msg,
};
printk(KERN_INFO "Loading kernel module\n");
nl_sk = netlink_kernel_create(&init_net, NETLINK_USER, &cfg);
if (!nl_sk) {
printk(KERN_ALERT "Error creating socket.\n");
return -10;
}
return 0;
}
static void __exit hello_exit(void){
printk(KERN_INFO "exiting hello module\n");
netlink_kernel_release(nl_sk);
}
module_init(hello_init); module_exit(hello_exit);
MODULE_LICENSE("GPL");
And then the user-space program:
#include <sys/socket.h>
#include <linux/netlink.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#define NETLINK_USER 31
#define MAX_PAYLOAD 1024 /* maximum payload size*/
struct msghdr msg;
int main(){
struct sockaddr_nl src_addr, dest_addr;
struct nlmsghdr *nlh = NULL;
struct iovec iov;
int sock_fd;
sock_fd = socket(PF_NETLINK, SOCK_RAW, NETLINK_USER);
if (sock_fd < 0)
return -1;
memset(&src_addr, 0, sizeof(src_addr));
src_addr.nl_family = AF_NETLINK;
src_addr.nl_pid = getpid(); /* self pid */
bind(sock_fd, (struct sockaddr *)&src_addr, sizeof(src_addr));
memset(&dest_addr, 0, sizeof(dest_addr));
memset(&dest_addr, 0, sizeof(dest_addr));
dest_addr.nl_family = AF_NETLINK;
dest_addr.nl_pid = 0; /* For Linux Kernel */
dest_addr.nl_groups = 0; /* unicast */
nlh = (struct nlmsghdr *)malloc(NLMSG_SPACE(MAX_PAYLOAD));
memset(nlh, 0, NLMSG_SPACE(MAX_PAYLOAD));
nlh->nlmsg_len = NLMSG_SPACE(MAX_PAYLOAD);
nlh->nlmsg_pid = getpid();
nlh->nlmsg_flags = 0;
strcpy(NLMSG_DATA(nlh), "Hello");
iov.iov_base = (void *)nlh;
iov.iov_len = nlh->nlmsg_len;
msg.msg_name = (void *)&dest_addr;
msg.msg_namelen = sizeof(dest_addr);
msg.msg_iov = &iov;
msg.msg_iovlen = 1;
printf("Sending message to kernel\n");
sendmsg(sock_fd, &msg, 0);
printf("Waiting for message from kernel\n");
/* Read message from kernel */
recvmsg(sock_fd, &msg, 0);
printf("Received message payload: %s\n", (char*)NLMSG_DATA(nlh));
close(sock_fd);
return 0;
}
Now If i run this, all is fine and well and it gives me the console output:
Sending message to kernel
Waiting for message from kernel
Received message payload: Hello from kernel
and the kernel log output from dmesg:
[ 3160.679609] exiting hello module
[ 3165.140816] Loading kernel module
[ 3169.678258] Entering: hello_nl_recv_msg
[ 3169.678260] Netlink received msg payload:Hello
But for this project, we are using multi threaded applications calling the API, so I wanted to try and give each calling thread an own Netlink socket. To do that I had to make
struct msghdr msg;
into a locally declared variable.
Problems arise
When I moved it into the main function things instantly broke. This in the way that the kernel doesn't even enter the Netlink callback function, so I guess the user space program fails to even write to it, but it still returns the correct amount of written bytes from the sendmsg() function.
This is what's output to console when having the msghdr locally declared:
Sending message to kernel
Waiting for message from kernel
And then it hangs, and needs to be SIGINT'ed, and the kernel log doesn't show anything about the LKM receiving any data.
So I started wondering if it could be an addressing error happening when it was locally declared, so to try it out I converted the msghdr into a dynamically allocated pointer in the local scope, and you know what, it worked! It gave the same console and kernel log output as the original example.
Soooo, my actual question is really one for educational purpose and to understand why it is behaving in this way.
Why does a globally declared variable work, where a locally declared one does not?
And furthermore, why does a locally declared, dynamically allocated, pointer work?
Am I missing something on a fundamental level here?
TL;DR:
Why does it not work to have the msghdr struct locally declared in the user space program, while a globally declared or a local dynamic pointer does?
Maybe when it's on the stack, the memory for it isn't zeroed and you have garbage in some field.