Why does the pthread_barrier_wait does not take thread priorities into consideration?

Question

I am using barriers to synchronise my two threads which will perform task_1 and task_2 respectively.

After they are synchronised, I would like that the task with higher priority to start executing before the tasks with lower priorities.

I was surprised to notice that even though task_2 has a lower priority than task_1, sometimes task_2 starts executing before task_1.

#include <pthread.h>
#include <sched.h>
#include <unistd.h>
#include <errno.h>
#include <stdio.h>
#include <sys/types.h>

#define MAX_PRIORITY 99
#define MIN_PRIORITY 1

pthread_t threads[2];
pthread_barrier_t barrier;

void set_priority(int priority, int t_id){
    int policy = SCHED_FIFO;
    struct sched_param param;

    param.sched_priority = priority;

    pthread_attr_t attr;
    pthread_attr_init (&attr);

    pthread_setschedparam(pthread_self(), policy, &param);

    pthread_getschedparam(pthread_self(), &policy, &param);
}

int set_core(int core_id) {
    int num_cores = sysconf(_SC_NPROCESSORS_ONLN);
    if (core_id < 0 || core_id >= num_cores)
        return EINVAL;

    cpu_set_t cpuset;
    CPU_ZERO(&cpuset);
    CPU_SET(core_id, &cpuset);

    pthread_t current_thread = pthread_self();
    return pthread_setaffinity_np(current_thread, sizeof(cpu_set_t), &cpuset);
}
void create_task(int task_number, void *task) {
    int rc = pthread_create(&threads[task_number - 1], NULL, task, NULL);
    if(rc != 0) {
        printf("pthread_create(%d) error %d\n", task_number - 1, rc);
        pthread_exit(0) ;
    }
}

void schedule_task(int task_number, int priority) {
    set_core(2); //running tasks only in 2nd processor core
    set_priority(priority, task_number);
}

void start_task_1() {
    printf("Task 1 Started \n");
    sleep(1); //do task 1
    printf("Task 1 Endeded\n");
}

void start_task_2() {
    printf("Task 2 Started \n");
    sleep(1); //do task 2
    printf("Task 2 Endeded\n");
}

void task_1(void *thread_param) {
    schedule_task(1, MAX_PRIORITY);
    pthread_barrier_wait(&barrier);

    start_task_1();

    pthread_exit(NULL);
}

void task_2(void *thread_param) {
    schedule_task(2, MIN_PRIORITY);
    pthread_barrier_wait(&barrier);

    start_task_2();

    pthread_exit(NULL);
}

int main() {
    pthread_barrier_init(&barrier, NULL, 2);

    create_task(1, task_1);
    create_task(2, task_2);

    for (int i = 0; i < 2; i++) {
        pthread_join(threads[i], NULL);
    } 

    pthread_barrier_destroy(&barrier);
}

Is that an expected behaviour in POSIX threads? What can I do to enforce that task_1 always starts before task_2?

I am executing the program as root, and I have ensured that the tasks priorities have been set accordingly.

Answer 1

Scheduling is dependent on the scheduling algorithm being used in the OS. You are trying to use FIFO which as per manpage is available only when real-time scheduling is enabled.

   Various "real-time" policies are also supported, for special time-
   critical applications that need precise control over the way in which
   runnable threads are selected for execution.  For the rules governing
   when a process may use these policies, see sched(7).  The real-time
   policies that may be specified in policy are:

        SCHED_FIFO    a first-in, first-out policy; and

        SCHED_RR      a round-robin policy.

If you are using Linux, scheduling algorithms are selected during kernel compile time, not sure if it is possible to change them during boot time or dynamically.

Answer 2

You're creating two different threads, and restricting them to separate cores. It's true that one of them has a higher priority setting than the other, but both are SCHED_FIFO threads which likely makes each one the highest priority task on its respective core (at least among user-level tasks). Since each core has a separate runnable task queue, the scheduling policy would only affect the comparison among tasks on the same core.

So, for all practical purposes, the two tasks are actually at the same effective priority. Hence, there is no reason to expect that task 1 will start executing any sooner than task 2 following the barrier wait. If you want to enforce that (while still having them run on separate cores), you'll need to use some other shared variable to have task 1 tell task 2 it's okay to start. That could be a semaphore, mutex-protected flag, or on most architectures, simply a busy loop on an atomically updated flag integer.