How to schedule real time cyclic task?

Question

We are a team of bachelor students currently working on building a legged robot. At the moment our interface to the robot is written in python using an sdk from the master board we are using.

In order to communicate with the master board sdk, we need to send a command every millisecond.

To allow us to send tasks periodically, we have applied the rt-preempt patch to our linux kernel. (Ubuntu LTS 20.04, kernel 5.10.27-rt36)

We are very new to writing real time applications, and have run into some issues where our task sometimes will have a much smaller time step than specified. In the figure below we have plotted the time of each cycle of the while loop where the command is being sent to the sdk. (x axis is time in seconds and y axis is the elapsed time of an iteration, also in seconds) As seen in the plot, one step is much smaller than the rest. This seems to happen at the same exact time mark every time we run the script.

cyclic_task_plot

We set the priority of the entire script using:

pid = os.getpid()
sched = os.SCHED_FIFO
param = os.sched_param(98)
os.sched_setscheduler(pid, sched, param)

Our cyclic task looks like this:

dt is set to 0.001

while(_running):
    if direction:
        q = q + 0.0025
        if (q > np.pi/2).any():
            direction = False
    else:
        q = q - 0.0025
        if (q < -np.pi/2).any():
            direction = True

    master_board.track_reference(q, q_prime)


    #Terminate if duration has passed
    if (time.perf_counter() - program_start > duration):
        _running = False

    cycle_time = time.perf_counter() - cycle_start
    time.sleep(dt - cycle_time)
    cycle_start = time.perf_counter()

    timestep_end = time.perf_counter()
    time_per_timestep_array.append(timestep_end - timestep_start)
    timestep_start = time.perf_counter()

We suspect the issue has to do with the way we define the sleep amount. Cycle_time is meant to be the time that the calculations above time.sleep() takes, so that: sleep time + cycle time = 1ms. However, we are not sure how to properly do this, and we're struggling with finding resources on the subject.

How should one properly define a task such as this for a real time application?

We have quite loose requirements (several milliseconds), but it is very important to us that it is deterministic, as this is part of our thesis and we need to understand what is going on.

Any answers to our question or relevant resources are greatly appreciated.

Link to the full code: https://drive.google.com/drive/folders/12KE0EBaLc2rkTZK2FuX_goMF4MgWtknS?usp=sharing

Answer 1

    timestep_end = time.perf_counter()
    time_per_timestep_array.append(timestep_end - timestep_start)
    timestep_start = time.perf_counter()

You're recording the time between timestep_start from the previous cycle and timestep_end from the current cycle. This interval does not accurately represent the cycle time step (even if we assume that no task preemption takes place); it excludes the time consumed by the array append function. Since the outlier seems to happen at the same exact time mark every time we run the script, we could suspect that at this point the array exceeds a certain size where an expensive memory reallocation has to take place. Regardless of the real reason, you should remove such timing inaccuracies by recording the time between cycle starts:

    timestep_end = cycle_start
    time_per_timestep_array.append(timestep_end - timestep_start)
    timestep_start = cycle_start