Buffer underruns on x310 when transmitting and receiving from same channel

Question

I'm running on an x310 across dual 10 Gigabit ethernet, outfitted with twin basic tx rx daughterboards. I'm running on UHD version 3.11.0. Ideally, I would like two simultaneous transmit and receive streams, utilizing both channels to transmit and receive. I don't want to have to use 2 x310s for 2 receive and transmit streams

When I transmit and receive at the same time on the same channel, I get a lot of U's printed out to console signaling underflows, no matter what the rate. HOWEVER, if I transmit and receive on separate channels (tx_streamer has stream_args at channel 1 and rx_streamer has stream_args at channel 0) it works just fine.

I've attached the source code of a complete but simple program that will hopefully demonstrate my problem. In this program, two threads are created: a transmit thread and a receive thread. The receive thread is constantly receiving data to a buffer and overwriting that buffer with new data. The transmit thread is constantly transmitting 0's from a prefilled buffer.

If anyone has an x310 running across 10Gbps ethernet, can you compile and run my program to test if this problem occurs not just for me?

Here's what we have already tested:

• I'm running on a server system rocking two 12 core intel xeon processors. (https://ark.intel.com/products/91767/Intel-Xeon-Processor-E5-2650-v4-30M-Cache-2_20-GHz). My network card is the recommended x520 da2. Someone had previously suggested NUMA to be an issue, but I don't think this is the case as the program works when we switch to transmitting and receiving on separate channels.

• Since the program works just fine when we are transmitting and receiving on separate channels, I'm led to believe this is not a CPU power issue.

• I've tested only transmitting and only receiving. We can transmit at 200MS/s across both channels and we can receive at 200MS/s across both channels, but we cannot transmit and receive from the same channel. This suggests our network card is working properly and we can handle the high rate.

• I've tried my program on UHD 3.10.2 and the problem still occurs

• I've tried setting the tx_metadata waiting 2 second before transmitting. The problem still occurs.

• I've tried running the example program txrx_loopback_from_file and that works for simultaneous receive and transmit, but I have no idea why.

From the last point, I'm lead to believe that I am somehow calling the uhd API wrong, but I have no idea where the error is. Any help would be greatly appreciated.

Thanks,

Jason

#include <iostream>
#include <iomanip>
#include <stdlib.h>
#include <vector>
#include <csignal>
#include <thread>

#include <uhd/utils/thread_priority.hpp>
#include <uhd/utils/safe_main.hpp>
#include <uhd/usrp/multi_usrp.hpp>
#include <uhd/types/tune_request.hpp>

typedef std::complex<short> Complex;

// Constants and signal variables
static bool stop_signal_called = false;
const int NUM_CHANNELS = 1;
const int BUFF_SIZE = 64000;

//function prototypes here
void recvTask(Complex *buff, uhd::rx_streamer::sptr rx_stream);
void txTask(Complex *buff, uhd::tx_streamer::sptr tx_stream, uhd::tx_metadata_t md);
void sig_int_handler(int){
    std::cout << "Interrupt Signal Received" << std::endl;
    stop_signal_called = true;
}

int UHD_SAFE_MAIN(int argc, char *argv[]) {

    uhd::set_thread_priority_safe();

    //type=x300,addr=192.168.30.2,second_addr=192.168.40.2
    std::cout << std::endl;
    std::cout << boost::format("Creating the usrp device") << std::endl;
    uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make(std::string("type=x300,addr=192.168.30.2"));
    std::cout << std::endl;

    //set stream args
    uhd::stream_args_t stream_args("sc16"); 

    double samp_rate_tx = 10e6;
    double samp_rate_rx = 10e6;
    uhd::tune_request_t tune_request(0);

    //Lock mboard clocks
    usrp->set_clock_source(std::string("internal"));

    //set rx parameters
    usrp->set_rx_rate(samp_rate_rx);
    usrp->set_rx_freq(tune_request);
    usrp->set_rx_gain(0);

    //set tx parameters
    usrp->set_tx_rate(samp_rate_tx);
    usrp->set_tx_freq(tune_request);
    usrp->set_tx_gain(0);

    std::signal(SIGINT, &sig_int_handler);
    std::cout << "Press Ctrl + C to stop streaming..." << std::endl;

    //create buffers, 2 per channel (1 for tx, 1 for rx)
    // transmitting complex shorts -> typedef as Complex
    Complex *rx_buffs[NUM_CHANNELS];
    Complex *tx_buffs[NUM_CHANNELS];

    for (int i = 0; i < NUM_CHANNELS; i++){
        rx_buffs[i] = new Complex[BUFF_SIZE];
        tx_buffs[i] = new Complex[BUFF_SIZE];
        // only transmitting 0's 
        std::fill(tx_buffs[i], tx_buffs[i]+BUFF_SIZE, 0);
    }

//////////////////////////////////////////////////////////////////////////////
////////////////START RECEIVE AND TRANSMIT THREADS////////////////////////////
//////////////////////////////////////////////////////////////////////////////

    printf("setting up threading\n");

    //reset usrp time 
    usrp -> set_time_now(uhd::time_spec_t(0.0));

    // set up RX streams and threads
    std::thread rx_threads[NUM_CHANNELS];
    uhd::rx_streamer::sptr rx_streams[NUM_CHANNELS];
    for (int i = 0; i < NUM_CHANNELS; i++){
        stream_args.channels = std::vector<size_t>(1,i);
        rx_streams[i] = usrp->get_rx_stream(stream_args);
        //setup streaming
        auto stream_mode = uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS;
        uhd::stream_cmd_t stream_cmd(stream_mode);
        stream_cmd.num_samps = 0;
        stream_cmd.stream_now = true;
        stream_cmd.time_spec = uhd::time_spec_t();
        rx_streams[i]->issue_stream_cmd(stream_cmd);

        //start rx thread
        std::cout << "Starting rx thread " << i << std::endl;
        rx_threads[i] = std::thread(recvTask,rx_buffs[i],rx_streams[i]);
    }


    // set up TX streams and threads
    std::thread tx_threads[NUM_CHANNELS];
    uhd::tx_streamer::sptr tx_streams[NUM_CHANNELS];
    // set up TX metadata
    uhd::tx_metadata_t md;
    md.start_of_burst = true;
    md.end_of_burst = false;
    md.has_time_spec = true;
    // start transmitting 2 seconds later
    md.time_spec = uhd::time_spec_t(2); 

    for (int i = 0; i < NUM_CHANNELS; i++){
        //does not work when we transmit and receive on same channel, 
        //if we change to stream_args.channels = std::vector<size_t> (1,1), this works for 1 channel.
        stream_args.channels = std::vector<size_t>(1,i);                            
        tx_streams[i] = usrp->get_tx_stream(stream_args);

        //start the thread
        std::cout << "Starting tx thread " << i << std::endl;
        tx_threads[i] = std::thread(txTask,tx_buffs[i],tx_streams[i],md);
    }

    printf("Waiting to join threads\n");

    for (int i = 0; i < NUM_CHANNELS; i++){
        //join threads
        tx_threads[i].join();
        rx_threads[i].join();
    }
    return EXIT_SUCCESS;
}

//////////////////////////////////////////////////////////////////////////////
////////////////RECEIVE AND TRANSMIT THREAD FUNCTIONS/////////////////////////
//////////////////////////////////////////////////////////////////////////////

void recvTask(Complex *buff, uhd::rx_streamer::sptr rx_stream){
    uhd::rx_metadata_t md;

    unsigned overflows = 0;

    //receive loop
    while(!stop_signal_called){
        size_t amount_received = rx_stream->recv(buff,BUFF_SIZE,md,3.0);
        if (amount_received != BUFF_SIZE){ printf("receive not equal\n");}
        //handle the error codes
        switch(md.error_code){
        case uhd::rx_metadata_t::ERROR_CODE_NONE:
            break;

        case uhd::rx_metadata_t::ERROR_CODE_TIMEOUT:
            std::cerr << "T";
            continue;

        case uhd::rx_metadata_t::ERROR_CODE_OVERFLOW:
            overflows++;
            std::cerr << "Got an Overflow Indication" << std::endl;
            continue;

        default:
            std::cout << boost::format(
                "Got error code 0x%x, exiting loop..."
            ) % md.error_code << std::endl;
            goto done_loop;
        }
    } done_loop:

    // tell receive to stop streaming
    auto stream_cmd = uhd::stream_cmd_t(uhd::stream_cmd_t::STREAM_MODE_STOP_CONTINUOUS);
    rx_stream->issue_stream_cmd(stream_cmd);

    //finished
    std::cout << "Overflows=" << overflows << std::endl << std::endl;
}

void txTask(Complex *buff, uhd::tx_streamer::sptr tx_stream, uhd::tx_metadata_t md){

    //transmit loop
    while(!stop_signal_called){
        size_t samples_sent = tx_stream->send(buff,BUFF_SIZE,md);

        md.start_of_burst = false;
        md.has_time_spec = false;
    }

    //send a mini EOB packet 
    md.end_of_burst = true;
    tx_stream -> send("",0,md);

    printf("End transmit \n");  
}