RDMA Read protection for local memory operations

Question

I have the following scenario: My server allocates a buffer of 1MB, which is periodically updated and written to (about every 50ms). The client is connected to the the server via Infiniband and periodically reads that buffer via RDMA Read (potentially even faster than it is updated).

My question is: Is there any way to ensure that the local write operation is atomic so that the RDMA read can only read valid and contiguous memory?

Did I correctly understand that the memory regions are only protected from parallel RDMA operations, but not local memory operations? Is there any way to accomplish this behaviour natively with the ibverbs API?

Answer 1

The answer may depend on the device. In RDMA Verbs, there are two different modes for atomic operation (which can be checked using the ibv_query_device verb). With IBV_ATOMIC_HCA, atomic operations are only atomic with respect to other operations from the same device, while IBV_ATOMIC_GLOB means that they are atomic also with respect to CPU operations and other RDMA devices.

If a device uses IBV_ATOMIC_HCA, you need to synchronize the RDMA reads with the CPU writes to ensure data consistency.