I am developing a distributed database system, suppose each table has 3 copies on different machines, if a write request comes in, the replication layer's work is to replicate that write data to other to nodes. The question is how to make the replication efficient in terms of throughput? The network bandwidth is 10Gbps.
Example:
cmd1: write on table 1
-> send to node A and node B
cmd2: write on table 2
-> send to node B and node C
cmd3: write on table 1
-> send to node A and node B
cmd4: write on table 3
-> send to node B and node D
cmd5: write on table 2
-> send to node B and node C
The above example shows the replicating command queues. To achieve high performance, what about use mulit-thread network replication? But if there are two writes to the same node, it would be better to combine the two and send altogether?
And suppose those writes will be stored on disks eventually (think of spinning disk), so to speed up the storing process, is there anything can be done when doing the replication? (if each table has been mapped to store on a specific disk on a node, writes are append-only).