How clock synchronization affects the performance of YugabyteDB

Question

[Question posted by a user on YugabyteDB Community Slack]

I would like to examine how the accuracy of clock synchronization affects the performance of YugabyteDB. How do you usually test such things internally?

Answer 1

To answer your question, we use Jepsen tests to verify correctness, which introduces clock skew issues (as a part of a larger set of failures it introduces). This report is old, but it explains the point: https://blog.yugabyte.com/yugabyte-db-1-2-passes-jepsen-testing/ (see “Pushing the boundary on tolerating clock skew”)

YugabyteDB uses a combination of the monotonic and the HLC to make sure it is much more resilient to clock skew. Note that the —max_clock_skew_usec config flag controls the maximum clock skew allowed between any two nodes in the cluster and has a default value of 50ms. Kyle calls this out in his report (emphasis is mine in these quotes): YugabyteDB uses Raft, which ensures linearizability for all (write) operations which go through Raft’s log. For performance reasons, YugabyteDB reads return the local state from any Raft leader immediately, using leader leases to ensure safety. Using CLOCK_MONOTONIC for leases (instead of CLOCK_REALTIME) insulates YugabyteDB from some classes of clock error, such as leap seconds. Between shards, YugabyteDB uses a complex scheme involving Hybrid Logical Clocks (HLCs). Yugabyte couples those clocks to the Raft log, writing HLC timestamps to log entries, and using those timestamps to advance the HLC on new leaders. This technique eliminates several places where poorly synchronized clocks could allow consistency violations.