I am reading the Designing Data Intensive Application book. In chapter 8, it discusses the use of Fencing Token for preventing concurrent writes to a network file.
The mechanism states that the a Lock service can give out Fencing Tokens, and the Storage Node checks it and rejects lower tokens given by a node whose lock lease has expired. In the particular example, it talked about the case where the node experienced a long GC pause, then tries to write to storage using a stale fencing token.
I am curious about a scenario where the node sends a fencing token which is accepted by the storage node, and writes some data into the storage, then it experienced a long pause which causes the lease to expire. In such a case, would this already leave the network file in a corrupted state? If so, how can this be prevented?
I guess a similar question in nature is - what happens when distributed lock lease expires while a resource is being modified? Does the client automatically extend the lease?
Thanks!
The receiving system has to be aware about fencing tokens; a network file, or ftp, or any other "general access" resource won't be able to deal with the fencing token.
From your question: "the node sends a fencing token which is accepted by the storage node, and writes some data into the storage, then it experienced a long pause which causes the lease to expire" - who is experiencing the long pause?
If it's the storage node - that should be ok, as storage node should be designed around this problem.
If it's the sender node, then the fencing token is the signal for the storage node, that those requests (with older fencing tokens) has to be ignored.
Since many systems don't support fencing tokens, some literature recommends to use "Red Lock algorithm". But Martin Kleppmann (the author of Designing Data Intensive apps) argues that Red Lock is not a correct algorithm.