I would like someone from Google to provide some guidelines on the durability and availability guarantees provided by the Cloud Bigtable service.
Here is my understanding so far:
The fact that the minimum cluster requires 3 nodes suggests that, at least within a zone, the data is highly durable and replicated to 3 nodes.
However, this answer by a Googler states that "Cloud Bigtable doesn’t replicate data" — directly contradicting the quote on the Cloud Bigtable homepage which claims it "is built with a replicated storage strategy". So which is it? Is it replicated or not? And if so, how many copies are kept?
The fact that clusters can only be set up within a particular zone suggests that the availability of a cluster is tied directly to the availability of that zone. So if I want to have a highly available Bigtable-based data storage, would it be best practice to set up independent clusters across multiple zones and handle the synchronisation of writes across the clusters myself?
There is no information on whether Bigtable clusters across zones are independent or not. If I were to set up clusters across multiple zones, and one zone goes down, could we expect the clusters in other zones to carry on working? Or is there some underlying single point of failure which could impact clusters even across zones?
Compared to the App Engine datastore which is very specific about these details, the Cloud Bigtable documentation is rather lacking — or, at least, I've not managed to find a page which goes into detail on these aspects.
The Cloud Bigtable docs are similarly vague on other aspects, e.g. on the matter of size limits for values, the documentation states that individual values should stay below "~10 MB per cell". What on earth does "~10 MB" mean?! Can I hardcode a limit of exactly 10MB and expect it to always work or will that change from day to day dependent on unknown factors?
Anyway, apologies if I sound agitated. I genuinely would like to use the Bigtable service. But I, like presumably many others, need to understand the durability/availability aspects of it before being able to invest in it. Thank you.
On replication: The answer you referenced is referring to replication of data across Bigtable Clusters, which is not supported at this time. (For example, a Bigtable Cluster in the United States replicating its writes to a second Cluster in Europe)
This concept is separate from replication of data within a Bigtable cluster, which is analogous to replication in HDFS, which is something that the product absolutely does today.
On availability: Yes, the availability of a Bigtable Cluster is tied to the availability of a Google Cloud Zone.
On Independence: Yes, Cloud Bigtable clusters are independent across zones. An outage in one zone should not impact the availability of other zones.
On data per cell: We do not reject writes >10Mb per cell, we have this set as a guideline for getting optimal performance.