We have this HBase cluster: 30+ nodes, 48 tables, 40+TB on HDFS level, replication factor 2. Due to disk failure on two nodes, we have a corrupt file on HDFS.
Current HDFS status
Excerpt of hdfs fsck /
output, which shows a corrupt HBase region file:
/user/hbase/table_foo_bar/295cff9c67379c1204a6ddd15808af0b/n/ae0fdf7d0fa24ad1914ca934d3493e56:
CORRUPT blockpool BP-323062689-192.168.12.45-1357244568924 block blk_9209554458788732793
/user/hbase/table_foo_bar/295cff9c67379c1204a6ddd15808af0b/n/ae0fdf7d0fa24ad1914ca934d3493e56:
MISSING 1 blocks of total size 134217728 B
CORRUPT FILES: 1
MISSING BLOCKS: 1
MISSING SIZE: 134217728 B
CORRUPT BLOCKS: 1
The filesystem under path '/' is CORRUPT
The lost data is not recoverable (the disks are broken).
Current HBase status
According to HBase on the other hand, everything is fine and dandy
hbase hbck
says:
Version: 0.94.6-cdh4.4.0
...
table_foo_bar is okay.
Number of regions: 1425
Deployed on: ....
...
0 inconsistencies detected.
Status: OK
Moreover, it seems that we can still query data from the non-lost blocks of the corrupt region file (as far as I think I was able to check based on the start and end row key of the region).
Next steps
- Because the file block data is not recoverable, it seems the only option is to remove the complete corrupt file (with
hadoop fs -rm
orhadoop fsck -delete /
). This will "fix" corruption at the HDFS level. - However, I'm afraid removing the HDFS file will introduce corruption at the HBase level as a complete region file will be gone
- I considered
hadoop fsck -move /
to move the corrupt file to/lost+found
and see how HBase would take that, but moving to/lost+found
is not as reversible as it seems, so I'm hesitant about that as well
Concrete questions:
Should I just remove the file? (Losing the data corresponding to that region is reasonably fine for us.) What bad things happen when you manually remove a HBase region file in HDFS? Does it just remove the data or would it introduce ugly metadata corruption in HBase that also have to be taken care of?
Or can we actually leave the situation as-is, which seems to work at the moment (HBase is not complaining about/seeing corruption)?
FYI: I decided to bite the bullet and manually deleted the corrupt file from HDFS with:
(
hdfs fsck -move
did not work for me, not sure why)After that, I checked HBase's health with
hbck
, but no inconsistencies were detectedSo in our case the manual deletion of the region file did not introduce HBase corruption, if I understand correctly, which is nice, but confusing. (I hope this does not backfire and the corruption doesn't manifest itself at a later point in time)
issue closed
Your mileage may vary.