- I used hbase hbck and got such error:
ERROR: Region { meta => t1002,036,1707080843568.3b36c407cad3bcd56ab04a8c7f30a749., hdfs => hdfs://hacluster/hbase/data/default/t1002/3b36c407cad3bcd56ab04a8c7f30a749, deployed => , replicaId => 0 } not deployed on any region server. ERROR: There is a hole in the region chain between 036 and 042. You need to create a new .regioninfo and region dir in hdfs to plug the hole. ERROR: Found inconsistency in table t1002
- Then I tried to fix it with hbase hbck -fixAssignments t1002 But it is still inconsistent:
2024-02-14 16:13:46,418 INFO [hbasefsck-pool1-t36] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 3b36c407cad3bcd56ab04a8c7f30a749, NAME => 't1002,036,1707080843568.3b36c407cad3bcd56ab04a8c7f30a749.', STARTKEY => '036', ENDKEY => '042'} 2024-02-14 16:13:47,525 INFO [hbasefsck-pool1-t36] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 3b36c407cad3bcd56ab04a8c7f30a749, NAME => 't1002,036,1707080843568.3b36c407cad3bcd56ab04a8c7f30a749.', STARTKEY => '036', ENDKEY => '042'}
2024-02-14 16:14:47,450 WARN [hbasefsck-pool1-t36] util.HBaseFsck: Unable to complete check or repair the region 't1002,036,1707080843568.3b36c407cad3bcd56ab04a8c7f30a749.'. java.io.IOException: Region {ENCODED => 3b36c407cad3bcd56ab04a8c7f30a749, NAME => 't1002,036,1707080843568.3b36c407cad3bcd56ab04a8c7f30a749.', STARTKEY => '036', ENDKEY => '042'} failed to move out of transition within timeout 120000ms at org.apache.hadoop.hbase.util.HBaseFsckRepair.waitUntilAssigned(HBaseFsckRepair.java:149) at org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:2223) at org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:2435) at org.apache.hadoop.hbase.util.HBaseFsck.access$1200(HBaseFsck.java:203) at org.apache.hadoop.hbase.util.HBaseFsck$CheckRegionConsistencyWorkItem.call(HBaseFsck.java:1994) at org.apache.hadoop.hbase.util.HBaseFsck$CheckRegionConsistencyWorkItem.call(HBaseFsck.java:1982) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2024-02-14 16:14:47,454 WARN [hbasefsck-pool1-t36] util.HBaseFsck: Skip region 't1002,036,1707080843568.3b36c407cad3bcd56ab04a8c7f30a749.' 2024-02-14 16:14:47,493 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService 2024-02-14 16:14:47,493 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0xeb0113b8423ca6f3 2024-02-14 16:14:47,500 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0xeb0113b8423ca6f3 2024-02-14 16:14:47,500 INFO [main] zookeeper.ZooKeeper: Session: 0xeb0113b8423ca6f3 closed Exception in thread "main" java.io.IOException: 1 region(s) could not be checked or repaired. See logs for detail. at org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1948) at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:697) at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:731) at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:5198) at org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:4961) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:4949) 2024-02-14 16:14:48,097 INFO [Thread-10] util.HBaseFsck: Finishing hbck
How to resolve this problem, what need to do further?