Resolve the “Master startup cannot progress” issue in Hbase
Symptom:
During Hbase service Startup/Restart, The Hbase master fails to initialize, Running any Hbase-related command would give the below error messages.
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
There are some cases, Where read might work (Like list, scan, etc), but the write will fail with the above messages. This is because the standby Master would have been working correctly (Only in a case where hbase:meta regions are assigned and hbase:namespace regions failed to assign)
ERROR snippet:
When Hbase:namespace is not online
WARN org.apache.hadoop.hbase.master.HMaster: hbase:namespace,,1619324234259.3249a4a942bf6b562134a7de50f1bg4d34. is NOT online; state={3249a4a942bf6b562134a7de50f1bg4d34 state=OPENING, ts=1619324234259, server=<Region server host>,16020,1619324234259}; ServerCrashProcedures=true. Master startup cannot progress,in holding-pattern until region onlined.
To find the region id for Hbase:namespace
hbase:namespace,,1619324234259.3249a4a942bf6b562134a7de50f1bg4d34. is NOT online;
region id: 3249a4a942bf6b562134a7de50f1bg4d34
When Hbase:meta is not online
WARN org.apache.hadoop.hbase.master.HMaster: hbase:meta,,1.1699230890 is NOT online; state={1699230890 state=OPENING, ts=1632436940, server=,16020,133732342343}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined.
INFO org.apache.zookeeper.ZooKeeper: Session: 0xff8453461c0384 closed
INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
To find region id for Hbase:meta
org.apache.hadoop.hbase.master.HMaster: hbase:meta,,1.1699230890
region id = 1699230890
Root Cause:
There is a high chance the Hbase namespace table or meta table has not been appropriately assigned. Please make sure by validating the HBase Master UI page -> system tables
If the Hbase master WebUI is not up – Check the Master server logs for the sentence “Master startup cannot progress”
Solution:
We can assign the regions manually using the hbck2 tool
hbase hbck -j <hbck2>.jar assigns -o <regionId>
Ref: https://github.com/apache/hbase-operator-tools/blob/master/hbase-hbck2/README.md
Please feel free to add your points and chat with us if you need any information.
