Resolve “hole in the region chain” issue in Hbase

In Hbase, Table data are split into multiple regions and managed by region servers. The region consists of rows between the start key and the end key.

Each region’s start key and end key of the previous region should be the same and if there is any gap found between the start key and end key it is considered a hole in the regions

Example of a regions (without any hole)

REGION_NAME	START_KEY	END_KEY
example_table,,1339480003942.c398716bcd07s847ghe00d613eaf07.	​	06666666
example_table,06666666,1339480003942.d398234bcd07k34jfm30d613eaf07.	06666666	08888899
​example_table,08888899,1339480003942.2adjdkj22ew9560ae2211186a7095w.	08888899	11111109

Example of regions (with hole)

REGION_NAME	START_KEY	END_KEY
example_table,06666666,1339480003942.c3e338a42891e3cenvew338e73e8fd8e3.	34433330	45500441
example_table,08888899,1339480003942.d210a1ce0373cfd7233d419323522e87.	46666832	55550044

For any adjacent regions the start key and end key of the previous regions should be the same else it is called a hole in the region chains

To find the ERROR message

Capture the hbck output and search for the keyword “hole“,

hbase hbck -details | grep ERROR | grep hole

ERROR: There is a hole in the region chain between 34433330 and 88888832.  You need to create a new .regioninfo and region dir in hdfs to plug the hole.

To Find the exact regions with hole

Get the list of regions for the given table as below

hbase> list_regions <tablename>

REGION_NAME	START_KEY	END_KEY
example_table,34433330,1339480003942.c3e338a42891e3cenvew338e73e8fd8e3.	34433330	45500441
example_table,46666832,1339480003942.d210a1ce0373cfd7233d419323522e87.	46666832	55550044
​example_table,55550044,1339480003942.d45f9a42891e3ciwo435338e73e8gd9se.	55550044	56664664

Here we see hole in the region chain. First one, the region (c3e338a42891e3cenvew338e73e8fd8e3) END_KEY and next region (d210a1ce0373cfd7233d419323522e87) START_KEY don’t match and there is a gap.

To Fix this issue

We can merge these regions between holes.

hbase> merge_region ‘c3e338a42891e3cenvew338e73e8fd8e3′,’d210a1ce0373cfd7233d419323522e87’, true

REGION_NAME	START_KEY	END_KEY
example_table,06666666,1339480003942.c3e338a42891e3cenvew338e73e8fd8e3.	34433330	55550044
example_table,55550044,1339480003942.d45f9a42891e3ciwo435338e73e8gd9se.	55550044	56664664

As we can see 2 regions are merged into a single regions and there is No holes after the merge