Resolve "org.apache.hadoop.hbase.replication.regionserver.ReplicationSink: Unable to accept edit"

Table of Contents

Why we are seeing this ERROR:

The ERROR means your Regions server is unable to accept any more edits from the source, the write rate is higher than the flushes, and your “memstore” limit reached its max

You will be observing the below errors in the Hbase Regions server logs

ERROR org.apache.hadoop.hbase.replication.regionserver.ReplicationSink: Unable to accept edit because:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 43422 actions: ConnectionClosedException: 43422 times, servers with issues: 
        at org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:54)
        
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 800 actions: org.apache.hadoop.hbase.RegionTooBusyException

“org.apache.hadoop.hbase.replication.regionserver.ReplicationSink” class accepts the edits from different sources and will replicate in the cluster.

Symptoms:

The above Error messages in the regions server logs can impact performance and in the worst case will cause Region server failure

Resolution:

As we can see the write is higher than the flush rate, Due to that Region servers are unable to handle the load
To fix this issue, We need to tune the Region server property to make sure it can able handle the load

One of the recommendations would be to increase the “memstore” size to accommodate the load

hbase.hregion.memstore.flush.size
hbase.hregion.memstore.block.multiplier

Double the value from the currently configured value like 256 -> 512MB

Usually, We use to see this kind of issue, If we have Hbase replication enabled, So it would be wiser to validate if the Hbase replication is working as expected or not

Hbase shell 
status 'replication'

SOURCE: PeerID=2, AgeOfLastShippedOp=0, SizeOfLogQueue=682, TimeStampsOfLastShippedOp=Mon Jun 12 08:18:10 UTC 2022, Replication Lag=34234254
SINK  : AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Sat Jun 12 10:23:12 UTC 2022

– Check the response for lag and the last shipped date, Lag should be a minimum value and the last shipped date should be the latest date.

Feel free to comment or start a discussion, If the above fix is not solving the issue