Why we are seeing this ERROR:
The ERROR means your Regions server is unable to accept any more edits from the source, the write rate is higher than the flushes, and your “memstore” limit reached its max
You will be observing the below errors in the Hbase Regions server logs
ERROR org.apache.hadoop.hbase.replication.regionserver.ReplicationSink: Unable to accept edit because: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 43422 actions: ConnectionClosedException: 43422 times, servers with issues: at org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:54) org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 800 actions: org.apache.hadoop.hbase.RegionTooBusyException
“org.apache.hadoop.hbase.replication.regionserver.ReplicationSink” class accepts the edits from different sources and will replicate in the cluster.
The above Error messages in the regions server logs can impact performance and in the worst case will cause Region server failure
- As we can see the write is higher than the flush rate, Due to that Region servers are unable to handle the load
- To fix this issue, We need to tune the Region server property to make sure it can able handle the load
One of the recommendations would be to increase the “memstore” size to accommodate the load
Double the value from the currently configured value like 256 -> 512MB
Usually, We use to see this kind of issue, If we have Hbase replication enabled, So it would be wiser to validate if the Hbase replication is working as expected or not
Hbase shell status 'replication'
SOURCE: PeerID=2, AgeOfLastShippedOp=0, SizeOfLogQueue=682, TimeStampsOfLastShippedOp=Mon Jun 12 08:18:10 UTC 2022, Replication Lag=34234254 SINK : AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Sat Jun 12 10:23:12 UTC 2022
– Check the response for lag and the last shipped date, Lag should be a minimum value and the last shipped date should be the latest date.
Feel free to comment or start a discussion, If the above fix is not solving the issue
Good Luck with your learning!