Resolve “MetaException(message:Timeout when executing method: get_partitions” ERROR in Hive

“MetaException(message:Timeout when executing method: get_partitions” Occurs, When your query unable to get the meta information from the HiveMetaStore server within the configured timeout setting

By default, it is set to 600000ms => Hive.metastore.client.socket.timeout =600000ms

Impact:

This issue will cause query/job failure

Symptom:

Can able to observer the below error message in the Console or driver logs

ERROR hive.ql.metadata.Hive  - MetaException(message:Timeout when executing method: get_partitions; 129000 exceeds 600000ms)

Troubleshooting:

Usually, these errors occur due to 2 reasons, I have come across

• When the HiveMetaStore is overloaded
• When the table meta size is enormous

When the HiveMetaStore (HMS) is overloaded

In some cases, even for small tables, the HMS is unable to get the meta information back within the timeout value.

In such cases, you will be seeing the below WARN in the HS2 logs

WARN  [PrivilegeSynchronizer]: metastore.RetryingMetaStoreClient (:()) - MetaStoreClient lost connection. Attempting to reconnect (24 of 24) after 5s. getTable
org.apache.thrift.transport.TTransportException: null
	at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]

This shows the Hive Metastore was overloaded and hence there was not enough pool available for queries to run

Ideally, We need to tune the HMS to accommodate more queries and for better performance

When the table meta size is enormous:

This would be the typical case for this type of problem, Double check the size of the table and the partition data

If you are seeing a huge number of partitions (Which result in metadata size huge), Then increasing the timeout would be the efficient way as it requires more time to fetch the meta information from the backend database

Resolution :

When the HiveMetaStore (HMS) is overloaded

In this case, We need to tune the HMS property to increase the power to handle multiple requests

Try increasing the below property in HIVE

datanucleus.connectionPool.maxPoolSize
Hive.server2.async.exec.threads
hive.server2.async.exec.wait.queue.size
Hive.server2.thrift.max.worker.threads

NOTE: These are properties that help to handle the request efficiently, But they based on the Load on your system, Do a few trials and errors to achieve the ideal value

When the table meta size is enormous

When the table meta size is enormous and has multiple partitions, then it would be ideal for increasing the timeout, As it needs that much time to get the meta information back

hive.metastore.client.socket.timeout

600000ms -> 180000ms

For Spark:

If in case, you are using spark engine to run queries on top of this hive table and getting the same error

ERROR hive.ql.metadata.Hive  - MetaException(message:Timeout when executing method: get_partitions; 129000 exceeds 600000ms)

Then we need to pass the same property via spark command as below

spark-submit --conf spark.hadoop.hive.metastore.client.socket.timeout=1200

Good Luck with your Learning !!!