Resolve “TTransportException: MaxMessageSize reached” Hive
Symptoms
The “TTransportException: MaxMessageSize reached” exception will cause hive query failures when it tries to access a table that has a huge number of partitions. This will occurs irrespective of the clients like a spark or other JDBC, ODBC connections
0: jdbc:hive2://<Hiveserver2>> select count(*) from test_db.test_table; Error: Error while compiling statement: FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.transport.TTransportException: MaxMessageSize reached
Cause
By default, MaxMessageSize in Thrift is 100MB in the earlier version and that is being encountered below when HS2 makes the get_partitions_by_expr() call to HMS and receives a large number of partitions.
This usually happens, When there where many partitions with more columns and high columns string size.
Replicating the Issue
The issue can be replicated by creating a hive table with more than 10k partitions with more columns. The idea is to exceed the details MaxMessageSize in Thrift
ERROR Message: HS2 logs
WARN org.apache.hadoop.hive.metastore.RetryingMetaStoreClient: [11absdf-6347-47dd-c324-7e1edfsdf34338 HiveServer2-Handler-Pool: Thread-381224]: MetaStoreClient lost connection. Attempting to reconnect (1 of 1) after 1s. listPartitionsByExpr
org.apache.thrift.transport.TTransportException: MaxMessageSize reached
at org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96) ~
at org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97) ~
at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:390) ~
at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39) ~
Solution:
Due to the above Exception, There has been Jira created “Make thrift max message size configurable” to make the MaxMessageSize configurable, Which is resolved recently
Now the default value for the below property has been bumped to 1GB
hive.metastore.server.max.message.size hive.server2.thrift.max.message.size
Vanilla Hadoop
There are cases, Where you will be hitting the same issue even after having the default value as 1 GB. At that time, We can able to configure the below values in the hive-site.xml. This hive-site.xml has to be updated across the cluster node
hive.thrift.client.max.message.size=2147483647 metastore.thrift.client.max.message.size=2147483647
NOTE: 2GB is the upper limit for the above property, It should not be configured more than 2GB. To exact we need to configure the value as “2147483647” which is 2GB -1byte (To avoid overflow error)
Cloudera Distribution
If you are using Cloudera as a distribution, Then this value can be easily configured from the Cloudera manager Webui as below
Set the following configuration in bytes on both Hive & Hive-on-tez
Navigate as below in the Cloudera manager and update the property
CM > Hive & Hive-on-tez > Configuration > Hive Client Advanced Configuration Snippet (Safety Valve) for hive-site.xml CM > Hive & Hive On Tez > Configuration > Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml hive.thrift.client.max.message.size=2147483647 metastore.thrift.client.max.message.size=2147483647
NOTE: Value should be in bytes and not in GB
Set the following conf in Hive-on-tez
Navigate as below
CM > Hive-on-tez > Configuration > Hive Client Advanced Configuration Snippet (Safety Valve) for hive-site.xml CM > Hive On Tez > Configuration > Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml hive.plan.mapwork.serialization.skip.properties=impala_intermediate_stats_chunk.*
This helped to resolve the issue in my case, Please do try it out and comment below
Good Luck with your Learning !!
Related Topics:
How to Create a Hive table with Ozone Filesystem
Resolve “MetaException(message:Timeout when executing method: get_partitions” ERROR in Hive
Resolve “org.apache.hadoop.hive.ql.lockmgr.LockException(No record of transaction could be found)”
