“Memory limit exceeded” usually, happens When the query reached its max memory limit and is unable to allocate any more memory
Memory limit exceeded: Failed to allocate row batch EXCHANGE_NODE (id=897) could not allocate 11.00 KB without exceeding limit. Error occurred on backend server :22000 Memory left in process limit: 231.43 GB Memory left in query limit: -112.25 KB
(Similar issue with respect to memory and the resolution would be the same)
memory limit exceeded. Limit=256.00 MB Consumption=293.21 MB
- We need to check the Impala query profile and Coordinator daemon logs to find where exactly it failed with the Out of Memory issue
- In this case, it failed in the “EXCHANGE_NODE” step, Where it is unable to allocate any more memory to that query, But the impala process still has enough memory left in that node
Memory left in process limit: 231.43 GB
Memory left in query limit: -112.25 KB
- This shows that the Impala process still has ~231GB left and can be allocated to the query limit
This will cause Impala query failure
To resolve this problem, We need to increase the mem_limit for that query or to the impala queue (default mem_limit)
Additional points to be noted:
- Make sure you have proper table stats available for all the tables involved in the query else run the below command before the actual query
- Gathering stats Would help Impala to create a more optimized query plan, which results in better performance.
Good Luck with your Learning !!