Resolve “Job aborted due to stage failure” in Spark
When it comes to troubleshooting Spark issues. One thing you get used to it is knowing what the error exactly…
Spark
Blogs post related to Apache Spark
Resolve “Could not find CoarseGrainedScheduler” in Spark
In this article, we will understand and learn about the CoarseGrainedScheduler and why we are encountering this error in the…
FIX – TypeError: an integer is required (got type bytes)
In this article, we will learn about the “TypeError: an integer is required (got type bytes)” that occurs in PySpark…
How to Save DataFrame as a CSV File in Spark
Spark provides a lot of APIs to save DataFrame to multiple formats like CSV, Parquet, Hive tables, etc. In this…
How to Save Spark DataFrame directly to Hive
I hope you have encountered a similar situation, Where you wanted to do some manipulation on a spark dataframe and…
What are broadcast variables in Spark
Broadcast variables are commonly used by Spark developers to optimize their code for better performance. This article will provide a…
Resolve the “Container killed by YARN for exceeding memory limits” Issue in Hive, Tez, and Spark jobs
“Container killed by YARN for exceeding memory limits” usually happens, When the JVM usage goes beyond the Yarn container memory…
Why Spark/MR not considering UTF-8 encoding
Reading/WRITING UTF-8 enabled file Sometimes, we could have encountered issues in which Spark returns non-ASCII characters in the wrong format….
How to read and write Excel files with Spark
Apache Spark is a powerful data processing framework, Commonly, Spark is used to process data stored in various formats, including…
Difference between groupByKey and reduceByKey in Spark
groupByKey and reduceByKey are the two different operations that help to transform RDD (Resilient Distributed Datasets). What is the difference…
Understanding the Spark stack function for pivoting data
Hello! If you’re into big data processing, you’ve probably heard of Spark, right? It’s a popular distributed computing framework used…
How to set Apache Spark Executor and Driver Memory/Cores ( pyspark and spark-submit)
In multiple cases, We need to increase the Driver/executors memory/cores to improve performance or to avoid Out of Memory issues
Spark Driver in Apache Spark and Where does the spark driver run?
Drivers are the one that starts the spark context or session in Spark, which helps in communicating with resource managers and runs tasks in
How to Enable Debug Mode in spark-submit, spark-shell, and pyspark
We often need to enable debug log level in the spark to understand the issue and troubleshoot, In this article,…
What is the difference between apache spark and pyspark
Have you been wondering what the difference is between Apache Spark and Pyspark, and which one to use for big…
Script to collect thread dump (Jstack)
Jstack is a command line tool that helps to capture the thread dump of the java process. Using the thread…
Handling Data Skew in Apache Spark Application
What is Data skew? Let’s take a basic example of “CONSTRUCTION WORKERS“ In the above example: Skew happened due to…
How to Run the Spark history server locally
Sharing a step-by-step guide to the setup of the Spark history server locally (Mac or Windows). This helps to debug…
Difference between DataFrame, Dataset, and RDD in Spark
Short History of Spark: — Spark was created in Berkeley back in 2009 — An evolution of the MapReduce concepts…
What is the difference between Cache and Checkpoint in Spark
Spark is a data processing framework that helps to process data faster. It uses in-memory and multiple nodes to run…
Resolve “org.apache.hadoop.hive.serde2.SerDeException: Unexpected tag” in Spark and Hive
We usually see the ERROR “org.apache.hadoop.hive.serde2.SerDeException: Unexpected tag” in Spark, When you are trying to connect the hive…
Resolve “Task serialization failed: java.lang.StackOverflowError” in Spark
“Task serialization failed: java.lang.StackOverflowError” usually happens, When the JVM encounters a situation where it is unable to create a…
Total size of serialized results of tasks (1024.5 MB) is bigger than spark.driver.maxResultSize
“failure: Total size of serialized results of x tasks (1024.5 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)” in Spark
How to Enable Kerberos Debugging for Spark Application and “hdfs dfs” Command
Kerberos debugging involves enabling debug log level for the Krb5LoginModule module at the JVM level, This would help us to…
How to Kill Running Yarn Application ( Spark, Hive, and Tez)
One of the easiest ways to kill a Spark application is by issuing the “yarn kill” command
spark.driver.memoryOverhead and spark.executor.memoryOverhead explained
In this article, We will learn about memory overhead configuration in spark and explore more about spark.driver.memoryOverhead & spark.executor.memoryOverhead and…
Yarn application stuck in the ACCEPTED state (Includes Spark, Hive, Tez, and MapReduce jobs)
Usually, the Yarn application will stuck in the ACCEPTED state, When it didn’t find enough resources to create a new container in the cluste
“Futures timed out” issue in spark
“Futures timed out” is a common error that can occur when running Spark applications. In this article, We will learn,…
How to Access Kudu table from Spark
There are multiple use cases, Where we need to access Kudu from spark to store and retrieve data, In this…
How to run Spark job with Ozone Filesystem
Apache Spark is a popular distributed computing framework for big data processing and Ozone is a distributed object store that…
Resolve the “User application exited with status 1” issue in Spark
“User application exited” means an application has stopped running and it can happen for multiple reasons. In specific, We are…
How to Access HBase from Spark
As we all know, Spark is an open-source, distributed processing framework used in big data, It helps perform analytics on…
Difference between map and flatMap in Spark
Apache Spark is a powerful distributed framework that leverages in-memory caching and optimized query execution to produce faster results. The…
How to read and write XML files using Spark?
Spark is a powerful framework for processing large datasets in a distributed manner. In this article, we will discuss, how…
Difference between map and mapValues functions in Spark
Spark is a distributed framework, Which uses in-memory computation power to process a large volume of data much faster. One…
Resolve the “java.lang.OutOfMemoryError: Java heap space” issue in Spark and Hive(Tez and MR)
OutOfMemoryError is not a surprise for spark as it is a memory-centric framework, To deal with memory issues, We need…
Resolve the “container exited with a non-zero exit code 143” issue in Spark, Hive, and Tez
exit code 143 usually, happens if the container is going Out of Memory (OOM) and unable to handle the large data set