What is the difference between apache spark and pyspark
Have you been wondering what the difference is between Apache Spark and Pyspark, and which one to use for big data processing? In this article, we will be discussing this in more detail, along with real-time performance test results.
Apache Spark is a distributed computing framework, originally accessible using the Scala programming language. PySpark is the interface that gives access to the Spark framework using the Python programming language. Both platforms offer a wide range of libraries and tools for big data processing
Introduction
Apache Spark is an open-source, general-purpose distributed computing framework designed to process large volumes of data in a distributed and parallelized manner. It can handle batch processing, stream processing, machine learning, and graph processing workloads. Spark has a core engine that manages the distributed processing, and it comes with several libraries for various tasks such as SQL, MLlib, GraphX, and Streaming.
PySpark, on the other hand, is a Python API for Spark that allows developers to write Spark applications using Python. It provides a Python interface to Spark’s core distributed processing engine and libraries, enabling Python developers to leverage Spark’s distributed computing capabilities without having to learn a new language.
Differences between Apache Spark and PySpark
Here are the key differences between Apache Spark and PySpark:
Programming Language
One of the most obvious differences between Apache Spark and PySpark is the programming language. Spark is written in Scala, and its API is available in Scala, Java, Python, R, and SQL. PySpark, as the name suggests, is a Python API for Spark that allows developers to write Spark applications using Python.
# spark-shell Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_141) Type in expressions to have them evaluated. Type :help for more information. scala>
# pyspark Using Python version 2.7.5 (default, Oct 14 2020 14:45:30) SparkSession available as 'spark'. >>>
Ease of Use
PySpark is often considered easier to use than Spark, especially for Python developers who are not familiar with Scala or Java. PySpark’s Pythonic API makes it easy to write Spark applications in Python without having to learn a new language.
Performance
In terms of performance, there is no significant difference between Spark and PySpark, as PySpark simply provides a Python API for Spark’s core distributed processing engine. However, some operations may be slower in Python than in Scala or Java, which could affect the performance of PySpark applications.
Libraries
Both Spark and PySpark come with several libraries for various tasks, such as SQL, MLlib, GraphX, and Streaming. These libraries are available in both languages, but PySpark’s API makes it easier to use them in Python.
Examples
To illustrate the difference between Apache Spark and PySpark, let’s take a look at some examples.
Word Count program to compare spark’s scala API vs python API
Example of word count using Spark’s Scala API:
val text = sc.textFile("/tmp/input_1GB.txt")
val words = text.flatMap(line => line.split(" "))
val wordCounts = words.map(word => (word, 1)).reduceByKey(_ + _)
wordCounts.saveAsTextFile("/tmp/output.txt")
# spark-shell --num-executors 5 --executor-memory 4g --executor-cores 3
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/03/06 12:20:40 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered!
Spark context Web UI available at http://<hostname>:4040
Spark context available as 'sc' (master = yarn, app id = application_1678076864502_0003).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.0
/_/
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_141)
Type in expressions to have them evaluated.
Type :help for more information.
scala> val text = sc.textFile("/tmp/input_1GB.txt")
text: org.apache.spark.rdd.RDD[String] = /tmp/input_1GB.txt MapPartitionsRDD[1] at textFile at <console>:24
scala> val words = text.flatMap(line => line.split(" "))
words: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[2] at flatMap at <console>:25
scala> val wordCounts = words.map(word => (word, 1)).reduceByKey(_ + _)
wordCounts: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[4] at reduceByKey at <console>:25
scala> wordCounts.saveAsTextFile("/tmp/output.txt")Example of word count using PySpark’s Python API:
from pyspark import SparkContext
text = sc.textFile("/tmp/input_1GB.txt")
words = text.flatMap(lambda line: line.split(" "))
wordCounts = words.map(lambda word: (word, 1)).reduceByKey(lambda a, b: a + b)
wordCounts.saveAsTextFile("/tmp/output_py.txt")
# pyspark --num-executors 5 --executor-memory 4g --executor-cores 3
Python 2.7.5 (default, Oct 14 2020, 14:45:30)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/03/06 12:25:29 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered!
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 2.4.0
/_/
Using Python version 2.7.5 (default, Oct 14 2020 14:45:30)
SparkSession available as 'spark'.
>>> from pyspark import SparkContext
>>> text = sc.textFile("/tmp/input_1GB.txt")
>>> words = text.flatMap(lambda line: line.split(" "))
>>> wordCounts = words.map(lambda word: (word, 1)).reduceByKey(lambda a, b: a + b)
>>> wordCounts.saveAsTextFile("/tmp/output_py.txt")Sample Output and it’s the same for both PythonAPI and ScalaAPI
hadoop fs -cat /user/test_user/output.txt/part-00007 | head -n 20
(u'b7jH', 10)
(u'c3AX%', 10)
(u'{/r{', 10)
(u'zJqd', 20)
(u'5L9u%G', 10)
(u'ok\ttPL', 10)
(u'(W-s', 10)
(u'Ux~w8', 10)
(u"Xl\t`'", 10)
(u'><Uv', 10)
(u'i*eI', 10)
(u']@rf', 10)
(u'$MPr', 10)
(u'NZj}', 10)
(u"2'.4", 10)
(u"'k{0", 10)
(u"K6+'m", 10)
(u'OW`1', 10)
(u'6m=fX', 10)
(u'Y#Ps', 10)
As you can see, the PySpark code is very similar to the Spark code, but with some minor syntax differences due to the use of Python instead of Scala.
Performance difference with the above example
In the above scala and python api example I have used the same 1GB file to see the performance. To do a quick test, I have used the below Linux command to create 1GB file
Linux Command to create 1 GB file
dd if=<(strings </dev/urandom) bs=4K count=25600 of=/tmp/text
The above command will create random words and save them to a temporary file “/tmp/text” and can be used for our performance testing
- Input File Size: 1 GB
- Executor Count: 5
- Executor Memory: 4GB
- Spark Scala API Runtime: 3.5 Min
- Spark Python API Runtime: 7.9 Min
In the above example, We could see the Scala API is having upper hand in terms of performance, Because Apache Spark is developed in Scala, it gives you access to the most up-to-date capabilities. Python is slower but easier to learn, whereas Scala is faster but more difficult to master.
Please do share your feedback if you have any different results in your use case and will be open to discussing this further.
Check here to know more about spark and the difference between DataFrame, Dataset, and RDD
Conclusion
In conclusion, both Apache Spark and Pyspark are powerful tools for big data processing, offering a range of libraries and tools for different use cases. Apache Spark provides a unified analytics engine for big data processing, while Pyspark provides a Pythonic way of working with Spark. The choice between the two depends on the specific use case and the preference of the developer.
However, both platforms have a strong and active community of developers and users, making them excellent choices for big data processing.
Good Luck with your Learning !!
