Get started Spark with Databricks and PySpark

Get started working with Spark and Databricks with pure plain Python

In the beginning, the Master Programmer created the relational database and file system. But the file system in a single machine became limited and slow. The data darkness was on the surface of database. The spirit of map-reducing was brooding upon the surface of the big data.

And Master Programmer said, let there be Spark, and there was Spark.

There is already Hadoop, why bother Spark

If the relational database is a well-maintained data garden; Hadoop is a clutter data forest, and it can grow to an unlimited size.

To put data into the garden, data need to be carefully cleaned and grow there structurally. While in the Hadoop forest, ladies and gentlemen, don’t worry, any data is fine here, text, numerical numbers, even audio and videos with no data size and type limitation.

But there are still some shortages in Hadoop, where Spark comes to solve. Here list 4 key differences for me.

1. In Hadoop, every mapping and reducing action use disk storage as the data middle man, and disk operation is slow. Spark optimize the process by leveraging memory direct data access. In other words, store some intermediate data in memory to boost the performance. (That is why you always read the official Spark introduction portray itself much faster than Hadoop, nothing magic here.)
2. Hadoop is basically a distributed file system that can be extended to unlimited size with its map-reducer and batch scheduler. But you need to use Java to implement real applications. Spark comes to provide operation languages like Python, and R. Provide useful tools for data streaming, Machine Learning, and data analytic.
3. Hadoop doesn’t include a job scheduler and needs 3rd – party scheduler involved, Sparks comes with its own job scheduler.
4. Hadoop is much cheaper and low RAM required. Spark requires more RAM. Ok, this one is an advantage of Hadoop instead of a disadvantage.

And with PySpark, we can interact with Spark fully in pure plain Python code, in Jupyter Notebook, or Databricks Notebook. This is a great plus from Spark.

There is already Spark, why bother Databricks

Spark is open-sourced, free, and powerful, why bother using Databricks? To set up a useful Spark cluster, and leverage the distributed storage, we need to build at least 2 machines, virtually or physically. Next, set up the Driver and worker nodes, configure the network and securities, etc.

A lot more manual work to be done for simply run a Spark "hello world". Don’t mention if you get error messages like JAVA_HOME can’t be found, or can’t find Spark path.

Databricks provides a unbox and ready-to-use environment by solving all these tedious configurations.

Take Azure Databricks for example, after several mouse clicks, and several minutes waiting for the cluster to spin up. We have a fully-featured Spark system. They call it Databricks.

Unlike the free Spark, Databricks is usually charged by the cluster size and usage. Be careful, choose the right size when creating your first instance.

One more thing to note, please do remember the Databricks runtime version you selected. I’d prefer to select the LTS 7.3. later, when you install the databricks-connect the version should be the same.

Start the connection

I am going to use Python to do everything, so should I install pyspark package? No, To use Python to control Databricks, we need first uninstall the pyspark package to avoid conflicts.

pip uninstall pyspark

Next, install the databricks-connect. which include all PySpark functions with a different name. (Ensure you already have Java 8+ installed in your local machine)

pip install -U "databricks-connect==7.3.*"

Before configuring the client connection to Databricks cluster, go to the Databricks UI grab the following information and write down to your note. Detail steps can be found here

1. access token: dapib0fxxxxxxxxx6d288bac04855bccccd
2. workspace URL: [https://adb-8091234370581234.18.azuredatabricks.net/](https://adb-8091234370581234.18.azuredatabricks.net/)
3. cluster id: 1234-12345-abcdef123
4. port number:15001
5. org id: 8091234370581234 ,org id also appears in the workspace url.

When you have all the above information ready, go configure your local PySpark connection to the Databricks cluster.

databricks-connect configure

follow the guide, you won’t miss the path. After this, use this Python code to test the connection.

# python 
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
print('spark session created.')

If you are welcomed with "spark session created.", a live and kicking Spark cluster is running in the cloud. We can do some big data analysis now.

The Databricks storage: BDFS

Hadoop’s HDFS from Hadoop allows users to build scalable massive storage upon local disks. BDFS is almost the same as HDFS. The difference is its backend storage is cloud-based.

You can use dbutils to remotely manage the BDFS with Python,

To get dbutils object handler in your local Python context. The official document assumes you are using Databricks Notebook and omit this step. Makes users confused when trying to use it in plain Python code.

from pyspark.dbutils import DBUtils
dbutils = DBUtils(spark) # the spark object here 
                         # is already initialized above

List the files and folders from the /mnt/ folder

dbutils.fs.ls('dbfs:/mnt/')

And you will get information like this:

[FileInfo(path='dbfs:/mnt/folder1/', name='folder1/', size=123),
 FileInfo(path='dbfs:/mnt/folder2/', name='folder2/', size=123),
 FileInfo(path='dbfs:/mnt/tmp/', name='tmp/', size=123)]

The dbutils official document list all other operations.

Upload a CSV file to DBFS with Python

Prepare A Bible CSV file on your local disk. Replace the [username] with yours to run the below code.

import urllib.request
bible_url = "https://raw.githubusercontent.com/scrollmapper/bible_databases/master/csv/t_kjv.csv"
urllib.request.urlretrieve(bible_url,"/home/[username]/temp/bible_kjv.csv")

Now, upload the bible CSV file to BDFS.

bible_csv_path = "file:/home/[username]/temp/bible_kjv.csv"
dbutils.fs.cp(bible_csv_path,"/tmp/bible_kjv.csv")

Use mv to replace cp if you decide to move the file instead of copying.

Use Spark Dataframe to analyze the data

Read the just uploaded Bible CSV file and encapsulate it in a Spark Dataframe(in contrast to Pandas Dataframe).

bible_spark_df = spark.read.format('csv')
                 .options(header='true')
                 .load('/tmp/bible_kjv.csv')
bible_spark_df.show()

You shall see result

+-------+---+---+---+--------------------+
|     id|  b|  c|  v|                   t|
+-------+---+---+---+--------------------+
|1001001|  1|  1|  1|In the beginning ...|
|1001002|  1|  1|  2|And the earth was...|
|1001003|  1|  1|  3|And God said, Let...|
...
|1001019|  1|  1| 19|And the evening a...|
|1001020|  1|  1| 20|And God said, Let...|
+-------+---+---+---+--------------------+
only showing top 20 rows

If you are Pandas Dataframe fans, it is easy to transform data to Pandas Dataframe

bible_pandas_df = bible_spark_df.toPandas()

Let’s use Spark Dataframe to see how many verses of each book.

bible_spark_df.groupBy("b")
              .count()
              .sort("count",ascending=False)
              .show()

You shall see the result

Book 1 is Genesis, this book contains 1533 verses.

With the help of creating a temp view, we can also query the data using Spark SQL

bible_spark_df.createOrReplaceTempView('bible')
bible_sql_result = spark.sql('''
    select * from bible 
    where id == 1001001
''')
bible_sql_result.show()

The query result

+-------+---+---+---+--------------------+
|     id|  b|  c|  v|                   t|
+-------+---+---+---+--------------------+
|1001001|  1|  1|  1|In the beginning ...|
+-------+---+---+---+--------------------+

Save Spark Dataframe back to BDFS as a JSON file.

bible_spark_df.write.format('json').save('/tmp/bible_kjv.json')

For all Spark dataset operations, check out The Spark SQL, DataFrames and Datasets Guide

Spark Database and Tables

Spark also supports Hive database and tables, in the above sample, I create a temp view to enable the SQL query. But the temp view will disappear when the session end. To enable store data in Hive Table and can be queried with Spark SQL for the long run. we can store data in Hive tables.

First, create a Hive database

spark.sql("create database test_hive_db")

Next, write the bible spark Dataframe as a table. The database name here is kind of like a table folder.

bible_spark_df.write.saveAsTable('test_hive_db.bible_kjv')

For all information about Spark Hive table operations, check out Hive Tables

Wrap up and summary

Congratulation and thank you for reading through here. When I started learning Spark and Databricks, I got stuck when Book authors tried to introduce the Spark backend architecture with complex diagrams. I wrote this for those who never touched Spark before and want to get hands dirty without getting confused.

If you run all code successfully, you should be in a good position to start using Spark and Databricks. Spark and Databricks are just tools shouldn’t be that complex, can it be more complex than Python? (kidding)

One more thing to note, the default Databricks Get Started tutorial use Databricks Notebook, which is good and beautiful. But in real projects and work, you may want to write code in plain Python and manage your work in a git repository. I found Visual Studio Code with Python and Databricks extension is a wonderful tool that fully supports Databricks and Spark.