Googlielmo's blog

In the first part of this series we became familiar with ScalaTest . When it comes to unit test Scala Spark applications ScalaTest isn't enough: you need to add to the roster spark-testing-base . It is an Open Source framework which provides base classes for the main Spark abstractions like SparkContext, RDD, DataFrame, DataSet and Streaming. Let's start to explore all of the facilities provided by this framework and how it works along with ScalaTest with some simple examples. Let's consider the following Scala word count example found on the web: import org.apache.spark.{SparkConf, SparkContext} object SparkWordCount {    def main(args: Array[String]) {     val inputFile = args(0)     val outputFile = args(1)     val conf = new SparkConf().setAppName(" SparkWordCount ")     // Create a Scala Spark Context.     val sc = new SparkContext(conf)     // Load our input data....

This new series is about exploring useful frameworks and practices to unit testing Spark applications implemented in Scala . This first post is a quick introduction to ScalaTest, the most popular unit testing  framework for Scala applications. ScalaTest could be used to do unit testing with other languages like Scala.js and Java, but this post will focus on Scala only. I am going to refer to the latest table version (3.0.1) at the moment this post is being written. Let's see how ScalaTest works. Suppose we have a very simple Scala application with 2 classes, one called Basket : package ie.googlielmo.scalatestexample import scala.collection.mutable.ArrayBuffer class Basket { private val fruits = new ArrayBuffer[Fruit] def addFruit (fruit: Fruit) { fruits += fruit} def removeFruit (fruit: Fruit) { fruits -= fruit} def getFruits = fruits .toList }   which has a single attribute, an ArrayBuffer of Fruit , a simple case class: package ie.googlielmo.scalatestexa...