giovedì 8 giugno 2017

Unit testing Spark applications in Scala (Part 2): Intro to spark-testing-base

In the first part of this series we became familiar with ScalaTest. When it comes to unit test Scala Spark applications ScalaTest isn't enough: you need to add to the roster spark-testing-base. It is an Open Source framework which provides base classes for the main Spark abstractions like SparkContext, RDD, DataFrame, DataSet and Streaming. Let's start to explore all of the facilities provided by this framework and how it works along with ScalaTest with some simple examples. Let's consider the following Scala word count example found on the web:

import org.apache.spark.{SparkConf, SparkContext}

object SparkWordCount { 
  def main(args: Array[String]) { 
   val inputFile = args(0) 
   val outputFile = args(1) 
   val conf = new SparkConf().setAppName("SparkWordCount") 
   // Create a Scala Spark Context. 
   val sc = new SparkContext(conf) 
   // Load our input data. 
   val input = sc.textFile(inputFile) 
   // Split up into words. 
   val words = input.flatMap(line => line.split(" ")) 
   // Transform into word and count. 
   val counts = => (word, 1)).reduceByKey{case (x, y) => x + y} 
   // Save the word count back out to a text file, causing evaluation.   counts.saveAsTextFile(outputFile) 

This class expects a text file as input and first splits all of the words this contains and finally counts all of the occurrences for each one.
Add the ScalaTest dependency to sbt or Maven as described in the first part of this series. Then add the scala-testing-base dependency to sbt:
libraryDependencies += "com.holdenkarau" % "spark-testing-base_2.11" % "2.1.0_0.6.0"
or Maven:


The code above is definitively not the best way to implement this class. Probably a better way (in terms of readability and maintenance) could be the following, implementing two separate methods (one for the mapping and one for the reduction by key):

import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.rdd.RDD

object SparkWordCount {
  def main(args: Array[String]) {
    val conf = new SparkConf().setAppName("SparkWordCount")

    val sc = new SparkContext(conf)
    val file = sc.textFile(args(0))
  def countWordsInFile = splitFile _ andThen countWords _
  def splitFile(wordsByLine: RDD[String]): RDD[String] = {
    wordsByLine.flatMap(line => line.split(" "))

  def countWords(words: RDD[String]): RDD[(String, Int)] = { => (word, 1)).reduceByKey(_ + _)

Let's implement a test class for it now. We are going to combine ScalaTest and spark-testing-base together. Let's choose a ScalaTest style (for example FunSuite) and the SharedSparkContext trait:

class SparkWordCountScalaTestingBaseSuite extends FunSuite with SharedSparkContext

SharedSparkContext initializes (before the tests start) and provides a SparkContext instance to be shared through all of the test in the class and stops it at the end of the test method execution. So no need to add extra code to initialize it and stop it. This way the development of a test class focuses only on the body of the test methods. For the class under test above is:

val fileLines = Array("Line One", "Line Two", "Line Three", "Line Four")

test("splitFile should split the file into words"){
    val inputRDD: RDD[String] = sc.parallelize[String](fileLines)
    val wordsRDD = SparkWordCount.splitFile(inputRDD)
    assert(wordsRDD.count() == 8)

test("countWordsInFile should count words") {
    val inputRDD: RDD[String] = sc.parallelize[String](fileLines)
    val results = SparkWordCount.countWordsInFile(inputRDD).collect
    assert(results.contains(("Line", 4)))

sc is the variable name of the built-in SparkContext provided by SharedSparkContext.
In the next post(s) we will walk through all of the other base classes provided by the spark-testing-base framework, starting from those related to RDDs.

martedì 6 giugno 2017

3 signs your company doesn't have a DevOps culture

If in your company one or more of these 3 things happens:
  • new software releases use to go to production quarterly
  • there are QA teams
  • there are DevOps teams
then it's official: no DevOps culture over there.
Good news is that now you know where the problems are and could start fixing ;)

lunedì 5 giugno 2017

Hubot & SDC

My first Open Source Hubot script has been released and is available in my GitHub space. It provides support to check the status of pipelines in a Streamsets Data Collector server. It is still in alpha release, but the development is ongoing, so new features and improvements will be constantly implemented. Enjoy it!

venerdì 19 maggio 2017

Started playing with Hubot

In the past weeks, in order to explore new ways to improve DevOps people daily job introducing chatbots, I had a chance to evaluate and play with Hubot. It is an Open Source chat robot implemented by GitHub Inc. which is easy to program using simple scripts written in CoffeeScript and runs on Node.js. I started almost from scratch, being this my first production experience with Node.js and the first experience at all with CoffeeScript.
In this post I am sharing just the basics to start implementing a personal Hubot. Prerequisites to follow this tutorial are Node.js and the npm package manager for JavaScript. Download and install the latest versions for your OS. In this post I am going to refer to Node.js 6.10.3 and npm 4.6.1.
First of all you need to install the Hubot generator:

npm install -g yo generator-hubot

Then create the directory for your first Hubot:

mkdir firstbot

and generate the bot instance through the yeoman generator:

cd firstbot 
yo hubot

At creation time you will be asked for some information: the bot owner, the bot name, a description for it and the adapter to use. An adapter is the interface to the service you want your Hubot to run on. Hubot provides two official adapters, Shell and Campfire, but several third party Open Source adapters are available for the most popular chat services (Slack, XMPP, Facebook Messenger, etc.).
Now you can start the bot. The start script has been generated in the bin directory inside the bot home. Run


and the bot is ready to interact with you in a command shell. You can check for a list of the available commands by typing

firstbot help

Let's see now how it is possible to implement and add a custom script (scripts give really power to a bot). A script must export at least one function. So the first line of code (excluding comments) has to be the following:

module.exports = (robot) ->

Then you can start to add your code. The most common interaction between the bot and humans is based on messages, so the hear and respond methods are the most used:

robot.hear /badger/i, (res) ->
    # your code here

robot.respond /open the pod bay doors/i, (res) ->
    # your code here

It is possible, through regular expressions, to capture content from the input messages using  res.match. Example:

robot.respond /open the (.*) doors/i, (res) ->
    doorType = res.match[1]
    if doorType is "pod bay"
      res.reply "I'm afraid I can't let you do that."
      res.reply "Opening #{doorType} doors"

The bot can do more complex things, like HTTP requests for example:

    .get() (err, res, body) ->
      # your code here

Here's an example of HTTP request to a Jenkins server to get the results of the unit test for a given execution of a build job by parsing the JSON content of the response:

robot.hear /Unit tests status for (.*) build number (.*)/i, (res) ->
        buildJobTestResultUrl = res.match[1] + res.match[2] + "/testReport/api/json?pretty=true"
            .header('Accept', 'application/json')
            .get() (err, response, body) ->
                data = null
                    data = JSON.parse(body)
                    res.send "Test results: #{data.passCount} passed; #{data.failCount} failed; #{data.skipCount} skipped."
                    #res.send "#{body} content."
                catch error
                   res.send "Ran into an error parsing JSON :( #{error}"

The first match there is the build job URL and the second one is the build number. The output by  the bot would be like this:

Test results: 32 passed; 0 failed; 0 skipped.

Once you have completed a script implementation, in order to register it you have to save it with the .coffee extension in the scripts directory of the bot home and then restart the bot to use it.

The process of implementing bots in Hubot and enhancing them through scripts is pretty straightforward. Furthermore there are several hundreds available scripts ready to be installed through npm: so please check that list before implementing anything.

I will share next more interesting scripts and tips on Hubot.

mercoledì 10 maggio 2017

Handling URL redirection with JGit

JGit is a Java library to programmatically do actions versus a local or remote Git repository. It is quite powerful, but it comes with an issue: when trying to connect to a remote repository it doesn't handle URL redirection. When you try to connect to a remote repo like in the example below (the connection attempt is to one of my repos in GutHub):

String uri = "";
String userName ="XXXXXXX"
String password = "YYYYYYY"
LsRemoteCommand remoteCommand = Git.lsRemoteRepository();
try {
            Collection <Ref> refs =
                    remoteCommand.setCredentialsProvider(new UsernamePasswordCredentialsProvider(userName, password))
} catch(GitAPIException e) {

everything is fine because there is no redirection. But trying to connect to another remote repository URL which performs redirection the following exception happens:

Caused by: org.eclipse.jgit.errors.TransportException: https://<url>/<repo_name>.git: 302 Found

Unfortunately the JGit library can't handle this, but a workaround is possible. Using curl with the following options:

curl -u <username>:<password> -v https://<url>/<repo_name>.git

From the output of this command you can retrieve the URL to which this one redirects and then replace the original one in your application code. This way the connection is successful and you can do all the actions you need to the remote repo.