venerdì 19 maggio 2017

Started playing with Hubot

In the past weeks, in order to explore new ways to improve DevOps people daily job introducing chatbots, I had a chance to evaluate and play with Hubot. It is an Open Source chat robot implemented by GitHub Inc. which is easy to program using simple scripts written in CoffeeScript and runs on Node.js. I started almost from scratch, being this my first production experience with Node.js and the first experience at all with CoffeeScript.
In this post I am sharing just the basics to start implementing a personal Hubot. Prerequisites to follow this tutorial are Node.js and the npm package manager for JavaScript. Download and install the latest versions for your OS. In this post I am going to refer to Node.js 6.10.3 and npm 4.6.1.
First of all you need to install the Hubot generator:

npm install -g yo generator-hubot

Then create the directory for your first Hubot:

mkdir firstbot

and generate the bot instance through the yeoman generator:

cd firstbot 
yo hubot

At creation time you will be asked for some information: the bot owner, the bot name, a description for it and the adapter to use. An adapter is the interface to the service you want your Hubot to run on. Hubot provides two official adapters, Shell and Campfire, but several third party Open Source adapters are available for the most popular chat services (Slack, XMPP, Facebook Messenger, etc.).
Now you can start the bot. The start script has been generated in the bin directory inside the bot home. Run

./bin/hubot

and the bot is ready to interact with you in a command shell. You can check for a list of the available commands by typing

firstbot help

Let's see now how it is possible to implement and add a custom script (scripts give really power to a bot). A script must export at least one function. So the first line of code (excluding comments) has to be the following:

module.exports = (robot) ->

Then you can start to add your code. The most common interaction between the bot and humans is based on messages, so the hear and respond methods are the most used:

robot.hear /badger/i, (res) ->
    # your code here

robot.respond /open the pod bay doors/i, (res) ->
    # your code here


It is possible, through regular expressions, to capture content from the input messages using  res.match. Example:

robot.respond /open the (.*) doors/i, (res) ->
    doorType = res.match[1]
    if doorType is "pod bay"
      res.reply "I'm afraid I can't let you do that."
    else
      res.reply "Opening #{doorType} doors"


The bot can do more complex things, like HTTP requests for example:

robot.http("https://midnight-train")
    .get() (err, res, body) ->
      # your code here


Here's an example of HTTP request to a Jenkins server to get the results of the unit test for a given execution of a build job by parsing the JSON content of the response:

robot.hear /Unit tests status for (.*) build number (.*)/i, (res) ->
        buildJobTestResultUrl = res.match[1] + res.match[2] + "/testReport/api/json?pretty=true"
        res.robot.http(buildJobTestResultUrl)
            .header('Accept', 'application/json')
            .get() (err, response, body) ->
                data = null
                try
                    data = JSON.parse(body)
                    res.send "Test results: #{data.passCount} passed; #{data.failCount} failed; #{data.skipCount} skipped."
                    #res.send "#{body} content."
                catch error
                   res.send "Ran into an error parsing JSON :( #{error}"
                   return


The first match there is the build job URL and the second one is the build number. The output by  the bot would be like this:

Test results: 32 passed; 0 failed; 0 skipped.

Once you have completed a script implementation, in order to register it you have to save it with the .coffee extension in the scripts directory of the bot home and then restart the bot to use it.

The process of implementing bots in Hubot and enhancing them through scripts is pretty straightforward. Furthermore there are several hundreds available scripts ready to be installed through npm: so please check that list before implementing anything.

I will share next more interesting scripts and tips on Hubot.

mercoledì 10 maggio 2017

Handling URL redirection with JGit

JGit is a Java library to programmatically do actions versus a local or remote Git repository. It is quite powerful, but it comes with an issue: when trying to connect to a remote repository it doesn't handle URL redirection. When you try to connect to a remote repo like in the example below (the connection attempt is to one of my repos in GutHub):

String uri = "https://github.com/virtualramblas/publishtokafka-plugin.git";
String userName ="XXXXXXX"
String password = "YYYYYYY"
LsRemoteCommand remoteCommand = Git.lsRemoteRepository();
try {
            Collection <Ref> refs =
                    remoteCommand.setCredentialsProvider(new UsernamePasswordCredentialsProvider(userName, password))
                    .setHeads(true)
                    .setRemote(uri)
                    .call();
} catch(GitAPIException e) {
            e.printStackTrace();
}


everything is fine because there is no redirection. But trying to connect to another remote repository URL which performs redirection the following exception happens:

Caused by: org.eclipse.jgit.errors.TransportException: https://<url>/<repo_name>.git: 302 Found

Unfortunately the JGit library can't handle this, but a workaround is possible. Using curl with the following options:

curl -u <username>:<password> -v https://<url>/<repo_name>.git

From the output of this command you can retrieve the URL to which this one redirects and then replace the original one in your application code. This way the connection is successful and you can do all the actions you need to the remote repo. 

martedì 18 aprile 2017

Tutorial: How to Create a Custom StreamSets Origin That Supports Parallel Execution

If you are interested in implementing a custom SDC origin which supports parallel execution then you can read my new DZone article. You can find this tutorial (with updates) in my GitHub space as well. Enjoy it!

sabato 25 marzo 2017

Unit testing Spark applications in Scala (Part 1): Intro to ScalaTest

This new series is about exploring useful frameworks and practices to unit testing Spark applications implemented in Scala. This first post is a quick introduction to ScalaTest, the most popular unit testing  framework for Scala applications.
ScalaTest could be used to do unit testing with other languages like Scala.js and Java, but this post will focus on Scala only. I am going to refer to the latest table version (3.0.1) at the moment this post is being written.
Let's see how ScalaTest works. Suppose we have a very simple Scala application with 2 classes, one called Basket:

package ie.googlielmo.scalatestexample

import scala.collection.mutable.ArrayBuffer

class Basket  {
  private val fruits = new ArrayBuffer[Fruit]

  def addFruit (fruit: Fruit) { fruits += fruit}
  def removeFruit (fruit: Fruit) { fruits -= fruit}
  def getFruits = fruits.toList
}
 
which has a single attribute, an ArrayBuffer of Fruit, a simple case class:

package ie.googlielmo.scalatestexample

case class Fruit(name:String)

and methods to add to or remove a fruit from the basket and to get the list of current fruits in the basket.
In order to write unit tests for the Basket class we need to add the ScalaTest dependency in the sbt build file for the project as follows:

libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.1" % "test"

Or, if you use Maven,  add the dependency to the project pom.xml file as follows:

<dependency>
  <groupId>org.scalatest</groupId>
  <artifactId>scalatest_2.11</artifactId>
  <version>3.0.1</version>
  <scope>test</scope>
</dependency>

The central unit of composition in ScalaTest is Suite, which represents a suite of tests. ScalaTest provides several style traits that extend Suite. They override its lifecycle methods to support different testing styles. If you come from TDD and/or you are familiar with JUnit probably the first choice to start would be the FunSuite style trait. Basically it is a JUnit-like style but coming with some benefits of BDD. In this case a test suite for the Basket class could be implemented this way:

import org.scalatest.FunSuite

class BasketSuite extends FunSuite with BeforeAndAfter {
  var basket : Basket = _

  before {
    basket = new Basket()
  }

  test("An empty Basket should have 0 fruits") {
    assert(basket.getFruits.size == 0)
  }

  test("Adding one fruit") {
    basket.addFruit(new Fruit("Pear"))
    assert(basket.getFruits.size == 1)
  }

  test("Removing one fruit") {
    basket.addFruit(new Fruit("Banana"))
    basket.removeFruit(new Fruit("Pear"))
    assert(basket.getFruits.size == 1)
  }
}

The BeforeAndAfter trait provides the before and after methods to initialize and destroy objects used across the unit tests for a test suite class. The description used for each test method is the label you will see in the reports generated at the end of unit tests execution.

If you are used to BDD probalby a better choice for you would be the FlatSpec suite trait. The implementation of unit test for the Basket class would be something like this:

package ie.googlielmo.scalatestexample

import org.scalatest.FlatSpec

class BasketSpec extends FlatSpec {
  var basket : Basket = new Basket()

  "An empty Basket" should "have 0 fruits" in {
    assert(Set.empty.size == 0)
  }

  it should "contain 1 fruit when adding one" in {
    basket.addFruit(new Fruit("Pear"))
    assert(basket.getFruits.size == 1)
  }

  it should "contain 1 fruit when removing one" in {
    basket.addFruit(new Fruit("Banana"))
    basket.removeFruit(new Fruit("Pear"))
    assert(basket.getFruits.size == 1)
  }
}

Again, the description chosen for each test method here is the label shown the in the reports generated at the end of unit tests execution.
There are 8 Suite traits available in ScalaTest. I am not going to desribe all of them in this post, you can find all of the details in the official framework documentation. I will probably add some more details in the next posts of this series when describing specific test cases for Spark applications.
It is possible, just in case, to use different testing styles for the same Scala project. The full battery of unit tests will be executed any way, as shown in the snapshot below:

Note for Maven users only: you need to disable SureFire and enable ScalaTest in order to execute ScalaTest unit tests. This can be done registering the following plugin into the project pom.xml file: 
 
<plugin>
  <groupId>org.scalatest</groupId>
  <artifactId>scalatest-maven-plugin</artifactId>
  <version>1.0</version>
  <configuration>
    <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
    <junitxml>.</junitxml>
    <filereports>WDF TestSuite.txt</filereports>
  </configuration>
  <executions>
    <execution>
      <id>test</id>
      <goals>
        <goal>test</goal>
      </goals>
    </execution>
  </executions>
</plugin>

In the next post we will learn how to start unit testing simple Spark jobs and why ScalaTest alone isn't enough for this purpose. Stay tuned!

mercoledì 8 marzo 2017

Publish to Kafka Jenkins plugin release 1.0 available

I am glad to announce that the first major release of the Jenkins post-build action plugin to publish build jobs execution data to a Kafka topic is now available on my GitHub space. You can find all the details in the project README file. Please send me your feedback commenting this post or adding a new issue (enhancement, change request or bug) to the GitHub plugin repository whether you have a chance to use it. Thanks a lot.