Googlielmo's blog

This new series is about exploring useful frameworks and practices to unit testing Spark applications implemented in Scala . This first post is a quick introduction to ScalaTest, the most popular unit testing  framework for Scala applications. ScalaTest could be used to do unit testing with other languages like Scala.js and Java, but this post will focus on Scala only. I am going to refer to the latest table version (3.0.1) at the moment this post is being written. Let's see how ScalaTest works. Suppose we have a very simple Scala application with 2 classes, one called Basket : package ie.googlielmo.scalatestexample import scala.collection.mutable.ArrayBuffer class Basket { private val fruits = new ArrayBuffer[Fruit] def addFruit (fruit: Fruit) { fruits += fruit} def removeFruit (fruit: Fruit) { fruits -= fruit} def getFruits = fruits .toList }   which has a single attribute, an ArrayBuffer of Fruit , a simple case class: package ie.googlielmo.scalatestexa...

I am glad to announce that the first major release of the Jenkins post-build action plugin to publish build jobs execution data to a Kafka topic is now available on my GitHub space . You can find all the details in the project README file. Please send me your feedback commenting this post or adding a new issue (enhancement, change request or bug) to the GitHub plugin repository whether you have a chance to use it. Thanks a lot.

Having completed all of the steps described in the first two posts of this series you should be able to start and use Pinpoint. To test that everything is working fine you can use the testapp web application which is part of its quickstart bundle. For this purpose you could start the collector and the web UI from the quickstart as well: %PINPOINT_HOME%\quickstart\bin\start-collector.cmd %PINPOINT_HOME%\quickstart\bin\start-web.cmd Then start the testapp application: %PINPOINT_HOME%\quickstart\bin\start-testapp.cmd Check that everything is fine connecting to the web UIs:     Pinpoint Web - http://localhost:28080     TestApp - http://localhost:28081 Start to do some actions in the testapp application and see through the web UI which information are sent to Pinpoint. Now you can profile any Java web and standalone application of yours. You need to download the agent jar in any location in the application hosting machine. Then, for standalone app...

This second post of the Pinpoint series covers the configuration of the HBase database where the monitoring data are written by the collector and from which they are read by the web UI. I did the first evaluation of Pinpoint on a MS Windows machine, so here I am going to cover some specific installation details for this OS family. For initial evaluation purposes a standalone HBase server (which runs all daemons within a single JVM) is enough. Database installation Here I am referring to the latest stable release (1.2.4) of HBase available at the time this post is being written. This release supports both Java 7 and Java 8: I am referring to Java 8 here. Cygwin isn't going to be used for this installation purposes. Of course you start downloading the tarball with the HBase binaries and then unpack its content. Rename the hbase-1.2.4 directory to hbase . Set up the JAVA_HOME variable to the JRE to use (if you don't have already done it in this installation machine). E...

I started a journey evaluating Open Source alternatives to commercial New Relic and AppDynamics tools to check if some is really ready to be used in a production environment. One cross-platform Application Performance Management (APM) tool that particularly caught my attention is Pinpoint . The current release supports mostly Java applications and JEE application servers and provides support also for the most popular OS and commercial relational databases. APIs are available to implement new plugins to support specific systems. Pinpoint has been modeled after Google Dapper and promises to install agents without changing a single line of code and mininal impact (about 3% increase in resource usage) on applications performance. Pinpoint is licensed under the Apache License, Version 2.0 . Architecture Pinpoint has three main components:  - The collector: it receives monitoring data from the profiled applications. It stores those information in HBase .  - The web UI: the f...

Spark Streaming brings Apache Spark's language integrated APIs to write streaming jobs the same way as for writing batch jobs. It allows to build fault tolerant applications and reuse the same code for batch and interactive queries. Kafka is an Open Source message broker written in Scala . It is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant and wicked fast. Cassandra is an Open Source distributed database management system designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failure. This post walks through the basics of the implementation of a simple streaming application integrating those three technologies. The code example is written in Scala. The releases I am referring to in this post are the following:  Scala 2.11.8  Spark 1.6.2  Kafka Client APIs 0.8.2.11  Cassandra 3.9  Datastax Spark-Cassandra Connecto...

Another interesting Hadoop User Group (HUG) Ireland Meetup next week (Monday October 3rd 2016) in Dublin at the Bank of Ireland premises in Grand Canal Square :   http://www.meetup.com/it-IT/hadoop-user-group-ireland/events/234240469/?eventId=234240469&chapter_analytics_code=UA-55809013-2 If you are in the Dublin area on Monday and interested in Machine Learning, please attend this event to learn more and start networking with other Big Data professionals. Hope to meet you there!