Googlielmo's blog

I am going to present a preview of the next release 1.5 and insights about the future release 2.0 of the publishtokafka Jenkins plugin at the Dublin Jenkins meetup on August 31st. Please come in if you're in the Dublin area to learn more about this plugin of mine and to have a chat about Jenkins stuff. Here's the map for the event: https://goo.gl/maps/BGT7MRig1Tv

A hot topic in the sdc-user group during the past weeks has been about how to schedule the start and stop of SDC pipelines. Usage of the SDC REST APIs has been suggested in some threads, but because the general impression I have is that the audience doesn't have a clear idea about them, I decided to write an article on DZone to help and clarify once and for all how to do it. Enjoy it!

In the first part of this series we became familiar with ScalaTest . When it comes to unit test Scala Spark applications ScalaTest isn't enough: you need to add to the roster spark-testing-base . It is an Open Source framework which provides base classes for the main Spark abstractions like SparkContext, RDD, DataFrame, DataSet and Streaming. Let's start to explore all of the facilities provided by this framework and how it works along with ScalaTest with some simple examples. Let's consider the following Scala word count example found on the web: import org.apache.spark.{SparkConf, SparkContext} object SparkWordCount {    def main(args: Array[String]) {     val inputFile = args(0)     val outputFile = args(1)     val conf = new SparkConf().setAppName(" SparkWordCount ")     // Create a Scala Spark Context.     val sc = new SparkContext(conf)     // Load our input data....

If in your company one or more of these 3 things happens: new software releases use to go to production quarterly there are QA teams there are DevOps teams then it's official: no DevOps culture over there. Good news is that now you know where the problems are and could start fixing ;)

My first Open Source Hubot script has been released and is available in my GitHub space . It provides support to check the status of pipelines in a Streamsets Data Collector server. It is still in alpha release, but the development is ongoing, so new features and improvements will be constantly implemented. Enjoy it!

In the past weeks, in order to explore new ways to improve DevOps people daily job introducing chatbots, I had a chance to evaluate and play with Hubot . It is an Open Source chat robot implemented by GitHub Inc. which is easy to program using simple scripts written in CoffeeScript and runs on Node.js . I started almost from scratch, being this my first production experience with Node.js and the first experience at all with CoffeeScript. In this post I am sharing just the basics to start implementing a personal Hubot. Prerequisites to follow this tutorial are Node.js and the npm package manager for JavaScript. Download and install the latest versions for your OS. In this post I am going to refer to Node.js 6.10.3 and npm 4.6.1. First of all you need to install the Hubot generator: npm install -g yo generator-hubot Then create the directory for your first Hubot: mkdir firstbot and generate the bot instance through the yeoman generator: cd firstbot  yo hubot At creat...

JGit is a Java library to programmatically do actions versus a local or remote Git repository. It is quite powerful, but it comes with an issue: when trying to connect to a remote repository it doesn't handle URL redirection. When you try to connect to a remote repo like in the example below (the connection attempt is to one of my repos in GutHub): String uri = "https://github.com/virtualramblas/publishtokafka-plugin.git"; String userName ="XXXXXXX" String password = "YYYYYYY" LsRemoteCommand remoteCommand = Git.lsRemoteRepository(); try {             Collection <Ref> refs =                     remoteCommand.setCredentialsProvider(new UsernamePasswordCredentialsProvider(userName, password))                     .setHeads(true)           ...