Googlielmo's blog

The Spark + AI Summit North America edition 2020 is going virtual and access to keynotes, sessions and virtual events is for free. You have to pay only to attend pre-conference and conference training, AMA sessions, VIP sessions and certification exams. The Summit will happen from June 22nd to 26th 2020. This year's keynote speakers include Francois Chollet, the creator of Keras . All details in the official website: https://databricks.com/sparkaisummit/north-america-2020

You're still in time to get some interesting Machine Learning and AI eBooks and videos for $10 each at Packt Publishing . This promotion covers also my book " Hands-on Deep Learning with Apache Spark ". What are you waiting for? Go and check out for titles before the offer expires! By the way, if you want to listen to follow-ups on topics covered by my book and get in touch with me in person, I am going to give talks at the following events in October and November: Big Data Days , Moscow, Russian Federation, October 8th-10th Spark+AI Summit Europe , Amsterdam, Netherlands, October 16th-17th Big Things Conference , Madrid, Spain, November 20th-21st

My book "Hands-on Deep Learning with Apache Spark" is finally available on Packt. Here's the final cover: This is the book content: 1: THE APACHE SPARK ECOSYSTEM 2: DEEP LEARNING BASICS 3: EXTRACT, TRANSFORM, LOAD 4: STREAMING 5: CONVOLUTIONAL NEURAL NETWORKS 6: RECURRENT NEURAL NETWORKS 7: TRAINING NEURAL NETWORKS WITH SPARK 8: MONITORING AND DEBUGGING NEURAL NETWORK TRAINING 9: INTERPRETING NEURAL NETWORK OUTPUT 10: DEPLOYING ON A DISTRIBUTED SYSTEM 11: NLP BASICS 12: TEXTUAL ANALYSIS AND DEEP LEARNING 13: CONVOLUTION 14: IMAGE CLASSIFICATION 15: WHAT'S NEXT FOR DEEP LEARNING? DeepLearning4J (Scala), but also Keras and TensorFlow (Python) are the reference frameworks. More topics on Deep Learning on the JVM and Spark would be covered in the next months in this blog.

Sparklens is a profiling tool for Spark with a built-in Spark Scheduler simulator: it makes easier to understand the scalability limits of Spark applications. It helps in understanding how efficiently is a given Spark application using the compute resources provided to it. It has been implemented and is maintained at Qubole . It is Open Source ( Apache License 2.0 ) and has been implemented in Scala. One interesting characteristic of Sparklens is its ability to generate estimates with a single run of a Spark application . It reports info such as estimated completion time and estimated cluster utilization with different number of executors, a Job/Stage timeline which shows how the parallel stages were scheduled within a job and lots of interesting per stage metrics. There are four ways to use Sparklens: Live mode Offline mode Run on event-history file Notebooks In this post I am focusing on live and offline modes only. Live mode Sparklens can run at application execution...

We are almost there: my "Hands-On Deep Learning with Apache Spark" book, Packt Publishing ,  is going to be available by the end of this month: https://www.packtpub.com/big-data-and-business-intelligence/hands-deep-learning-apache-spark In this book I try to address the sheer complexity of the technical and analytical parts, and the speed at which Deep Learning solutions can be implemented on Apache Spark . The book starts explaining the fundamentals of Apache Spark and Deep Learning. Then it details how to set up Spark for performing DL and the principles of distributed modelling and different types of neural nets. Example of implementation of DL models like CNN, RNN, LSTM on Spark are presented. A reader should get a hands-on experience of what it takes and a general feeling of the complexity he/she would deal with. During the course of the book, some popular DL frameworks such as DL4J , Keras and TensorFlow are used to train distributed models. The main goal of ...

In part 1 we have learned how to test data lineage info collection with Spline from a Spark shell. The same can be done in any Scala or Java Spark application. The same dependencies for the Spark shell need to be registered in your build tool of choice (Maven, Gradle or sbt): groupId: za.co.absa.spline artifactId: spline-core version: 0.3.5 groupId: za.co.absa.spline artifactId: spline-persistence-mongo version:0.3.5 groupId: za.co.absa.spline artifactId:spline-core-spark-adapter-2.3 version:0.3.5 With reference to Scala and Spark 2.3.x, a Spark job like this: // Create the Spark session val sparkSession = SparkSession    .builder()    .appName("Spline Tester")    .getOrCreate()   // Init Spline System.setProperty("spline.persistence.factory", "za.co.absa.spline.persistence.mongo.MongoPersistenceFactory") System.setProperty("spline.mongodb.url", args(0)) System.setProperty("spline.mongodb.name", args(1)) imp...

One interesting and promising Open Source project that caught my attention lately is Spline , a data lineage tracking and visualization tool for Apache Spark , maintained at  Absa . This project consists of 2 parts: a Scala library that works on the drivers which, by analyzing the Spark execution plans, captures the data lineages and a web application which provides a UI to visualize them. Spline supports MongoDB and HDFS as storage systems for the data lineages in JSON format. In this post I am referring to MongoDB. You can start playing with Spline through the Spark shell. Just add the required dependencies to the shell classpath as follows (with reference to the latest 0.3.5 release of this project): spark-shell --packages "za.co.absa.spline:spline-core:0.3.5,za.co.absa.spline:spline-persistence-mongo:0.3.5,za.co.absa.spline:spline-core-spark-adapter-2.3:0.3.5" Running the Spark shell with the command above on Ubuntu and some other Linux distro, whether some issue on...

This Friday November 23rd 2018 would be Black Friday at Packt Publishing too! Each book or video, including the latest releases, could be purchased for US$ 10 only. It would be also possible to pre-order my upcoming book " Hands-on Deep Learning with Apache Spark " for US$ 10. Please remember that this convenient price is valid on Friday 23rd only. Enjoy it!

I am proud to share that I will give a talk at the AI With the Best 2018 conference. The title of my talk is "Why Scala for Data Science?" and it is part of the "AI in Action" track. There I am going to cover some topics of my upcoming book . The conference will happen on September Friday 14th 2018. It is an online event. Buying a ticket for this event will give you also access to the recording of all the talks in the next 2 months after the conference end (just in case you should miss some during the live streaming). You will have also a chance to interact with the speakers and book 1:1 time with some of them. Please have a look at the list of speakers and talk topics: it is very impressive. I hope you will attend it!

Thanks to all the people who attended my talk "The Journey to TensorFlow on the JVM Stack" at the Google I/O Extended 2018 in Dublin. The slide deck is available on my SlideShare space . More code examples would be available on this blog in the next months.

An interesting conference on Scala is going to happen in Kiev (Ukraine) on April 20th-21st: https://www.scalaua.com/ Some early bird tickets should be still available. Here's a list of the speakers already confirmed for this event. Have a look at this short video to get some insight from the 2017 edition.

I have started doing evaluation of Livy for potential case scenarios where this technology could help and I'd like to share some findings with others who would like to approach this interesting Open Source project. It has been started by Cloudera and Microsoft and it is currently in the process of being incubated by the Apache Software Foundation. The official documentation isn't comprehensive at the moment, so I hope my posts on this topic could help someone else. Apache Livy is a service to interact with Apache Spark through a REST interface. It enables both submissions of Spark jobs or snippets of Spark code. The following features are supported: The jobs can be submitted as pre-compiled jars, snippets of code or via Java/Scala client API. Interactive Scala, Python, and R shells. Support for Spark 2.x and Spark1.x, Scala 2.10 and 2.11. It doesn't require any change to Spark code. It allows long running Spark Contexts that can be used for multiple Spark jobs, by...

In the first part of this series we became familiar with ScalaTest . When it comes to unit test Scala Spark applications ScalaTest isn't enough: you need to add to the roster spark-testing-base . It is an Open Source framework which provides base classes for the main Spark abstractions like SparkContext, RDD, DataFrame, DataSet and Streaming. Let's start to explore all of the facilities provided by this framework and how it works along with ScalaTest with some simple examples. Let's consider the following Scala word count example found on the web: import org.apache.spark.{SparkConf, SparkContext} object SparkWordCount {    def main(args: Array[String]) {     val inputFile = args(0)     val outputFile = args(1)     val conf = new SparkConf().setAppName(" SparkWordCount ")     // Create a Scala Spark Context.     val sc = new SparkContext(conf)     // Load our input data....

This new series is about exploring useful frameworks and practices to unit testing Spark applications implemented in Scala . This first post is a quick introduction to ScalaTest, the most popular unit testing  framework for Scala applications. ScalaTest could be used to do unit testing with other languages like Scala.js and Java, but this post will focus on Scala only. I am going to refer to the latest table version (3.0.1) at the moment this post is being written. Let's see how ScalaTest works. Suppose we have a very simple Scala application with 2 classes, one called Basket : package ie.googlielmo.scalatestexample import scala.collection.mutable.ArrayBuffer class Basket { private val fruits = new ArrayBuffer[Fruit] def addFruit (fruit: Fruit) { fruits += fruit} def removeFruit (fruit: Fruit) { fruits -= fruit} def getFruits = fruits .toList }   which has a single attribute, an ArrayBuffer of Fruit , a simple case class: package ie.googlielmo.scalatestexa...

Spark Streaming brings Apache Spark's language integrated APIs to write streaming jobs the same way as for writing batch jobs. It allows to build fault tolerant applications and reuse the same code for batch and interactive queries. Kafka is an Open Source message broker written in Scala . It is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant and wicked fast. Cassandra is an Open Source distributed database management system designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failure. This post walks through the basics of the implementation of a simple streaming application integrating those three technologies. The code example is written in Scala. The releases I am referring to in this post are the following:  Scala 2.11.8  Spark 1.6.2  Kafka Client APIs 0.8.2.11  Cassandra 3.9  Datastax Spark-Cassandra Connecto...

Developing things for Spark sometimes you don't have a choice in terms of language to use (it is the case of the GraphX APIs, where Scala is the only choice at the moment), but in some other cases you can choose between two different JVM languages (Java or Scala). Coming from a long background in Java and from my shorter experience in Scala, I can say that for sure some advantages using Scala in Spark programming are a better compactness and readability of the code. Have a look at the following simple Java code taken from one of the examples bundled with the Spark distribution: import java.io.Serializable; import java.util.List; import org.apache.spark.SparkConf; import org.apache.spark.api.java.JavaRDD; import org.apache.spark.api.java.JavaSparkContext; import org.apache.spark.api.java.function.Function; import org.apache.spark.sql.DataFrame; import org.apache.spark.sql.Row; import org.apache.spark.sql.SQLContext; public class JavaSparkSQL {     public static class Per...

Few weeks ago we had a need to move to Scala in order to do some things in Spark using APIs available for that language only. In order to speed up things while starting to learn it and minimize the impact on the existing components and the ongoing CI process, we found the dirty way discussed in this post to quickly start with. Started to use the Scala IDE for Eclipse and did the following actions to create new Scala projects:  - Create a new Scala project ( File -> New -> Scala Project ).  - Add the Maven nature to it ( Configure -> Convert to Maven Project ). All of our existing projects in the same area are built through Maven and our Jenkins CI servers use Maven to build after any code change and do a lot of other actions (Unit Tests execution, static analysis of the code, code coverage and many others) through it. That's the main reason we are not using sbt. The m2eclipse-scala and the m2e plugins are bundled with the Scala IDE, so no need to install them. ...

I am starting today a new series of posts about Apache Kafka ( http://kafka.apache.org/ ). Kafka is an Open Source message broker written in Scala ( http://www.scala-lang.org/ ). Originally it has been developed by LinkedIn ( https://ie.linkedin.com/ ), but then it has been released as Open Source in 2011 and it is currently maintained by the Apache Software Foundation ( http://www.apache.org/ ). Why one should prefer Kafka to a traditional JMS message broker? Here's a short list of convincing reasons: It's fast: a single Kafka broker running on commodity hardware can handle hundreds of megabytes of reads and writes per second from thousands of clients. Great scalability: it can be easily and transparently expanded without downtime.  Durability and Replication: messages are persisted on disk and replicated within the cluster to prevent data loss (setting a proper configuration using the high number of available configuration parameters you could achieve zero data loss). Pe...