Skip to main content

Posts

Skills Matter Infiniteconf 2019 in July: I hope to see you there!

I am going to talk at the Skills Matter Infiniteconf 2019 , which will happen in London (UK) on July 4th and 5th 2019. It is a very interesting conference with lots of unusual and fantastic talks. The agenda is still WIP, but it looks promising. My talk would be on Friday 5th at 1 PM. I am going to share some of my past experiences on applying Machine Learning and Deep Learning to DevOps, in particular in managing Apache Spark applications and clusters. All the details about this conference in the official website. Early bird tickets are available before June 11th. I hope to meet you there!

Voxxed Days Milan 2019 review

Finally I found few minutes to share my impressions after attending the first Voxxed Days event in Italy, which happened in Milan on April 13th 2019 . I was one of the speakers there: my talk was about Deep Learning on Apache Spark with DeepLearning4J (a follow up of some topics from my book ). There were 3 sessions in parallel. The level of the talks was really high and it was hard for me and any other participant to choose which one to follow at a given time slot. The good news is that all of the sessions have been recorded and yesterday the first videos (those from the main session) have been published on YouTube . Once they will be online, I suggest you to watch all of the videos you can, but here are some suggestions among those I had a chance to attend in person at the event. I put my comments to a minimum to reduce spoiling ;) Opening key note by Mario Fusco : he was the main organizer of the event. In the opening key note he presented the agenda. He recently wrote a ...

The Kubernetes Spark operator in OpenShift Origin (Part 1)

This series is about the Kubernetes Spark operator by Radanalytics.io on OpenShift Origin . It is an Open Source operator to manage Apache Spark clusters and applications. In order to deploy the operator on OpenShift Origin, the first time you need to clone the GitHub repository for it: git clone https://github.com/radanalyticsio/spark-operator.git Then login to the cluster using the OpenShift command-line oc : oc login -u <username>:<password> Assuming, like in the OpenShift Origin environments me and my teams used to work, that developers don't have permissions to create CRDs, you need to use Config Maps, so you have to create the operator using the operator-com.yaml file provided in the cloned repo: oc apply -f manifest/operator-cm.yaml The output of the command above should be like the following: serviceaccount/spark-operator created role.rbac.authorization.k8s.io/edit-resources created rolebinding.rbac.authorization.k8s.io/spark-operator-edit-reso...

Installing Minishift on Windows 10 Home

Minishift is a tool to run OpenShift Origin locally as a single node cluster inside a Virtual Machine. It is a good choice for development or doing PoCs locally before deploying things in a real OpenShift cluster. In this post I am going to explain how to install and run it on a Windows 10 Home machine, where no Hyper-V support is available. The only available alternative to Hyper-V is Oracle VirtualBox . You need to install it before going on with the Minishift installation: follow the instructions in the official website or use the Chocolatey package manager. If you don't have Chocolatey in the destination machine, you can install it by opening an Admin PowerShell and first checking which execution policy is set, by running the Get-ExecutionPolicy command. If it returns Restricted , then install by executing: Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1')) The fastest an...

See you tonight at the ODSC Dublin Meetup!

I hope to see you tonight at the April ODSC Dublin Meetup @ Jet.com in 40 Molesworth St . I am going to be the second speaker for the night. I am going to talk about importing pre-trained Keras and TensorFlow models into DL4J and the possibility of re-training them on Apache Spark . The first speaker would be John Kane from Cogito .

The book is finally available on Packt!

My book "Hands-on Deep Learning with Apache Spark" is finally available on Packt. Here's the final cover: This is the book content: 1: THE APACHE SPARK ECOSYSTEM 2: DEEP LEARNING BASICS 3: EXTRACT, TRANSFORM, LOAD 4: STREAMING 5: CONVOLUTIONAL NEURAL NETWORKS 6: RECURRENT NEURAL NETWORKS 7: TRAINING NEURAL NETWORKS WITH SPARK 8: MONITORING AND DEBUGGING NEURAL NETWORK TRAINING 9: INTERPRETING NEURAL NETWORK OUTPUT 10: DEPLOYING ON A DISTRIBUTED SYSTEM 11: NLP BASICS 12: TEXTUAL ANALYSIS AND DEEP LEARNING 13: CONVOLUTION 14: IMAGE CLASSIFICATION 15: WHAT'S NEXT FOR DEEP LEARNING? DeepLearning4J (Scala), but also Keras and TensorFlow (Python) are the reference frameworks. More topics on Deep Learning on the JVM and Spark would be covered in the next months in this blog.

Sparklens: a tool for Spark applications optimization

Sparklens is a profiling tool for Spark with a built-in Spark Scheduler simulator: it makes easier to understand the scalability limits of Spark applications. It helps in understanding how efficiently is a given Spark application using the compute resources provided to it. It has been implemented and is maintained at Qubole . It is Open Source ( Apache License 2.0 ) and has been implemented in Scala. One interesting characteristic of Sparklens is its ability to generate estimates with a single run of a Spark application . It reports info such as estimated completion time and estimated cluster utilization with different number of executors, a Job/Stage timeline which shows how the parallel stages were scheduled within a job and lots of interesting per stage metrics. There are four ways to use Sparklens: Live mode Offline mode Run on event-history file Notebooks In this post I am focusing on live and offline modes only. Live mode Sparklens can run at application execution...