Googlielmo's blog

 I just realized that I am using Streamlit since almost one year now, posted about in Twitter or LinkedIn several times, but never wrote a blog post about it before. Communication in Data Science and Machine Learning is the key. Being able to showcase work in progress and share results with the business makes the difference. Verbal and non-verbal communication skills are important. Having some tool that could support you in this kind of conversation with a mixed audience that couldn't have a technical background or would like to hear in terms of results and business value would be of great help. I found that Streamlit fits well this scenario. Streamlit is an Open Source (Apache License 2.0) Python framework that turns data or ML scripts into shareable web apps in minutes (no kidding). Python only: no front‑end experience required. To start with Streamlit, just install it through pip (it is available in Anaconda too): pip install streamlit and you are ready to execute the working de...

  The Codex paper has been published yesterday. Codex is a GPT language model finetuned on publicly available code from GitHub which has Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot . This paper focuses on the work leading to the early Codex models. The main task is the generation of standalone Python functions from docstrings, and the automated evaluation of the correctness of code samples through unit tests (this is in contrast to natural language generation, where samples are typically evaluated by heuristics or by human evaluators). To solve a problem in the test set, the authors generate multiple samples from the models, and check if any of them passes the unit tests. The raw training dataset was collected in May 2020 from 54 million public software repositories hosted on GitHub, containing 179 GB of unique Python files under 1 MB. Then it has been filtered by removing files which were likely auto-generated, had average line l...

 Jupyter notebooks allows LaTeX rendering inside markdown. This way you can write complex math equations within a notebook. While LaTeX is the de facto standard for scientific documents, it hasn't a very friendly and intuitive syntax. handcalcs is an Open Source library for converting Python calculations into rendered LaTeX: just write the symbolic formula, followed by numeric substitutions and that's it. After install it (it is available through PyPI), in the simplest case you just need to import the render class and use the %%render magic command to render the content of a cell: Here another example of equation render and numeric substitution: It is also possible to render just the symbolic equation: or any way generate the corresponding LaTeX code: By default handcalcs renders code vertically, but it is possible to use the %%render params magic to save space by rendering in a single line or show just the result of a calculation: handcalcs allows to adjust precision, use Gr...

  Faker is an Open Source Python package that generates synthetic data that could be used for many things such as populating a database, do load testing or anonymize production data for development or ML purposes. Generating fully random data isn't a good choice: with Faker you can drive the generation process and tailor the generated data to your specific needs: this is the greatest value provided by Faker. This package comes with 23 built-in data providers, some other providers are available from the community. The available data providers cover majority of data types and cases, but it is possible any way make the generated data more meaningful by implementing a custom provider. Faker supports Python 3.6+ and it is available for installation through PyPI or Anaconda.  Here's a code example that shows how to implement a custom provider to generate synthetic data following the structure and constraints as for this Kaggle  dataset related to a restaurant data with consumer...

 In my career I have noticed that often organizations are reluctant on providing Engineering teams with licenses for software to draw diagrams. In the best case scenarios MS Visio is usually the only option available, which isn't the best experience when trying to draw modern software architectures. Several online options are available, but they require to share project details that cannot leave your organization network, so they couldn't be taken into account often. Also, while treating everything as code, it would be nice to have also diagrams as code. All these needs can be satisfied by adopting Diagrams . It is an Open Source Python package that allows you draw cloud system architecture diagrams programmatically and then put them under version control, (as at the end they are regular Python files). It supports cloud (AWS, Azure, GCP, Alibaba, Oracle) and on-prem system architecture diagrams. The Diagrams nodes include also Kubernetes, programming languages and frameworks. ...

The Spark + AI Summit North America edition 2020 is going virtual and access to keynotes, sessions and virtual events is for free. You have to pay only to attend pre-conference and conference training, AMA sessions, VIP sessions and certification exams. The Summit will happen from June 22nd to 26th 2020. This year's keynote speakers include Francois Chollet, the creator of Keras . All details in the official website: https://databricks.com/sparkaisummit/north-america-2020

In part 1 of this series a definition of CBIR has been given. Let's now understand what's the typical flow for it. The diagram in figure 1 shows that there are two parts, one that happens offline and another which is online: Figure 1 Starting from an image storage, a preliminary trained Deep Neural Network is used to extract the features from images. Extracted features are then stored in a feature database. This happens offline any time new images are added to the storage. What happens online is the search process itself. Any time a user uploads a query image, the same Deep Neural Network used for feature extraction is used to extract the input image features. Then the distance from the query image features and the features in the database are computed. The closer the distance, the higher the relevance is. The closest features are then sorted and the corresponding images are returned as results. Basic implementation: data set preparation To make things more clear,...

Content Based Image Retrieval (CBIR) systems are aimed to find similar images to an input image by querying an image data set. I am pretty sure you've used a CBIR several times before: here's the most popular one: The search in CBIRs isn't text-based as images are indexed by their visual content. Features are extracted from an image database and stored in some file system or object storage.Then, any time a new image is passed as search criterion, all those images which features are closest to the input image features are returned as results. The traditional feature extraction process relies on algorithms such as SURF or SIFT , but in this series I am going to describe my journey into CBIR using an unsupervised Deep Learning approach. Here's a preview of the first attempt, in order to search for food images (the input image in this case is an apple pie): In the next posts of this series I will walk through the details of the implementation before making thi...

I hope you are going to attend the 8th edition of Big Things 2019 , the data and AI conference which will happen in Madrid (Spain) on November 20th and 21st. This conference started in 2012 and this year changed its name from Big Data Spain to Big Things, as it became focused not only on Big Data, but also on whatever is related to AI. Among the speakers this year there will be big names such as Cassie Kozyrkov, Alberto Cairo, Jesse Anderson, Michael Armbrust, Suneel Marthi, Paco Nathan and many others. My talk will be on Thursday 21st at 1:55 PM local time. I am going to give an update on importing and re-training Keras / TensorFlow models in DL4J and Apache Spark . It is a follow-up of some of the topics covered in my book , considering changes related to new releases of DL4J, Keras and TensorFlow since it has been published in January this year. Please stop by if you are going to attend my talk and the conference. I really appreciated the conversations about Deep Learning ...

Colab is the Google's platform for working with Python notebooks and practice Deep Learning using different frameworks. It is a powerful platform, there is availability of GPUs or TPUs, it allows to use your Google Drive space for notebooks and data, has a friendly user interface and lots of useful features, but in order to install/update Python packages, it comes by default only with pip and no availability for conda . If you need to import a Python package which is available in Anaconda , but not in PyPi you need to install Anaconda or Miniconda yourself from a notebook. In this post I am explaining the simple steps to do it. Anaconda installation Create your notebook and from a code cell download the Anaconda installer: !wget -c https://repo.continuum.io/archive/Anaconda3-5.1.0-Linux-x86_64.sh This is the version that works fine for me. I have tried also with the latest release 2019.10, but the configuration then would have extra complexity. Now you need to make the do...

During last Spark+AI Summit Europe 2019 I had a chance to attend a talk from Miguel Martinez  who was presenting Rapids , the new Open Source framework from NVIDIA for GPU accelerated end-to-end Data Science and Analytics. Fig. 1 - Overview of the Rapids eco-system Rapids is a suite of Open Source libraries: cuDF cuML cuGraph cuXFilter I enjoied the presentation and liked the idea of this initiative, so I wanted to start playing with the Rapids libraries in Python on Colab , starting from cuDF, but the first attempt came with an issue that I eventually solved. So in this post I am going to share how I fixed it, with the hope it would be useful to someone else running into the same blocker. I am assuming here you are already familiar with Google Colab. I am using Python 3.x as Python 2 isn't supported by Rapids. Once you have created a new notebook in Colab, you need to check if the runtime for it is set to use Python 3 and uses a GPU as hardware accelerator. You...

Following my recent talk at day 2 of the Big Data Days in Moscow, it seems that the training on Apache Spark of Deep Learning models is a hot topic. If you are in Dublin on Wednesday November 6th and want to hear more about it, please join the next Data Science Ireland Meetup , which would be hosted by Mason, Hayes & Curran . The event will start at 6 PM, it will be moderated by Mark Kelly from Alldus and there will be talks from Brian McElligott, Partner at Mason Hayes & Curran and me. You can find all of the details in the official Meetup page. At the moment this post is written the full capacity (120 places) has been reached and there is one person in the waiting list, but if you're interested in I suggest you to join any way the waiting list and monitor it, as some people could cancel also last minute. I hope to meet you there.

In the 3rd part of this series I am going to complete the description started in part 2 of the data preparation process for training and evaluation purposes of a LSTM model in time series forecasting. The data set used is the same as for part 1 and part 2. Same as for all of the post of this series, I am referring to Python 3. In part 2 we have learned how to transform the time series into a supervised model. That isn't enough yet to feed our LSTM model: other two transformations are needed. The first thing to do is to transform the time series data in order to make it stationary. The input time series used for these posts presents values that are dependent on time. Looking at its plotting in part 1, we can notice an increasing trend up to January 2006, then a decreasing trend up to February 2010 and finally a second increasing trend from there to date. Transforming this data to stationary makes life easier: we can remove the trends from the observed values before training and ...

In the second post of this series we are going to learn how to prepare data for training and evaluation of a LSTM neural network for time series forecasting. Same as for any other post of this series I am referring to Python 3. The data set used is the same as for part 1 . LSTM (Long-Short Term Memory) neural networks are a specialization of RNNs (Recurrent Neural Networks) introduced by Sepp Hochreiter and Jurgen Schmiduber in 1997 to solve the problem of the Vanishing Gradient affecting RNNs. LSTMs are used in real-world applications of language translation, text generation, image captioning, music generation and time series forecasting. You can find more info about LSTMs in my book or wait for one of my next posts of this series. This post focuses mostly on one of the best practices for data preparation before using a data set for training and evaluation of a LSTM in a time series forecasting problem with the Keras library. Let's load the data set first: from pandas im...

During the latest part of my career I had a chance to work with Data Scientists having strong skills in Python. My tech background, after a start with C/C++, is in JVM programming languages mostly (but I had to touch several others during my career), so it was a great chance for me to learn more about practical Python, at least in the Machine Learning and Deep Learning spaces. I am no more a full time developer since few years, as I moved to other roles, but this didn't stop me following my passion for programming languages and staying hands-on with technology in general, not only because I enjoy it, but also because I believe whatever your role is, it is always important to understand the possibilities and the limits of technologies before making any kind of decision. In this new long series I am going first to share some of the things I have learned about doing time series forecasting through Deep Learning using Python (with  Keras and Tensorflow ), and finally I will pres...

I hope to see you tonight at the April ODSC Dublin Meetup @ Jet.com in 40 Molesworth St . I am going to be the second speaker for the night. I am going to talk about importing pre-trained Keras and TensorFlow models into DL4J and the possibility of re-training them on Apache Spark . The first speaker would be John Kane from Cogito .

The second post of this series focuses on how to run a Livy server instance and start playing with its REST APIs. The steps below are meant for a Linux environment (any distribution). Prerequisites The prerequisites to start a Livy server are the following: The JAVA_HOME env variable set to a JDK/JRE 8 installation. A running Spark cluster. Starting the Livy server Download the latest version ( 0.4.0-incubating at the time this post is written) from the official website and extract the archive content (it is a ZIP file). Then setup the SPARK_HOME env variable to the Spark location in the server (for simplicity in this post I am assuming that the cluster is in the same machine as for the Livy server, but in the next post I will go through the customization of the configuration files, including the connection to a remote Spark cluster, wherever it is). By default Livy writes its logs into the $LIVY_HOME/logs location: you need to manually create this directory. Finally ...

I have started doing evaluation of Livy for potential case scenarios where this technology could help and I'd like to share some findings with others who would like to approach this interesting Open Source project. It has been started by Cloudera and Microsoft and it is currently in the process of being incubated by the Apache Software Foundation. The official documentation isn't comprehensive at the moment, so I hope my posts on this topic could help someone else. Apache Livy is a service to interact with Apache Spark through a REST interface. It enables both submissions of Spark jobs or snippets of Spark code. The following features are supported: The jobs can be submitted as pre-compiled jars, snippets of code or via Java/Scala client API. Interactive Scala, Python, and R shells. Support for Spark 2.x and Spark1.x, Scala 2.10 and 2.11. It doesn't require any change to Spark code. It allows long running Spark Contexts that can be used for multiple Spark jobs, by...

During an Hadoop cluster installation on Red Hat/CentOS using Apache Ambari ( https://ambari.apache.org/ ), a DataNode could fail to install with the following error: resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum -d 0 -e 0 -y install snappy-devel' returned 1. Error: Package: snappy-devel-1.0.5-1.el6.x86_64 (HDP-UTILS-1.1.0.20)            Requires: snappy(x86-64) = 1.0.5-1.el6            Installed: snappy-1.1.0-1.el6.x86_64 (@ftp3-updates)                snappy(x86-64) = 1.1.0-1.el6            Available: snappy-1.0.5-1.el6.x86_64 (HDP-UTILS-1.1.0.20)                snappy(x86-64) = 1.0.5-1.el6  You could try using --skip-broken to work around the problem  You could try...