Googlielmo's blog

 I just realized that I am using Streamlit since almost one year now, posted about in Twitter or LinkedIn several times, but never wrote a blog post about it before. Communication in Data Science and Machine Learning is the key. Being able to showcase work in progress and share results with the business makes the difference. Verbal and non-verbal communication skills are important. Having some tool that could support you in this kind of conversation with a mixed audience that couldn't have a technical background or would like to hear in terms of results and business value would be of great help. I found that Streamlit fits well this scenario. Streamlit is an Open Source (Apache License 2.0) Python framework that turns data or ML scripts into shareable web apps in minutes (no kidding). Python only: no front‑end experience required. To start with Streamlit, just install it through pip (it is available in Anaconda too): pip install streamlit and you are ready to execute the working de...

  The Codex paper has been published yesterday. Codex is a GPT language model finetuned on publicly available code from GitHub which has Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot . This paper focuses on the work leading to the early Codex models. The main task is the generation of standalone Python functions from docstrings, and the automated evaluation of the correctness of code samples through unit tests (this is in contrast to natural language generation, where samples are typically evaluated by heuristics or by human evaluators). To solve a problem in the test set, the authors generate multiple samples from the models, and check if any of them passes the unit tests. The raw training dataset was collected in May 2020 from 54 million public software repositories hosted on GitHub, containing 179 GB of unique Python files under 1 MB. Then it has been filtered by removing files which were likely auto-generated, had average line l...

 Jupyter notebooks allows LaTeX rendering inside markdown. This way you can write complex math equations within a notebook. While LaTeX is the de facto standard for scientific documents, it hasn't a very friendly and intuitive syntax. handcalcs is an Open Source library for converting Python calculations into rendered LaTeX: just write the symbolic formula, followed by numeric substitutions and that's it. After install it (it is available through PyPI), in the simplest case you just need to import the render class and use the %%render magic command to render the content of a cell: Here another example of equation render and numeric substitution: It is also possible to render just the symbolic equation: or any way generate the corresponding LaTeX code: By default handcalcs renders code vertically, but it is possible to use the %%render params magic to save space by rendering in a single line or show just the result of a calculation: handcalcs allows to adjust precision, use Gr...

  Faker is an Open Source Python package that generates synthetic data that could be used for many things such as populating a database, do load testing or anonymize production data for development or ML purposes. Generating fully random data isn't a good choice: with Faker you can drive the generation process and tailor the generated data to your specific needs: this is the greatest value provided by Faker. This package comes with 23 built-in data providers, some other providers are available from the community. The available data providers cover majority of data types and cases, but it is possible any way make the generated data more meaningful by implementing a custom provider. Faker supports Python 3.6+ and it is available for installation through PyPI or Anaconda.  Here's a code example that shows how to implement a custom provider to generate synthetic data following the structure and constraints as for this Kaggle  dataset related to a restaurant data with consumer...

 In my career I have noticed that often organizations are reluctant on providing Engineering teams with licenses for software to draw diagrams. In the best case scenarios MS Visio is usually the only option available, which isn't the best experience when trying to draw modern software architectures. Several online options are available, but they require to share project details that cannot leave your organization network, so they couldn't be taken into account often. Also, while treating everything as code, it would be nice to have also diagrams as code. All these needs can be satisfied by adopting Diagrams . It is an Open Source Python package that allows you draw cloud system architecture diagrams programmatically and then put them under version control, (as at the end they are regular Python files). It supports cloud (AWS, Azure, GCP, Alibaba, Oracle) and on-prem system architecture diagrams. The Diagrams nodes include also Kubernetes, programming languages and frameworks. ...

 Photo by Dinu J Nair on Unsplash Today I want to introduce  TagUI , an RPA (Robotic Process Automation) Open Source tool I am using to automate test scenarios for web applications. It is developed and maintained by the AI Singapore national programme. It allows writing flows to automate repetitive tasks, such as regression testing of web applications. Flows are written in natural language : English and other 20 languages are currently supported. Works on Windows, Linux and macOS. The TagUI official documentation can be found  here . The tool doesn't require installation: just go the official GitHub repository and download the archive for your specific OS (ZIP for Windows, tar.gz for Linux or macOS). After the download is completed, unpack its content in the local hard drive. The executable to use is named  tagui  (.cmd in Windows, .sh for other OS) and it is located into the  <destination_folder>/tagui/src  directory. In order to ...

After a 7 months hiatus I have decided to go back posting on this blog. Lot of things happened across 2020 and 2021 that left me with little or no time at all to share my thoughts and findings. In this long period of time I have been involved in challenging ML/AI projects, managing them and interacting with people 100% remotely because of the COVID-10 pandemic, had a chance to experiment with many and in some cases successfully applications of new DL architectures and Python Open Source libraries, but also tune mine and my family personal life among all the style changes imposed by the pandemic. The reasons that led me to restart the blog are mostly the following: I have accumulated tons of technical topics that are worth to share with a wider audience. During the past months I have shared some through social networks such as LinkedIn and Twitter or in few virtual meetups or conferences, but they need more deep dive. This week I gave a workshop at the ODSC Europe 2021 conference and I...

 I am back to the blog after some months where I have been pretty busy with interesting and challenging projects and coping with the new "normal" of the COVID-19 era.  Lot of things happened and I have several stuff to share in the upcoming months. Today I am going to speak at the Big Things 2020, the Data and AI conference. This year it moved virtual. Registration is still open and for free. I hope you have a chance to attend my talk at 7:45 PM GMT+1. I am going to discuss about Adversarial Attacks to Computer Vision systems and mitigation strategies. I hope to meet you there and do also some networking in the dedicated chat area. Follow-ups for this topic will be shared in this blog in the upcoming weeks.

The Spark + AI Summit North America edition 2020 is going virtual and access to keynotes, sessions and virtual events is for free. You have to pay only to attend pre-conference and conference training, AMA sessions, VIP sessions and certification exams. The Summit will happen from June 22nd to 26th 2020. This year's keynote speakers include Francois Chollet, the creator of Keras . All details in the official website: https://databricks.com/sparkaisummit/north-america-2020

In part 1 of this series a definition of CBIR has been given. Let's now understand what's the typical flow for it. The diagram in figure 1 shows that there are two parts, one that happens offline and another which is online: Figure 1 Starting from an image storage, a preliminary trained Deep Neural Network is used to extract the features from images. Extracted features are then stored in a feature database. This happens offline any time new images are added to the storage. What happens online is the search process itself. Any time a user uploads a query image, the same Deep Neural Network used for feature extraction is used to extract the input image features. Then the distance from the query image features and the features in the database are computed. The closer the distance, the higher the relevance is. The closest features are then sorted and the corresponding images are returned as results. Basic implementation: data set preparation To make things more clear,...

Content Based Image Retrieval (CBIR) systems are aimed to find similar images to an input image by querying an image data set. I am pretty sure you've used a CBIR several times before: here's the most popular one: The search in CBIRs isn't text-based as images are indexed by their visual content. Features are extracted from an image database and stored in some file system or object storage.Then, any time a new image is passed as search criterion, all those images which features are closest to the input image features are returned as results. The traditional feature extraction process relies on algorithms such as SURF or SIFT , but in this series I am going to describe my journey into CBIR using an unsupervised Deep Learning approach. Here's a preview of the first attempt, in order to search for food images (the input image in this case is an apple pie): In the next posts of this series I will walk through the details of the implementation before making thi...

During these strange days of emergency we could observe, among many others from the academy and the industry, lots of initiatives by individuals or groups of Data Scientists using public available data sets to make predictions about the evolution of the COVID-19 pandemic or other healthcare related applications to help in diagnosing the symptoms of the virus whether test kits for it wouldn't be available. Every little helps, it is also wonderful to see this high level of genuine interest on this matter and I am one of those that encourage people being curious and altruistic. But, as others have already started warning lately, any personal initiative, to be effective, needs to be evaluated by subject matter experts. In this post I am going to provide a concrete example about the importance of this kind of collaboration. At the end of 2019, I got a very bad flu which then ended up in an acute bronchitis, from which I recovered very slowly. During the rest of my life, after my ch...

I am so excited to be attending the Shift AI conference this April in Dublin! It's @shiftconf_co 's #ArtificialIntelligence conference where we will discuss the newest technologies in data mining, machine learning and neural networks. While Ireland is a very active hub for AI in Europe, so far there has been a lack of good conferences happening here: so, Shift AI is welcome! I hope to see you at the event. Learn more over at https://ctt.ec/2K2o6+

I hope you are going to attend the 8th edition of Big Things 2019 , the data and AI conference which will happen in Madrid (Spain) on November 20th and 21st. This conference started in 2012 and this year changed its name from Big Data Spain to Big Things, as it became focused not only on Big Data, but also on whatever is related to AI. Among the speakers this year there will be big names such as Cassie Kozyrkov, Alberto Cairo, Jesse Anderson, Michael Armbrust, Suneel Marthi, Paco Nathan and many others. My talk will be on Thursday 21st at 1:55 PM local time. I am going to give an update on importing and re-training Keras / TensorFlow models in DL4J and Apache Spark . It is a follow-up of some of the topics covered in my book , considering changes related to new releases of DL4J, Keras and TensorFlow since it has been published in January this year. Please stop by if you are going to attend my talk and the conference. I really appreciated the conversations about Deep Learning ...

Colab is the Google's platform for working with Python notebooks and practice Deep Learning using different frameworks. It is a powerful platform, there is availability of GPUs or TPUs, it allows to use your Google Drive space for notebooks and data, has a friendly user interface and lots of useful features, but in order to install/update Python packages, it comes by default only with pip and no availability for conda . If you need to import a Python package which is available in Anaconda , but not in PyPi you need to install Anaconda or Miniconda yourself from a notebook. In this post I am explaining the simple steps to do it. Anaconda installation Create your notebook and from a code cell download the Anaconda installer: !wget -c https://repo.continuum.io/archive/Anaconda3-5.1.0-Linux-x86_64.sh This is the version that works fine for me. I have tried also with the latest release 2019.10, but the configuration then would have extra complexity. Now you need to make the do...

During last Spark+AI Summit Europe 2019 I had a chance to attend a talk from Miguel Martinez  who was presenting Rapids , the new Open Source framework from NVIDIA for GPU accelerated end-to-end Data Science and Analytics. Fig. 1 - Overview of the Rapids eco-system Rapids is a suite of Open Source libraries: cuDF cuML cuGraph cuXFilter I enjoied the presentation and liked the idea of this initiative, so I wanted to start playing with the Rapids libraries in Python on Colab , starting from cuDF, but the first attempt came with an issue that I eventually solved. So in this post I am going to share how I fixed it, with the hope it would be useful to someone else running into the same blocker. I am assuming here you are already familiar with Google Colab. I am using Python 3.x as Python 2 isn't supported by Rapids. Once you have created a new notebook in Colab, you need to check if the runtime for it is set to use Python 3 and uses a GPU as hardware accelerator. You...

Following my recent talk at day 2 of the Big Data Days in Moscow, it seems that the training on Apache Spark of Deep Learning models is a hot topic. If you are in Dublin on Wednesday November 6th and want to hear more about it, please join the next Data Science Ireland Meetup , which would be hosted by Mason, Hayes & Curran . The event will start at 6 PM, it will be moderated by Mark Kelly from Alldus and there will be talks from Brian McElligott, Partner at Mason Hayes & Curran and me. You can find all of the details in the official Meetup page. At the moment this post is written the full capacity (120 places) has been reached and there is one person in the waiting list, but if you're interested in I suggest you to join any way the waiting list and monitor it, as some people could cancel also last minute. I hope to meet you there.

If you are interested on hearing more about some way to do predictions of Apache Spark applications performance, please join tomorrow's Dublin Data Science Festival Meetup  which would start at 6 PM  local time at the Walmart Labs place . Two talks in agenda, the first one from Mirko Arnold (Walmart Labs) about Computer Vision and the second one from me. It would also be another great opportunity for networking.

You're still in time to get some interesting Machine Learning and AI eBooks and videos for $10 each at Packt Publishing . This promotion covers also my book " Hands-on Deep Learning with Apache Spark ". What are you waiting for? Go and check out for titles before the offer expires! By the way, if you want to listen to follow-ups on topics covered by my book and get in touch with me in person, I am going to give talks at the following events in October and November: Big Data Days , Moscow, Russian Federation, October 8th-10th Spark+AI Summit Europe , Amsterdam, Netherlands, October 16th-17th Big Things Conference , Madrid, Spain, November 20th-21st

I am glad I have been selected to speak at the next Spark+AI Summit Europe 2019 , which will happen in Amsterdam, Netherlands on October 15th-17th 2019. I am going to present some follow-up of one of the core topics of my book: memory management in distributed Deep Learning with DL4J on Apache Spark . More details about my talk will follow in the next weeks. As usual, the summit will have an impressive line-up of speakers, such as Matei Zaharia, Ali Ghodsi, Holden Karau, Luca Canali, Gael Varoquaux, Christopher Crosbie, Michael Armbrust and many others. I hope you will attend this event.