Googlielmo's blog

I have the pleasure of being invited this week to do a live webinar along with Pat Patterson at Streamsets. The title of the webinar is "Ultralight Data Movement for IoT with SDC Edge". Here's the link: https://go.streamsets.com/webinar-2018-06-27-ultralight-data-movement-for-iot-with-sdc-edge.html?es_p=7008945 I hope you will join us on Wednesday June 27th at 10 AM PT.

If you are in Munich on June 12th and want to know how to make your IoT data streaming infrastructure easier to maintain, please try to attend my talk at the Predictive Analytics World for Industry 4.0 . It will start at 3:30 PM. I hope to meet you there!

Thank you for attending my talk at the Data Driven Innovation Open Summit 2018 in Rome. I will post to YouTube the videos I have prepared about the 2 demos of the SDC and SDC Edge originally planned for this talk as soon as I complete the audio comment and some small editing.

AI, Machine Learning and Deep Learning are getting an hype nowadays even if most part of the algorithms and models at their core are around since long time: 1805 Least Squares 1812 Bayes' Theorem 1913 Markov Chains 1950 Turing's Learning Machine 1957 Perceptron 1967 Nearest Neighbor 1970 Automatic Differentiation 1972 TF-IDF 1980 Neocognitron 1981 Explanation Based Learning 1982 Recurrent Neural Network 1970 Back Propagation 1989 Reinforcement Learning 1995 Random Forest Algorithm 1995 Support Vector Machines 1997 LSTM So what are the reasons that speed up and accelerated the implementation and made possible today for the theory to become reality? There are several factors:  - Cheaper computation: in the past hardware was a constraining factor for AI/ML/DL. Late advance in hardware (coupled with improved tools and software frameworks) and new computational models (in particular around GPUs) have accelerated AI/ML/DL adoption.  - Cheaper storage: the increased number of availab...

Do you believe that Java is dead? Please join the Java with the Best online conference on April 17th-18th and then probably you will change your mind. Of course if you are a Java technology passionate you need to attend as well ;) 2 days, 50 talks, 3 parallel sessions (Core Java, Big Data, Machine Learning and Cloud Development with Java, Java Frameworks, Libraries and Languages).  I hope you will get a chance to attend my talk:

Few days ago Pat Patterson published an excellent article on DZone about Evolving Avro Schemas With Apache Kafka and StreamSets Data Collector. I recommend reading this interesting article. I followed this tutorial and today I want to share the details on how I quickly setup the environment for this purpose, just in case you should be interested on doing the same. I did it on a Linux Red Hat Server 7 (but the steps are the same for any other Linux distro) and using only images available in the Docker Hub . First start a Zookeeper node (which is required by Kafka ): sudo docker run --name some-zookeeper --restart always -d zookeeper and then a Kafka broker, linking the container to that for Zookeeper: sudo docker run -d --name kafka --link zookeeper:zookeeper ches/kafka Then start the Confluent Schema Registry (linking it to Zookeeper and Kafka): sudo docker run -d --name schema-registry -p 8081:8081 --link zookeeper:zookeeper --link kafka:kafka confluent/schema-registry ...

A hot topic in the sdc-user group during the past weeks has been about how to schedule the start and stop of SDC pipelines. Usage of the SDC REST APIs has been suggested in some threads, but because the general impression I have is that the audience doesn't have a clear idea about them, I decided to write an article on DZone to help and clarify once and for all how to do it. Enjoy it!

My first Open Source Hubot script has been released and is available in my GitHub space . It provides support to check the status of pipelines in a Streamsets Data Collector server. It is still in alpha release, but the development is ongoing, so new features and improvements will be constantly implemented. Enjoy it!

If you are interested in implementing a custom SDC origin which supports parallel execution then you can read my new DZone article . You can find this tutorial (with updates) in my GitHub space as well. Enjoy it!

The release 1.6.0.0 of the Streamsets Data Collector has been released on September 1st. This release comes with an incredible number of new features. Here are some of the most interesting: JDBC Lookup processor: it can perform lookups in a database table through a JDBC connection and then you can use the values to enrich records. JDBC Tee processor: it can write data to a database table through a JDBC connection, and then you can pass generated database column values to fields.  Support for reading data from paginated webpages through the HTTP origin. Support for Apache Kafka 0.10 and ElasticSearch 2.3.5. Enterprise security in the MongoDB origin and destination including SSL and login credentials. Whole File Data format: to move entire files from an origin system (Amazon S3 or Directory) to a destination system (Amazon S3, HDFS, Local File System or MapR FS). Using the whole file data format, you can transfer any type of file.  And many more. ...

One of the most used managed destinations in the SDC is the JDBC Producer. It allows data writing to a relational database table using a JDBC connection. The SDC release I am referring to in this post is the 1.5.1.2 running in the JVM 8. Installing a specific JDBC driver. In order to insert data in a database table SDC requires the specific JDBC driver for the database you need to use. This applies to the JDBC consumer origin as well. The first time you plan to add  a JDBC Producer destination to a pipeline you need to create a local directory in the SDC host machine external to the SDC installation directory. Example: /home/sdc-user/sdc-extras Then create the following sub-directory structure for all of the JDBC drivers: /home/sdc-user/sdc-extras/streamsets-datacollector-jdbc-lib/lib/ Finally copy the JDBC driver in that folder. Now it is time to make SDC aware of this directory. First you have to add the STREAMSETS_LIBRARIES_EXTRA_DIR environment variable and make it ...

StreamSets Data Collector (SDC) allows user authentication based on files or LDAP. By default, Data Collector uses file authentication. This post gives you details on how to switch to use your company's LDAP. To enable LDAP authentication you need to perform the following tasks: - Configure the LDAP properties for the Data Collector configuration editing the $SDC_CONF/sdc.properties file:      - set the value of the http.authentication.login.module property to ldap      - configure the value of the http.authentication.ldap.role.mapping property to map your LDAP groups to Data Collector roles following this syntax:             <LDAP_group>:<SDC_role>,<additional_SDC_role>,<additional_SDC_role>         Multiple roles can be mapped to the same group or vice versa. You need to use a semicolon to separate LDAP groups and commas to separate Data...

In order to show that the considerations done in my last post are general for any log shipping purpose, let's see now how the same process applies to a more real use case scenario: the log shipping and analysis of a MongoDB database logs. MongoDB logs pattern Starting from the release 3.0 (I am considering the release 3.2 for this post) the MongoDB logs come with the following pattern: <timestamp> <severity> <component> [<context>] <message> where:     timestamp is in iso8601-local format.     severity is the level associated to each log message. It is a single character field. Possible values are F (Fatal), E (Error), W (Warning), I (Informational) and D (Debug).     component is for a functional categorization of the log message. Please refer to the specific release of MongoDB you're using to know the full list of possible values.     context is the specific context for a me...

One common use case scenario for the Streamsets Data Collector (SDC) is the log shipping to some system, like ElasticSearch, for real-time analysis. To build a pipeline for this particular purpose in SDC is really simple and fast and doesn't require coding at all. For this quick tutorial I will use the SDC logs as example. The log data will be shipped to Elasticsearch and then visualized through a Kibana dashboard. Basic knowledge of SDC, Elasticsearch and Kibana is required for a better understanding of this post. These are the releases I am referring to for each system involved in this tutorial: JDK 8 Streamsets Data Collector 1.4.0 ElasticSearch 2.3.3 Kibana 4.5.1 Elasticsearch and Kibana installation You should have your Elasticsearch cluster installed and configured and a Kibana instance pointing to that cluster in order to go on with this tutorial. Please refer to the official documentation for these two products in order to complete their installation (if you do...

If you're involved in Big Data stuff and in the Dublin area on Monday June 13th 2016 I hope you have a chance to attend the monthly meetup event of the Hadoop User Group Ireland: http://www.meetup.com/hadoop-user-group-ireland/events/231165491/ The event will be hosted by Bank of Ireland in their premises in Grand Canal Square ( https://goo.gl/maps/bbh1XghukfJ2 ). I will do the second talk " Building a data pipeline to ingest data into Hadoop in minutes using Streamsets Data Collector ". The event will start at 6 PM Irish Time. I hope to meet you there.