Googlielmo's blog

One of the most used managed destinations in the SDC is the JDBC Producer. It allows data writing to a relational database table using a JDBC connection. The SDC release I am referring to in this post is the 1.5.1.2 running in the JVM 8. Installing a specific JDBC driver. In order to insert data in a database table SDC requires the specific JDBC driver for the database you need to use. This applies to the JDBC consumer origin as well. The first time you plan to add  a JDBC Producer destination to a pipeline you need to create a local directory in the SDC host machine external to the SDC installation directory. Example: /home/sdc-user/sdc-extras Then create the following sub-directory structure for all of the JDBC drivers: /home/sdc-user/sdc-extras/streamsets-datacollector-jdbc-lib/lib/ Finally copy the JDBC driver in that folder. Now it is time to make SDC aware of this directory. First you have to add the STREAMSETS_LIBRARIES_EXTRA_DIR environment variable and make it ...

StreamSets Data Collector (SDC) allows user authentication based on files or LDAP. By default, Data Collector uses file authentication. This post gives you details on how to switch to use your company's LDAP. To enable LDAP authentication you need to perform the following tasks: - Configure the LDAP properties for the Data Collector configuration editing the $SDC_CONF/sdc.properties file:      - set the value of the http.authentication.login.module property to ldap      - configure the value of the http.authentication.ldap.role.mapping property to map your LDAP groups to Data Collector roles following this syntax:             <LDAP_group>:<SDC_role>,<additional_SDC_role>,<additional_SDC_role>         Multiple roles can be mapped to the same group or vice versa. You need to use a semicolon to separate LDAP groups and commas to separate Data...

Before moving on to the other planned posts for this series, an important update about the Data Collector. Two new versions (1.3.0 and 1.3.1) that solve some critical bugs and introduce new features have been released since the first post publishing. This is a short list of the most significant benefits you get moving to the new releases: No issue running the Data Collector as for https://issues.streamsets.com/browse/SDC-2657 The workaround for this issue was to downgrade to the release 1.2.1.0, this way missing an important new feature like the Groovy Evaluator processor. The Hadoop FS and Local FS destination can now write files larger that 2 GB. A MongoDB destination is now available (up to release 1.2.2.0 a MongoDB database could have been set as origin only). Two new processors, Base64 Field Decoder and Base64 Field Encoder, have been implemented to work with Base64 binary data encoding/decoding. Enjoy it!

Before going further a quick post about topic deletion in Kafka (someone asked me about this). In part 2 of this series we created a topic called kafkatesting for testing purposes and to get familiar with the Java APIs to implement producers and consumers. When you're done with testing you will need to delete it. This could be done running the following command from a shell: $KAFKA_HOME/bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic kafkatesting Then if you check the list of existing topics for that cluster you could still see the topic there having the label "marked for deletion". This happens when you use the default properties file for Kafka or you didn't explicitly set to true the value of the delete.topic.enable property (the default value for it is false ) in your custom copy of that file. In order to make this configuration change effective you have to restart both Kafka and ZooKeeper.

StreamSets Data Collector ( https://streamsets.com/product/ ) is an Open Source lightweight and powerful engine that streams data in real time. It allows to configure data flows as pipelines through a web UI in few minutes. Among its many features, it makes possible to view real-time statistics and inspect data as it passes through the pipeline. In the first part of this series I am going to show the installation steps to run the Data Collector manually. I am referring to the release 1.2.1.0. The latest one (1.2.2.0) comes with a bug that prevents it to start (I have opened a ticket in the official Jira for this product ( https://issues.streamsets.com/browse/SDC-2657 ), but it is still unresolved at the time this post is written). The prerequisites for the installation are: OS: RedHat Enterprise Linux 6 or 7 or CentOS 6 or 7 or Ubuntu 14.04 or Mac OS X.  Java: Oracle or IBM JDK 7+. And now the installation steps:  - Download the full StreamSets Data Collector tar...

In the second part of this series I will describe the steps to install a Kafka single node-single broker cluster on a Linux machine. Here I am referring to the latest Kafka stable version (at the time of writing this post), 0.9.0.1, Scala 2.11. Prerequisites The only prerequisite needed is a JDK 7+. Installation - Move to the opt folder of your system    cd /opt   and then download the binaries of the latest release there:     wget http://www.us.apache.org/dist/kafka/0.9.0.1/kafka_2.11-0.9.0.1.tgz - Extract the archive content:     tar xzf kafka_2.11-0.9.0.1.tgz - Create the KAFKA_HOME variable:     echo -e "export KAFKA_HOME=/opt/kafka_2.11-0.9.0.1" >> /root/.bash_profile - Add the Kafka bin folder to the PATH:     echo -e "export PATH=$PATH:$KAFKA_HOME/bin" >> /root/.bash_profile - Reload the bash profile for the user:     source /root/.b...

Whether you should fail to start a mongod service on Linux and see in the database logs an error message like the following: Failed to unlink socket file /tmp/mongodb-27017.sock errno:1 Operation not permitted Its root cause is a permission issue: the previous restart/stop as root user didn't complete properly, so the temporary .sock file couldn't be deleted. Delete the file manually as root before restarting the service and the sun will shine again.

During an Hadoop cluster installation on Red Hat/CentOS using Apache Ambari ( https://ambari.apache.org/ ), a DataNode could fail to install with the following error: resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum -d 0 -e 0 -y install snappy-devel' returned 1. Error: Package: snappy-devel-1.0.5-1.el6.x86_64 (HDP-UTILS-1.1.0.20)            Requires: snappy(x86-64) = 1.0.5-1.el6            Installed: snappy-1.1.0-1.el6.x86_64 (@ftp3-updates)                snappy(x86-64) = 1.1.0-1.el6            Available: snappy-1.0.5-1.el6.x86_64 (HDP-UTILS-1.1.0.20)                snappy(x86-64) = 1.0.5-1.el6  You could try using --skip-broken to work around the problem  You could try...

A quick tutorial on how to install GitLab ( https://about.gitlab.com/ ) CE (Community Edition). The steps described in this post have been verified and tested on Red Hat Linux Server 6.x 64-bit, but they should work also on Red Hat 7.x, CentOS 6.x+, Oracle 6.x+ and Scientific Linux 6.x+ distributions.   Installation. We are going to use the Omnibus package. It includes almost all of the packages needed, but OpenSSH and Postfix. These are the installation steps (all of the commands below need to be issued with a user having root privileges.):  - Check whether the following prerequisites are present in the system: OpenSSH Server ( http://www.openssh.com/ ) and Postfix ( http://www.postfix.org/ ). If not, before proceeding in the GitLab installation, please install them following the official documentation and the specific steps for your Linux distribution.  - Download the Omnibus package (the command below is for CentOS 6.x/Red Hat 6.x/Oracle/Scientific Linux distribu...

An appendix to complete the tutorial about the Shiny Server (OS edition) installation: a quick reference to its configuration. The configuration file of a Shiny Server is stored in the /etc/shiny-server/shiny-server.conf file. It comes in a format like this: run_as shiny; server {   listen 3838;   location / {     site_dir /srv/shiny-server;         log_dir /var/log/shiny-server;         directory_index on;   } } Here's the explanation of the parameters: run_as : to set the system user to run R/Shiny applications. Don't use root. server : this block defines a single server details (see the following properties). listen : the server listening port. The default value is 3838. location : defines a location available at the base URL. site_dir : the directory hosting the Shiny web apps. log_dir : the directory where the Shiny Server logs are stored. directory_index : to enable ( on ) ...

RHadoop ( https://github.com/RevolutionAnalytics/RHadoop/wiki ) is a collection of five R packages (rhdfs, rmr2, rhbase, ravro, plyrmr) that allow users to manage and analyze data with Hadoop. Running any MapReduce function, also this simple one     from.dfs(mapreduce(to.dfs(1:100)))  through RHadoop on Linux servers you could face this exception: 2015-10-20 08:39:41,722 ERROR [main] org.apache.hadoop.streaming.PipeMapRed: configuration exception java.io.IOException: Cannot run program "Rscript": error=2, No such file or directory     at java.lang.ProcessBuilder.start(ProcessBuilder.java:1059)     at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:209)     at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav...

Shiny Server (https://www.rstudio.com/products/shiny/shiny-server/ ) is a server to put Shiny ( http://shiny.rstudio.com/ ) applications or interactive documents available over the web. The official documentation seems quite confusing to me, so I want to share this quick tutorial for people approaching it for the first time. The procedure explained in this post has been tested and replicated on several Linux Red Hat Servers 6.6+ 64-bit, but it should work on any Linux distribution that allows RPM management. The overall procedure could be of course automated, but for better understanding of this tutorial I am going to describe any manual step involved. Installation Connect to your Linux server as root and download the latest RPM for your specific architecture: wget https://download3.rstudio.org/centos5.9/x86_64/shiny-server-1.4.0.721-rh5-x86_64.rpm and then install it using yum: yum install --nogpgcheck shiny-server-1.4.0.721-rh5-x86_64.rpm At the end of the installation p...