Skip to main content

Posts

Quick guide to install GitLab (Community Edition)

A quick tutorial on how to install GitLab ( https://about.gitlab.com/ ) CE (Community Edition). The steps described in this post have been verified and tested on Red Hat Linux Server 6.x 64-bit, but they should work also on Red Hat 7.x, CentOS 6.x+, Oracle 6.x+ and Scientific Linux 6.x+ distributions.   Installation. We are going to use the Omnibus package. It includes almost all of the packages needed, but OpenSSH and Postfix. These are the installation steps (all of the commands below need to be issued with a user having root privileges.):  - Check whether the following prerequisites are present in the system: OpenSSH Server ( http://www.openssh.com/ ) and Postfix ( http://www.postfix.org/ ). If not, before proceeding in the GitLab installation, please install them following the official documentation and the specific steps for your Linux distribution.  - Download the Omnibus package (the command below is for CentOS 6.x/Red Hat 6.x/Oracle/Scientific Linux distribu...

HDFS file commands quick reference

Here's a quick reference of the most frequently used (and useful) HDFS (Hadoop Distributed File System) commands to manage files. $HADOOP_HOME/bin/hadoop fs -ls / Lists all of the files in the root HDFS directory. $HADOOP_HOME/bin/hadoop fs -ls /rawdata/server01 Lists all of the files in the HDSF directory at the given path. $HADOOP_HOME/bin/hadoop fs -mkdir /rawdata/ Creates a new directory in HDFS. $HADOOP_HOME/bin/hadoop fs -put /home/user/importdir/*.txt /rawdata/ Copies files from a local directory to HDFS at the specified path. The destination directory should have been created preliminarily. $HADOOP_HOME/bin/hadoop fs -get /rawdata/test01.txt /home/user/importdir/ Copies files from HDFS back to the local filesystem. $HADOOP_HOME/bin/hadoop fs -cp /rawdata/test01.txt /rawdatabackup/test01.txt Copies files within HDFS. $HADOOP_HOME/bin/hadoop fs -rm /rawdata/*.txt Deletes files from a HDFS directory. $HADOOP_HOME/bin/hadoop fs -rm -r /rawdata Deletes a ...

A web client for the Jolokia agents

I have just released the version 0.2 of jolokiaui ( https://github.com/virtualramblas/jolokiaui ), a web client for the Jolokia JVM agent ( https://jolokia.org/agent/jvm.html ) to monitor local and remote JVMs. It is implemented in R ( https://www.r-project.org/ ) using the Shiny framework ( http://shiny.rstudio.com/ ).  The current release is still an alpha, but new features would be added in the next months. Any feedback is welcome. Enjoy it.

One good reason to move to Java 8 (if you haven't already done so)

One of the most useful things introduced by Java 8 is the Optional class ( https://docs.oracle.com/javase/8/docs/api/java/util/Optional.html ). It is a container object that wraps either a value or null , to represent the absence of a value. You can quickly understand the benefits coming from it: a better way to handle the existence of null values with no need to explicitly check for nulls and possibly less chance to have NullPointerException exceptions. Some code like this // The method call below could return null User currentUser = getUser(id); Role role = null; if(currentUser != null) {   role = getRole(currentUser); } becomes: Optional<User> currentUser = Optional.ofNullable(getUser(id)); Optional<Role> role = currentUser.map(u -> getRole(currentUser));

IllegalStateException from Mongo storage plugin in Apache Drill

Apache Drill ( https://drill.apache.org/ ) is an Open Source framework that supports data-intensive distributed applications for interactive analysis of large scale datasets. It supports several kinds of filesystems and NoSQL databases, including HDFS and MongoDB. In this short post I want to show an exception you could deal with if your Drill installation has the MongoDB storage plugin enabled. If suddenly, after more than 24 of uptime, any SQL query (not only those against a MongoDB database) issued through Drill throw the following exception: SYSTEM ERROR: IllegalStateException: state should be: open [Error Id: 57a02508-1920-4360-a111-c2a55a7af15c on hostname:31010]] this should be related to a connection cache that expires and isn't automatically reset. This issue affects in particular the latest release (1.1.0) of Drill and it is still marked as unresolved at the time of this post writing ( https://issues.apache.org/jira/browse/DRILL-3522 ), but there is a patch available...

MRUnit Tutorial

Apache MRUnit ( https://mrunit.apache.org/ ) is an Open Source library that allows unit-testing for Hadoop Mappers, Reducers, and MapReduce programs. It provides a convenient integration between MapReduce and standard testing libraries such as JUnit and Mockito and helps (providing a set of interfaces and test harnesses) bridging the gap between MapReduce programs and those traditional libraries. It doesn't replace JUnit, but works on top of it. Before reading further, please be aware that knowledge of Hadoop MapReduce and JUnit is required for a better understanding of this post. The three core classes of MRUnit are the following: • MapDriver : the driver class responsible for calling the Mapper’s map() method. • ReducerDriver : the driver class responsible for calling the Reducer’s reduce() method. • MapReduceDriver : the combined MapReduce driver responsible for calling the Mapper’s map() method first, followed by an in-memory Shuffle phase. At the end of this phase the ...

Living with uncertainty

Just a quick note about Scrum (and Agile in general) on a concept that is often ignored. Scrum is intended for those kinds of work that defined processes have often failed to manage: uncertain requirements combined with unpredictable technology implementation risks (this distinction is not a blasphemy in the Agile context). These conditions usually exist during a new product (in particular software products) development: no matter how carefully you plan the future, it can never be more than a dream, unless you adjust your plan every day. So don't be scared by uncertainty, be ready to live with it and work hard to reduce it. This is what makes the difference between a good member of an Agile Team and a bad one: the good one manages and doesn't ignore uncertainty. This is true at each level of any self proclaimed agile company.