Exploring the Spline Data Tracker and Visualization tool for Apache Spark (Part 1)

One interesting and promising Open Source project that caught my attention lately is Spline, a data lineage tracking and visualization tool for Apache Spark, maintained at Absa. This project consists of 2 parts: a Scala library that works on the drivers which, by analyzing the Spark execution plans, captures the data lineages and a web application which provides a UI to visualize them.
Spline supports MongoDB and HDFS as storage systems for the data lineages in JSON format. In this post I am referring to MongoDB.
You can start playing with Spline through the Spark shell. Just add the required dependencies to the shell classpath as follows (with reference to the latest 0.3.5 release of this project):

spark-shell --packages "za.co.absa.spline:spline-core:0.3.5,za.co.absa.spline:spline-persistence-mongo:0.3.5,za.co.absa.spline:spline-core-spark-adapter-2.3:0.3.5"

Running the Spark shell with the command above on Ubuntu and some other Linux distro, whether some issue on downloading the Joda Time library (transitive dependency for one of the Spline components) should occur, please delete the .ivy1 and .m2 hidden sub-directories of the directory where the spark-shell command has been executed and then re-run it.
Assuming you have your Mongo server up and running and that you have already created an empty database for Spline, the first thing you need to do in the Spark shell is to specify the persistence factory class to use and then the connection string and the database name:

System.setProperty("spline.persistence.factory", "za.co.absa.spline.persistence.mongo.MongoPersistenceFactory")
System.setProperty("spline.mongodb.url", "mongodb://<username>:<password>@<server_name_or_ip>:<port>")
System.setProperty("spline.mongodb.name", "<database_name>")

You can now enable the Spline data lineage tracking:

import za.co.absa.spline.core.SparkLineageInitializer._
spark.enableLineageTracking()

and then start doing something which involves data:

val employeesJson =
    spark.read.json("/home/guglielmo/spark-2.3.2-bin-hadoop2.7/examples/src/main/resources/employees.json")

import spark.implicits._
val employeeNames = employeesJson.select(employeesJson("name"))
employeeNames.write.parquet("/home/guglielmo/spline/example/employee_names")

Whether the following exception should happen:

com.mongodb.MongoCommandException: Command failed with error 9: 'The 'cursor' option is required, except for aggregate with the explain argument' on server localhost:27017. The full response is { "ok" : 0.0, "errmsg" : "The 'cursor' option is required, except for aggregate with the explain argument", "code" : 9, "codeName" : "FailedToParse" }

then you have to update the MongoDB Java driver dependency to any release 3.6+ (it could be done by simply adding it to the list of packages when running the spark-shell command).
Starting the Spline web application:

java -jar spline-web-0.3.5-exec-war.jar -Dspline.mongodb.url=mongodb://<username>:<password>@<server_name_or_ip>:<port> -Dspline.mongodb.name=<database_name>

you can see the captured data lineage in the web UI (the default listening port is 8080):

This is just a starter. In part 2 of this series we are going to explore Spline under the hood.