Java vs Scala in Spark development

Developing things for Spark sometimes you don't have a choice in terms of language to use (it is the case of the GraphX APIs, where Scala is the only choice at the moment), but in some other cases you can choose between two different JVM languages (Java or Scala). Coming from a long background in Java and from my shorter experience in Scala, I can say that for sure some advantages using Scala in Spark programming are a better compactness and readability of the code. Have a look at the following simple Java code taken from one of the examples bundled with the Spark distribution:

import java.io.Serializable;
import java.util.List;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.sql.DataFrame;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SQLContext;

public class JavaSparkSQL {
    public static class Person implements Serializable {
        private String name;
        private int age;

        public String getName() {
          return name;
        }

        public void setName(String name) {
          this.name = name;
        }

        public int getAge() {
          return age;
        }

        public void setAge(int age) {
          this.age = age;
        }
    }
   
    public static void main(String[] args) throws Exception {
        SparkConf sparkConf = new SparkConf().setAppName("JavaSparkSQL");
        JavaSparkContext ctx = new JavaSparkContext(sparkConf);
        SQLContext sqlContext = new SQLContext(ctx);

        System.out.println("=== Data source: RDD ===");
        // Load a text file and convert each line to a Java Bean.
        JavaRDD<Person> people = ctx.textFile("examples/src/main/resources/people.txt").map(
          new Function<String, Person>() {
            public Person call(String line) {
              String[] parts = line.split(",");

              Person person = new Person();
              person.setName(parts[0]);
              person.setAge(Integer.parseInt(parts[1].trim()));

              return person;
            }
          });
       
        // Apply a schema to an RDD of Java Beans and register it as a table.
        DataFrame schemaPeople = sqlContext.createDataFrame(people, Person.class);
        schemaPeople.registerTempTable("people");
       
        // SQL can be run over RDDs that have been registered as tables.
        DataFrame teenagers = sqlContext.sql("SELECT name FROM people WHERE age >= 13 AND age <= 19");
       
        // The results of SQL queries are DataFrames and support all the normal RDD operations.
        // The columns of a row in the result can be accessed by ordinal.
        List<String> teenagerNames = teenagers.toJavaRDD().map(new Function<Row, String>() {
          public String call(Row row) {
            return "Name: " + row.getString(0);
          }
        }).collect();
        for (String name: teenagerNames) {
          System.out.println(name);
        }
       
        ctx.stop();
    }

}

The snippet above starts a Spark SQL Context, creates a Data Frame loading a comma separated content from a text file and mapping it using the Person POJO declared as inner class and finally executes a SQL statement on it printing the result to the standard output. One way to achieve the same in Scala could be the following:

import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.sql.SQLContext

object ScalaSparkSQL {
  case class Person(name: String, age: Int)
 
  def main(args: Array[String]) {
    val sparkConf = new SparkConf().setAppName("TxtOrJsonRelational").setMaster(args(0))
    val ctx = new SparkContext(sparkConf)
    val sqlContext = new SQLContext(ctx)
   
    // Importing the SQL context to give access to all the SQL functions and implicit conversions.
    import sqlContext.implicits._
   
    // Create a DataFrame from a text file
    val peopleDf =
        ctx.textFile("examples/src/main/resources/people.txt")
        .map(_.split(","))
        .map(p => Person(p(0), p(1).trim.toInt))
        .toDF()
   
    // Any RDD containing case classes can be registered as a table.  The schema of the table is
    // automatically inferred using scala reflection.
    peopleDf.registerTempTable("people");
   
    // Execute a SQL statement
    val teenagers = sqlContext.sql("SELECT name FROM people WHERE age >= 13 AND age <= 19")
    teenagers.show()
   
    ctx.stop()
  }
}

You can immediately notice less verbosity and code easier to read/maintain. And this is just a very basic example: imagine the benefits when writing complex programs.
Definitively I believe that Scala is a better choice than Java in Spark development, but at present time I cannot say if it is to be preferable also in different contexts. Many other factors and the real technical needs have to be taken into account for each single project.