spark-exercise-custom-ExternalClusterManager

Goal

The goal of the exercise is to register a custom ExternalClusterManager so scalania master URL is supported and the following works:

$ ./bin/spark-shell --master scalania
Error: Master must either be yarn or start with spark, mesos, local
Run with --help for usage help or --verbose for debug output

Steps

1. Create a brand new Scala/sbt project

   • Use the one and only IntelliJ IDEA

   • Project name: spark-external-cluster-manager

2. Add libraryDependencies for Spark 2.0.0 (RC5)

   • See https://gist.github.com/jborkowski/a77d9cc88067feea24261479afcc4ee1

3. ScalaTest

   • Follow Spark's own ExternalClusterManagerSuite

Hints

1. The custom Cluster Manager should extend ExternalClusterManager

2. The cluster manager support is loaded by ServiceLoader.load. See the code in Spark

3. See how and where the loading happens in Spark.

Solution

See a complete Scala/sbt project in spark-external-cluster-manager. It is a work in progress work.