Running Modes

Plz see Ch 10 and ch11 from spark in action book for more detailed discussion

When running in cluster mode, the driver runs on ApplicationMaster, the component that submits YARN container requests to the YARN ResourceManager according to the resources needed by the application. A simplified and high-level diagram of the application submission process is shown below.

When running in client mode, the driver runs outside ApplicationMaster, in the_spark-submit_script process from the machine used to submit the application.

Master URLs

http://spark.apache.org/docs/latest/submitting-applications.html

http://spark.apache.org/docs/latest/submitting-applications.html