When running Spark on Yarn do I need to install Spark on all nodes of Yarn Cluster

Question

I've got a 3 node YARN cluster and I want to run Spark on YARN. Do I need to install Spark on all nodes of yarn Cluster?

Answer 1

No, it is not necessary to install Spark on all the 3 nodes. Since spark runs on top of Yarn, it utilizes yarn for the execution of its commands over the cluster’s nodes.
So, you just have to install Spark on one node.

Comments

Hi ,

can you please tell What happens when yarn is started. How spark is introduced into the yarn cluster nodes? (which doesn't have spark in them)

---

Hi@Ashwin,

There are two deploy modes that can be used to launch Spark applications on YARN. In cluster mode, the Spark driver runs inside an application master process which is managed by YARN on the cluster, and the client can go away after initiating the application. In client mode, the driver runs in the client process, and the application master is only used for requesting resources from YARN.

To launch a Spark application in cluster mode:

$ ./bin/spark-submit --class path.to.your.Class --master yarn --deploy-mode cluster [options] <app jar> [app options]