How to use ftp scheme using Yarn in Spark application

Question

Hi. I want to use FTP for my Spark application. I have learnt that Spark supports this but the problem is that I am using Yarn with spark and Yarn does not support this. I need to have the scheme in the local disk. I want to know how I can download schemes for Yarn. Please help

Answer 1

In case Yarn does not support schemes that Spark supports, you will have to download the schemes on local disk before adding to Yarn's cache. You can use the in-built feature of Spark that let's you download the schemes that you want. Refer to the below command to do this:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.yarn.dist.forceDownloadSchemes= <list of schemes>