Trending questions in Apache Spark

0 votes
1 answer

Need help setting Spark yarn history server address

If you are running history server and ...READ MORE

Mar 27, 2019 in Apache Spark by Neha
 • 2,766 views
0 votes
1 answer

How can we use spark shell for scala without cluster?

You can run the Spark shell for ...READ MORE

Apr 28, 2019 in Apache Spark by Giri
 • 1,384 views
0 votes
1 answer

Set archives to be extracted in executor directory

I don't think you can copy and ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
 • 2,634 views
0 votes
1 answer

Apache Spark, usage of yield.

Yield is used in sequence comprehensions. It is ...READ MORE

Feb 22, 2019 in Apache Spark by Saruj
 • 4,044 views
0 votes
1 answer

How to make Spark wait for more time for acknowledgement?

Use the following command to increase the ...READ MORE

Mar 11, 2019 in Apache Spark by Raj
 • 3,204 views
0 votes
1 answer

How to authenticate Spark internal connections using a secret key?

You need to set the secret key ...READ MORE

Mar 13, 2019 in Apache Spark by Venu
 • 3,115 views
0 votes
1 answer

How to disable executor from fetching file from cache?

When a Spark application is running, the ...READ MORE

Mar 10, 2019 in Apache Spark by Siri
 • 3,176 views
0 votes
1 answer

How to set keystore path?

You have to set the path to ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
 • 2,912 views
0 votes
1 answer

How to change scheduling mode in Spark?

You can change the scheduling mode as ...READ MORE

Mar 12, 2019 in Apache Spark by Raj
 • 3,057 views
0 votes
1 answer

How to use Spark jars for Yarn distribution?

First, store upload this archive to hdfs and ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
 • 2,247 views
0 votes
0 answers

what are the memory issues in spark ?

Mar 18, 2019 in Apache Spark by satish kumar
 • 180 points • 2,725 views
0 votes
1 answer

How to set executors for static allocation in Spark Yarn?

Open Spark shell and run the following ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
 • 2,135 views
0 votes
1 answer

How to access private key password with Spark?

Spark allows you to retrieve the key ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
 • 2,654 views
0 votes
1 answer

How to increase HDFS replication level in Spark?

Hi @Raunak. You can change the replication ...READ MORE

Mar 27, 2019 in Apache Spark by Yash
 • 2,142 views
0 votes
1 answer

How to connect to Zookeeper after setting Spark recovery mode?

You have set Zookeeper as the recovery ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
 • 2,197 views
0 votes
1 answer

Spark workers are not accepting any job (Kubernetes-Docker-Spark)

When kubernetes picks 10.*.*.*/16 network as it's ...READ MORE

Mar 1, 2019 in Apache Spark by Hamza
 • 200 points • 3,234 views
0 votes
1 answer

Increasing memory to use for Yarn application master?

You can increase the memory dynamically by ...READ MORE

Mar 26, 2019 in Apache Spark by Tina
 • 2,046 views
0 votes
1 answer

Increase cores for yarn in Spark application

By default, only one core is used for ...READ MORE

Mar 26, 2019 in Apache Spark by Bhuvan
 • 1,991 views
0 votes
1 answer

How to set max executors for dynamic allocation?

You can set it by assigning the ...READ MORE

Mar 13, 2019 in Apache Spark by Venu
 • 2,540 views
0 votes
1 answer

How to save and retrieve the Spark RDD from HDFS?

You can save the RDD using saveAsObjectFile and saveAsTextFile method. ...READ MORE

May 29, 2018 in Apache Spark by Shubham
 • 13,490 points • 14,981 views
0 votes
1 answer

How to enable dynamic resource allocation in Spark?

To dynamically enable dynamic resource allocation, you ...READ MORE

Mar 12, 2019 in Apache Spark by veer
 • 2,534 views
0 votes
1 answer

How to use ftp scheme using Yarn in Spark application?

In case Yarn does not support schemes ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
 • 1,791 views
0 votes
1 answer

How to disable automatic remove of application of failures?

Yes, you have read it right. The ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
 • 1,832 views
0 votes
1 answer

Thread to use Yarn application master is limited

This is because the maximum number of ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
 • 1,710 views
0 votes
1 answer

How to retain Spark jar and app jar after staging?

By default, Spark jar, app jar, and ...READ MORE

Mar 27, 2019 in Apache Spark by Ginni
 • 1,674 views
0 votes
1 answer

How to get SQL configuration in Spark using Python?

You can get the configuration details through ...READ MORE

Mar 18, 2019 in Apache Spark by John
 • 2,013 views
0 votes
1 answer

Increase number of cores in Spark

Now that the job is already running, ...READ MORE

Feb 23, 2019 in Apache Spark by Reshma
 • 3,017 views
0 votes
1 answer

Client connection getting rejected due to SASL authentication

Seems like you have set the configuration ...READ MORE

Mar 14, 2019 in Apache Spark by Raj
 • 2,131 views
0 votes
1 answer

How to give user only view access for Spark application?

You can give users only view permission ...READ MORE

Mar 14, 2019 in Apache Spark by Raj
 • 2,125 views
0 votes
1 answer

How to change minimum rate per kafka partition?

Open spark shell using this command: $ spark-shell Then ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
 • 1,904 views
0 votes
1 answer

Setting directory for Spark recovery

You can set the directory to store ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
 • 1,599 views
0 votes
1 answer

Spark SQL in databricks

In sparkSql, we can use CASE when ...READ MORE

Feb 24, 2019 in Apache Spark by Rishi
 • 2,871 views
0 votes
1 answer

How to compress serialized RDD partition?

Yes, you can do this by enabling ...READ MORE

Mar 7, 2019 in Apache Spark by Pavitra
 • 2,344 views
0 votes
1 answer

Array of RDD

You can create an array of RDDs ...READ MORE

Mar 8, 2019 in Apache Spark by Raj
 • 2,309 views
0 votes
1 answer

Remove idle executor automatically

Yes, it is possible and is already ...READ MORE

Mar 12, 2019 in Apache Spark by Veer
 • 2,125 views
0 votes
1 answer

Spark - repartition() vs coalesce()

It avoids a full shuffle. If it's ...READ MORE

Oct 11, 2018 in Apache Spark by nitinrawat895
 • 11,380 points • 8,676 views
0 votes
1 answer

How to change worker cleanup interval?

The default interval time is 1800 seconds ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
 • 1,520 views
0 votes
1 answer

How to increase Garbage Collection speed?

The time interval between Garbage Collection is ...READ MORE

Mar 8, 2019 in Apache Spark by Pavitra
 • 2,249 views
0 votes
1 answer

How to enable SASL authentication after Spark authentication?

You can do this by setting the ...READ MORE

Mar 13, 2019 in Apache Spark by Venu
 • 1,993 views
0 votes
1 answer

Close file after writing write-ahead on driver

You can change the property to close ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
 • 1,720 views
0 votes
1 answer

How to disable credentials for services in Spark?

Disabling this feature will compromise the security ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
 • 1,883 views
0 votes
1 answer

How to set Spark recovery to Zookeeper?

There is a property of Spark which ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
 • 1,450 views
0 votes
1 answer

How to cleanup application work directories faster?

By default, the cleanup time is set ...READ MORE

Mar 26, 2019 in Apache Spark by Jyoti
 • 1,387 views
0 votes
1 answer

How to change default Spark dashboard port?

You can change it dynamically while using ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
 • 2,244 views
0 votes
1 answer

Changing the blacklist time of executor

By default, the node or executor is ...READ MORE

Mar 12, 2019 in Apache Spark by Raj
 • 2,019 views
0 votes
1 answer

Parquet to ORC format in Spark

I appreciate that you want to try ...READ MORE

Feb 15, 2019 in Apache Spark by Anjali
 • 3,094 views
0 votes
1 answer

Set Spark executable for R scripts

To change the default executable, assign the ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
 • 1,655 views
0 votes
1 answer

When is speculation enabled in Spark?

Speculation is enabled when a fraction of ...READ MORE

Mar 12, 2019 in Apache Spark by veer
 • 1,956 views
0 votes
1 answer

Prevent immediate shutdown of Streaming context after JVM shutdown

In technical terms, you want to gracefully shut down the ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
 • 1,643 views
0 votes
1 answer

Is it better to have one large parquet file or lots of smaller parquet files?

Ideally, you would use snappy compression (default) ...READ MORE

May 23, 2018 in Apache Spark by nitinrawat895
 • 11,380 points • 14,603 views
Page:

Most popular tags

"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.