Trending questions in Apache Spark

0 votes
3 answers

Lineage Graph in Spark

Whenever a series of transformations are performed ...READ MORE

Aug 28, 2018 in Apache Spark by shams
 • 3,670 points • 11,736 views
0 votes
1 answer

How to disable automatic remove of application of failures?

Yes, you have read it right. The ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
 • 1,191 views
0 votes
1 answer

How to enable dynamic resource allocation in Spark?

To dynamically enable dynamic resource allocation, you ...READ MORE

Mar 12, 2019 in Apache Spark by veer
 • 1,735 views
0 votes
1 answer

Spark workers are not accepting any job (Kubernetes-Docker-Spark)

When kubernetes picks 10.*.*.*/16 network as it's ...READ MORE

Mar 1, 2019 in Apache Spark by Hamza
 • 200 points • 2,175 views
0 votes
1 answer

How to retain Spark jar and app jar after staging?

By default, Spark jar, app jar, and ...READ MORE

Mar 27, 2019 in Apache Spark by Ginni
 • 1,006 views
0 votes
1 answer

How to give user only view access for Spark application?

You can give users only view permission ...READ MORE

Mar 14, 2019 in Apache Spark by Raj
 • 1,505 views
0 votes
1 answer

Setting directory for Spark recovery

You can set the directory to store ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
 • 1,033 views
0 votes
1 answer

Client connection getting rejected due to SASL authentication

Seems like you have set the configuration ...READ MORE

Mar 14, 2019 in Apache Spark by Raj
 • 1,493 views
0 votes
1 answer

Spark SQL in databricks

In sparkSql, we can use CASE when ...READ MORE

Feb 24, 2019 in Apache Spark by Rishi
 • 2,272 views
0 votes
1 answer

Array of RDD

You can create an array of RDDs ...READ MORE

Mar 8, 2019 in Apache Spark by Raj
 • 1,720 views
0 votes
1 answer

How to get SQL configuration in Spark using Python?

You can get the configuration details through ...READ MORE

Mar 18, 2019 in Apache Spark by John
 • 1,256 views
0 votes
1 answer

Remove idle executor automatically

Yes, it is possible and is already ...READ MORE

Mar 12, 2019 in Apache Spark by Veer
 • 1,448 views
0 votes
1 answer

How to compress serialized RDD partition?

Yes, you can do this by enabling ...READ MORE

Mar 7, 2019 in Apache Spark by Pavitra
 • 1,657 views
0 votes
1 answer

Increase number of cores in Spark

Now that the job is already running, ...READ MORE

Feb 23, 2019 in Apache Spark by Reshma
 • 2,214 views
0 votes
1 answer

How to increase Garbage Collection speed?

The time interval between Garbage Collection is ...READ MORE

Mar 8, 2019 in Apache Spark by Pavitra
 • 1,637 views
0 votes
1 answer

Spark - repartition() vs coalesce()

It avoids a full shuffle. If it's ...READ MORE

Oct 11, 2018 in Apache Spark by nitinrawat895
 • 11,380 points • 7,991 views
0 votes
1 answer

How to change worker cleanup interval?

The default interval time is 1800 seconds ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
 • 822 views
0 votes
1 answer

Prevent immediate shutdown of Streaming context after JVM shutdown

In technical terms, you want to gracefully shut down the ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
 • 1,069 views
0 votes
1 answer

Close file after writing write-ahead on driver

You can change the property to close ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
 • 1,028 views
0 votes
1 answer

How to set Spark recovery to Zookeeper?

There is a property of Spark which ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
 • 766 views
0 votes
1 answer

Changing the blacklist time of executor

By default, the node or executor is ...READ MORE

Mar 12, 2019 in Apache Spark by Raj
 • 1,359 views
0 votes
1 answer

Getting "buffer limit exceeded" exception inside Kryo.

Seems like the object being sent for ...READ MORE

Mar 7, 2019 in Apache Spark by Pavitra
 • 1,532 views
0 votes
1 answer

Set Spark executable for R scripts

To change the default executable, assign the ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
 • 1,009 views
0 votes
1 answer

How to disable credentials for services in Spark?

Disabling this feature will compromise the security ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
 • 1,174 views
0 votes
1 answer

How to cleanup application work directories faster?

By default, the cleanup time is set ...READ MORE

Mar 26, 2019 in Apache Spark by Jyoti
 • 694 views
0 votes
1 answer

Set Library to launch Yarn master

You can make use of Special Library path to ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
 • 615 views
0 votes
1 answer

Parquet to ORC format in Spark

I appreciate that you want to try ...READ MORE

Feb 15, 2019 in Apache Spark by Anjali
 • 2,385 views
0 votes
1 answer

Need to disable unpersist in Spark

You can dynamically change this function by ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
 • 959 views
0 votes
1 answer

How to set client authentication for SSL?

By default, this feature is disabled. To ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
 • 1,087 views
0 votes
1 answer

When is speculation enabled in Spark?

Speculation is enabled when a fraction of ...READ MORE

Mar 12, 2019 in Apache Spark by veer
 • 1,219 views
0 votes
1 answer

How to enable SASL authentication after Spark authentication?

You can do this by setting the ...READ MORE

Mar 13, 2019 in Apache Spark by Venu
 • 1,172 views
0 votes
1 answer

Can I set different protocol for SSL in Spark?

There is no protocol set by default. ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
 • 1,077 views
0 votes
1 answer

Get Spark SQL configuration in Java

You will need to use Spark session ...READ MORE

Mar 18, 2019 in Apache Spark by John
 • 937 views
0 votes
1 answer

How to change minimum rate per kafka partition?

Open spark shell using this command: $ spark-shell Then ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
 • 893 views
0 votes
1 answer

How to automatically kill executors on blacklisting?

You can set the property to directly ...READ MORE

Mar 12, 2019 in Apache Spark by Veer
 • 1,187 views
0 votes
1 answer

How to enable write ahead logs for receivers?

To enable write-ahead logs, run the following ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
 • 875 views
0 votes
1 answer

How can I run a blacklisted task?

When a task results in too many ...READ MORE

Mar 12, 2019 in Apache Spark by Raj
 • 1,177 views
0 votes
1 answer

How enable XSS protection for Spark application?

Spark has a built-in prevention system against XSS. ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
 • 1,007 views
0 votes
1 answer

How to save and retrieve the Spark RDD from HDFS?

You can save the RDD using saveAsObjectFile and saveAsTextFile method. ...READ MORE

May 29, 2018 in Apache Spark by Shubham
 • 13,490 points • 13,545 views
0 votes
1 answer

Limit displaying completed applications

By default, the number of completed applications ...READ MORE

Mar 25, 2019 in Apache Spark by hari
 • 571 views
0 votes
1 answer

How to set stage abort in Spark?

You can do it as follows. Use ...READ MORE

Mar 12, 2019 in Apache Spark by veer
 • 1,109 views
+1 vote
1 answer

Facing out-of-memory errors in Spark driver

I am guessing that the configuration set ...READ MORE

Feb 23, 2019 in Apache Spark by Rishab
 • 1,827 views
0 votes
1 answer

How to prevent executor from self-destructing?

I think there is a timeout set ...READ MORE

Mar 12, 2019 in Apache Spark by Veer
 • 1,094 views
0 votes
1 answer

How to change default Spark dashboard port?

You can change it dynamically while using ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
 • 1,324 views
0 votes
1 answer

Is it better to have one large parquet file or lots of smaller parquet files?

Ideally, you would use snappy compression (default) ...READ MORE

May 23, 2018 in Apache Spark by nitinrawat895
 • 11,380 points • 13,730 views
0 votes
1 answer

How to spread applications to less nodes?

You can limit the spread out by ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
 • 499 views
0 votes
1 answer

How to check if user has permission in Web UI?

You can implement this as follows: First, add ...READ MORE

Mar 14, 2019 in Apache Spark by Raj
 • 970 views
0 votes
1 answer

Installing Spark on Ubuntu

Hey. Follow these steps to install Spark ...READ MORE

Feb 20, 2019 in Apache Spark by Omkar
 • 69,220 points • 1,918 views
0 votes
1 answer

Network encryption for Spark

You can enable encryption for the Spark ...READ MORE

Mar 13, 2019 in Apache Spark by Venu
 • 1,000 views
0 votes
1 answer

Change number of completed drivers displayed

You can change the number of completed ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
 • 469 views
Page:

Most popular tags

"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.