Trending questions in Apache Spark

0 votes
1 answer

Retaining the batch by status APIs before Garbage Collection

By default, 1000 batches are retained by ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
 • 724 views
0 votes
1 answer

Using R to display configuration of Spark SQL

Try the below-mentioned code. sparkR.session() properties <- sql("SET -v") showDF(properties, ...READ MORE

Mar 18, 2019 in Apache Spark by John
 • 765 views
0 votes
1 answer

Sliding function in spark

The sliding function is used when you ...READ MORE

Jan 29, 2019 in Apache Spark by Omkar
 • 69,220 points • 2,840 views
0 votes
1 answer

How to change block manager port for driver only?

Yes, it is possible to change the ...READ MORE

Mar 10, 2019 in Apache Spark by Siri
 • 1,120 views
0 votes
1 answer

How to make Spark skip node locality?

If you set the node wait time ...READ MORE

Mar 11, 2019 in Apache Spark by Raj
 • 1,036 views
0 votes
1 answer

How to increase Spark memory for execution?

Probably the spill is because you have ...READ MORE

Mar 7, 2019 in Apache Spark by Pavitra
edited Mar 8, 2019 • 1,179 views
0 votes
1 answer

Increasing retry before blacklisting executor

You can do it like this: val sc ...READ MORE

Mar 12, 2019 in Apache Spark by Raj
 • 1,002 views
0 votes
1 answer

Not able to clone Hadoop configuration.

Run the following command in Spark shell ...READ MORE

Mar 10, 2019 in Apache Spark by Siri
 • 1,073 views
0 votes
1 answer

Set maximum reading rate from Kafka partition

You can do it by using the ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
 • 651 views
0 votes
1 answer

Change number of threads use by R back end

Refer to the below commands to know ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
 • 627 views
0 votes
1 answer

Spark context (sc) not found

Maybe the hadoop service didn't start properly. Try ...READ MORE

Feb 14, 2019 in Apache Spark by John
 • 2,076 views
0 votes
1 answer

Change heart beat time to R process

The heartbeat interval is assigned to the ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
 • 609 views
0 votes
1 answer

Password for keystore in Spark

You can dynamically set a password to ...READ MORE

Mar 15, 2019 in Apache Spark by karan
 • 784 views
0 votes
1 answer

How to set maximum receiving rate for backpressure mechanism?

You can set the maximum receiving rate ...READ MORE

Mar 18, 2019 in Apache Spark by John
 • 632 views
0 votes
1 answer

How to enable SASL encryption?

You can use dynamic configuration setting to ...READ MORE

Mar 14, 2019 in Apache Spark by Raj
 • 806 views
0 votes
1 answer

How many executors run when dynamic allocation is enabled?

The number of executors running by default ...READ MORE

Mar 13, 2019 in Apache Spark by Venu
 • 848 views
0 votes
1 answer

Spark logs not overwriting

Spark does not allow you to overwrite ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
 • 1,108 views
0 votes
1 answer

Configuring location of credential provider

To configure the location of the credential ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
 • 688 views
0 votes
1 answer

How to change encryption key value for local I/O?

There another property where you can set ...READ MORE

Mar 14, 2019 in Apache Spark by Raj
 • 723 views
0 votes
1 answer

Setting port for SSL for Spark application

You can set the port in the ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
 • 679 views
0 votes
1 answer

How to give modify access to group?

You can do this using the following ...READ MORE

Mar 15, 2019 in Apache Spark by Vikas
 • 672 views
0 votes
1 answer

How to monitor interrupted Spark tasks?

To enable monitoring interrupted tasks, run the following ...READ MORE

Mar 12, 2019 in Apache Spark by Veer
 • 785 views
0 votes
1 answer

How to limit the cores being used by a cluster?

You can set the maximum number of ...READ MORE

Mar 11, 2019 in Apache Spark by Raj
 • 824 views
0 votes
1 answer

How to restrict a group to only view in Spark?

You can do it dynamically be setting ...READ MORE

Mar 15, 2019 in Apache Spark by Raj
 • 631 views
0 votes
1 answer

Loading Spark properties dynamically

First, create an empty conf using this ...READ MORE

Feb 22, 2019 in Apache Spark by Mansoor
 • 1,526 views
0 votes
1 answer

Change encryption key length fro Spark application

You can do this by running the ...READ MORE

Mar 13, 2019 in Apache Spark by Venu
 • 706 views
0 votes
1 answer

Not able to preserve shuffle files in Spark

You lose the files because by default, ...READ MORE

Feb 24, 2019 in Apache Spark by Rana
 • 1,467 views
0 votes
1 answer

How to add modify access for Web UI user?

For a user to have modification access ...READ MORE

Mar 14, 2019 in Apache Spark by Raj
 • 642 views
0 votes
1 answer

How to disable broadcast checksum?

Run the following in the Spark shell: val ...READ MORE

Mar 9, 2019 in Apache Spark by Siri
 • 845 views
0 votes
1 answer

Log every block update in Spark

By default, Spark does not log all ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
 • 965 views
0 votes
1 answer

Error reading avro dataset in spark

For avro, you need to download and ...READ MORE

Feb 4, 2019 in Apache Spark by Omkar
 • 69,220 points • 2,253 views
0 votes
1 answer

Authenticate internal connections in Spark

To make Spark authenticate internal connections, you ...READ MORE

Mar 13, 2019 in Apache Spark by Venu
 • 653 views
0 votes
1 answer

Multidimensional Array in Scala

Multidimensional array is an array which store ...READ MORE

Feb 11, 2019 in Apache Spark by Omkar
 • 69,220 points • 1,927 views
0 votes
1 answer

Dynamic allocation: Set minimum number of executor

You cans et it dynamically like this:  val ...READ MORE

Mar 13, 2019 in Apache Spark by Venu
 • 631 views
0 votes
1 answer

How to relaunch tasks that are running slowly?

The technical term for what you want ...READ MORE

Mar 12, 2019 in Apache Spark by Veer
 • 650 views
0 votes
1 answer

Enable encryption for local Input and Output

You can enable local I/O encryption like ...READ MORE

Mar 14, 2019 in Apache Spark by Raj
 • 534 views
0 votes
1 answer

How to make driver update metrics quickly to executor?

There's a heartbeat signal sent to the ...READ MORE

Mar 10, 2019 in Apache Spark by Siri
 • 713 views
0 votes
1 answer

How to disable existing directory check?

To disable this, run the below commands: val ...READ MORE

Mar 10, 2019 in Apache Spark by Siri
 • 694 views
0 votes
1 answer

Delay requesting new executor in dynamic allocation

You can set the duration like this: val ...READ MORE

Mar 13, 2019 in Apache Spark by Venu
 • 511 views
0 votes
1 answer

How to set time for task speculation?

By default, the check for task speculation ...READ MORE

Mar 12, 2019 in Apache Spark by Veer
 • 550 views
0 votes
1 answer

Key Factor Algorithms used for encryption.

The default key factor algorithm used is PBKDF2WithHmacSHA1. You ...READ MORE

Mar 13, 2019 in Apache Spark by Venu
 • 492 views
0 votes
1 answer

How to increase the amount of data to be transferred to shuffle service at the same time?

The amount of data to be transferred ...READ MORE

Mar 1, 2019 in Apache Spark by Omkar
 • 69,220 points • 1,003 views
0 votes
1 answer

Increasing retry before blacklisting a node

You can do it dynamically using the ...READ MORE

Mar 12, 2019 in Apache Spark by Raj
 • 556 views
0 votes
1 answer

How to increase wait time to launch data-local task?

You can increase the locality wait time ...READ MORE

Mar 11, 2019 in Apache Spark by Raj
 • 554 views
0 votes
1 answer

What port the Spark dashboard run on?

Spark dashboard by default runs on port ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
 • 759 views
0 votes
1 answer

Spark shuffle service port number

The default port that shuffle service runs ...READ MORE

Mar 1, 2019 in Apache Spark by Omkar
 • 69,220 points • 959 views
0 votes
1 answer

Components of Spark

Spark core: The base engine that offers ...READ MORE

Mar 8, 2019 in Apache Spark by Raj
 • 646 views
0 votes
1 answer

How to delay live entity updates on Spark ?

You can do this by increasing the ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
 • 719 views
0 votes
1 answer

Prevent jobs to be killed from Web UI

You need to be careful with this. ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
 • 673 views
0 votes
1 answer

Spark event log location

Unless and until you have not changed ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
 • 669 views
Page:

Most popular tags

"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.