Trending questions in Apache Spark

0 votes
1 answer

Explain vararg arguments.

Hey, With varargs, we can pass a variable ...READ MORE

Jul 24, 2019 in Apache Spark by Gitika
 • 65,730 points • 820 views
0 votes
1 answer

Is fetching data from apache flume webcrawling?

Web crawling is a program or automated ...READ MORE

Jul 11, 2019 in Apache Spark by Esha
 • 1,331 views
0 votes
0 answers

What is immutabiliity in Spark?

Can anyone explain what is immutability in ...READ MORE

Jul 23, 2019 in Apache Spark by Risha
 • 837 views
0 votes
1 answer

How to calculate the result of formula with Scala?

Hi, You can use a simple mathematical calculation ...READ MORE

Jul 1, 2019 in Apache Spark by Gitika
 • 65,730 points • 1,712 views
0 votes
1 answer

Scala join comma delimited file as tables

Dataframe creation commands:​ Now we will register them ...READ MORE

Jul 9, 2019 in Apache Spark by Suraj
 • 1,364 views
0 votes
1 answer

Is it mandatory to start Hadoop to run spark application?

Hi, No, not mandatory, but there is no ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,730 points • 1,483 views
0 votes
1 answer

What is SparkCore functionalities?

Hey, Spark Core is a base engine of ...READ MORE

Jul 4, 2019 in Apache Spark by Gitika
 • 65,730 points • 1,524 views
0 votes
0 answers

When we create an RDD, does it bring the data and load it into the memory?

Can anyone suggest when we create an ...READ MORE

Jul 3, 2019 in Apache Spark by monalisa
recategorized Jul 4, 2019 by Gitika • 1,560 views
0 votes
1 answer

When we create an RDD, does it bring the data and load it into the memory?

Hi, No. An RDD is made up of ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,730 points • 1,410 views
0 votes
1 answer

Error while reading multiline Json

peopleDF: org.apache.spark.sql.DataFrame = [_corrupt_record: string] The above that ...READ MORE

May 23, 2019 in Apache Spark by Conny
 • 3,216 views
0 votes
1 answer

How can you use "for" statement in scala to print list from collection?

Hi, You can use for loop in scala using ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,730 points • 1,300 views
0 votes
1 answer

How can we optimize and minimize the memory when work with scala use case?

Hi, There is a term in Scala that is ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,730 points • 1,271 views
0 votes
0 answers

How to create RDD as string file?

Can anyone suggest how to create RDD ...READ MORE

Jul 5, 2019 in Apache Spark by anand
 • 1,339 views
0 votes
1 answer

How will you explain yield keyword in Scala?

Hi, Yield keyword can be used either before ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,730 points • 1,224 views
0 votes
1 answer

How to assign block expression in scala?

Hi, You can follow this example to know ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,730 points • 1,210 views
0 votes
1 answer

what are the spark real time issues ?

Some of the issues I have faced ...READ MORE

Mar 18, 2019 in Apache Spark by Sharman
 • 5,904 views
0 votes
1 answer

Spark: Saving file csv

 If you need a single output file ...READ MORE

May 22, 2019 in Apache Spark by Rishi
 • 3,056 views
0 votes
1 answer

How to implement two level loop in scala?

Hi, You can use two level loops using the ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,730 points • 1,093 views
0 votes
1 answer

What is persist() in Spark?

Hi, Spark’s RDDs are by default recomputed each ...READ MORE

Jul 2, 2019 in Apache Spark by Gitika
 • 65,730 points • 1,234 views
0 votes
1 answer

What is meant by Transformation? Give some examples.

Hi, The transformations are the functions that are ...READ MORE

Jul 3, 2019 in Apache Spark by Gitika
 • 65,730 points • 1,180 views
0 votes
1 answer

Difference between RDD as val and var

Variable declaration can be done in two ...READ MORE

May 23, 2019 in Apache Spark by Arun
 • 2,914 views
+1 vote
1 answer

map vs mapValues in Spark

There is a difference between the two: mapValues ...READ MORE

Jun 29, 2018 in Apache Spark by nitinrawat895
 • 11,380 points • 16,993 views
0 votes
1 answer

What does reduce action do in Spark?

Hey, Reduce action converts an RDD to a ...READ MORE

Jul 4, 2019 in Apache Spark by Gitika
 • 65,730 points • 1,056 views
0 votes
1 answer

How to print loop with condition in scala?

Hi, Yes, in scala there is a guard condition where ...READ MORE

Jul 6, 2019 in Apache Spark by Gitika
 • 65,730 points • 949 views
0 votes
1 answer

By which components spark ecosystem libraries are composed of?

Hi, Spark ecosystem libraries are composed of various ...READ MORE

Jul 1, 2019 in Apache Spark by Gitika
 • 65,730 points • 1,014 views
0 votes
1 answer

How to store files in executor's working directory?

You have to specify a comma-separated list ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
 • 4,799 views
0 votes
1 answer

Spark CLI issue

For spark.read.textFile we need spark-2.x. Please try ...READ MORE

Jun 19, 2019 in Apache Spark by Maahi
 • 1,149 views
0 votes
1 answer

Doubt in display(id, name, salary) before display function

The statement display(id, name, salary) is written before the display function ...READ MORE

Jun 19, 2019 in Apache Spark by Ritu
 • 1,118 views
0 votes
1 answer

How to set extra JVM options for Spark application?

You cans set extra JVM options that ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
 • 4,393 views
0 votes
1 answer

How to change the location of Spark event logs?

You can change the location where you ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
 • 5,162 views
0 votes
1 answer

How to set cpu cores for spark task?

By default, each task is allocated with ...READ MORE

Mar 12, 2019 in Apache Spark by Veer
 • 4,848 views
0 votes
1 answer

Spark2-submit does not generate output file.

To generate the output file, you can ...READ MORE

Feb 24, 2019 in Apache Spark by Esha
 • 5,331 views
0 votes
1 answer

Starting Spark in Windows

Run below commands spark-class org.apache.spark.deploy.master.Master spark-class org.apache.spark.deploy.worker.Worker spark://192.168.254.1:7077 NOTE: The ...READ MORE

May 22, 2019 in Apache Spark by Reshma
 • 1,449 views
0 votes
1 answer

How to get Spark SQL configuration?

First create a Spark session like this: val ...READ MORE

Mar 18, 2019 in Apache Spark by John
 • 4,180 views
0 votes
3 answers

Can anyone explain fold() operation in Spark?

Fold in spark Fold is a very powerful ...READ MORE

Aug 23, 2018 in Apache Spark by samarth295
 • 2,220 points • 13,546 views
0 votes
1 answer

Starting Spark Scala console

To get command prompt for Scala open ...READ MORE

May 24, 2019 in Apache Spark by Cassy
 • 1,063 views
0 votes
1 answer

Increase Yarn wait time for Sparkcontext

The default time that the Yarn application waits ...READ MORE

Mar 27, 2019 in Apache Spark by Rohit
 • 3,567 views
0 votes
1 answer

What is Executor Memory in a Spark application?

Every spark application has same fixed heap ...READ MORE

Jan 5, 2019 in Apache Spark by Frankie
 • 9,830 points • 7,054 views
0 votes
1 answer

SparkContext.addFile() not able to update file.

Spark by default won't let you overwrite ...READ MORE

Mar 10, 2019 in Apache Spark by Siri
 • 4,116 views
0 votes
1 answer

How to change commiter algorithm version in Spark?

To change to version 2, run the ...READ MORE

Mar 10, 2019 in Apache Spark by Siri
 • 4,102 views
0 votes
1 answer

How to enable worker cleanup in Spark?

To enable cleanup, open the spark shell ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
 • 3,385 views
0 votes
1 answer

Disable Web UI for Spark Application

You can disable it like this: val sc ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
 • 4,172 views
0 votes
1 answer

What is Spark Core?

It is not like a CPU to ...READ MORE

Mar 8, 2019 in Apache Spark by Raj
 • 3,972 views
0 votes
1 answer

How to enable SSL for Spark application?

You can do it dynamically like this: val ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
 • 3,205 views
0 votes
1 answer

Spark Yarn: Changing maximum number of time to submit application

By default, the maximum number of times ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
 • 2,582 views
0 votes
1 answer

How can we use spark shell for scala without cluster?

You can run the Spark shell for ...READ MORE

Apr 28, 2019 in Apache Spark by Giri
 • 1,196 views
0 votes
1 answer

what are the job optimization Technics in spark and scala ?

There are different methods to achieve optimization ...READ MORE

Mar 18, 2019 in Apache Spark by Veer
 • 2,923 views
0 votes
1 answer

Need help setting Spark yarn history server address

If you are running history server and ...READ MORE

Mar 27, 2019 in Apache Spark by Neha
 • 2,531 views
0 votes
2 answers

In a Spark DataFrame how can I flatten the struct?

// Collect data from input avro file ...READ MORE

Jul 4, 2019 in Apache Spark by Dhara dhruve
 • 6,944 views
0 votes
1 answer

Set archives to be extracted in executor directory

I don't think you can copy and ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
 • 2,433 views
Page:

Most popular tags

"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.