Execute Spark sql query within withColumn clause is Spark Scala

0 votes

I have a dataframe which has one of the column called "Query" having the select statement present. Want to execute this query and create a new column having actual results from the query within the same dataframe.

+--------------+-----------+-----+--------------------------------+
|DIFFCOLUMNNAME|DATATYPE   |ISSUE|QUERY                           |
+--------------+-----------+-----+--------------------------------+
|Firstname     |StringType |YES  |Select Firstname from TempView  limit 1|
|LastName      |StringType |NO   |Select LastName from TempView  limit 1  |
|Designation   |StringType |YES  |Select Designation from TempView limit 1|
|Salary        |IntegerType|YES  |Select Salary from TempView    limit 1 |
+--------------+-----------+-----+--------------------------------+

When I try below

DF.withColumn("QueryResult", SparkConfiguration.sparkSession.sql(col("QUERY"))) 
Getting error as Type mismatch, Required String found column. 
I am thinking I need to use UDF here. But not sure how to write and use. Please helpspark

Sep 14, 2021 in Apache Spark by Pinksrider

edited 4 days ago 18 views

No answer to this question. Be the first to respond.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP