I want to run huge number of queries on Dataframe in Spark. I have a big data of text documents, I loded all documents into SparkDataFrame and create a temp table.
I have more than 50,000 terms, I want to get the document frequency for each by using the "table1".
I use the follwing:
DataFrame df=sqlContext.sql("select count(ID) from table1 where text like '%"+term+"%'");
but this scenario needs much time to finish because I have t run it from Spark Driver for each term.
Does anyone has idea how I can run all queries in distributed way?
Thank you && Best Regards,