spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lsn24 <lekshmi.s...@gmail.com>
Subject Filter one dataset based on values from another
Date Tue, 01 May 2018 04:02:40 GMT
Hi,
  I have one  dataset with parameters and another with data that needs to
apply/ filter based on the first dataset (Parameter dataset).

*Scenario is as follows:*

    For each row in parameter dataset, I need to apply the parameter row to
the second dataset.I will end up having multiple dataset. for each second
dataset i need to run  a bunch of calculation.

How can I achieve this in spark?

*Pseudo code for better understanding:*

Dataset<Parameter> paramsDataset = sparkSession.sql("select * from
paramsview");

Dataset<myData> myDataset = sparkSession.sql("select * from tempview");


Question: For each row in paramsDataset, I need to filter myDataset and run
some calculations on it. Is it possible to do that ? if not whats the best
way to solve it?

Thanks




--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message