spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lsn24 <>
Subject Filter one dataset based on values from another
Date Tue, 01 May 2018 04:02:40 GMT
  I have one  dataset with parameters and another with data that needs to
apply/ filter based on the first dataset (Parameter dataset).

*Scenario is as follows:*

    For each row in parameter dataset, I need to apply the parameter row to
the second dataset.I will end up having multiple dataset. for each second
dataset i need to run  a bunch of calculation.

How can I achieve this in spark?

*Pseudo code for better understanding:*

Dataset<Parameter> paramsDataset = sparkSession.sql("select * from

Dataset<myData> myDataset = sparkSession.sql("select * from tempview");

Question: For each row in paramsDataset, I need to filter myDataset and run
some calculations on it. Is it possible to do that ? if not whats the best
way to solve it?


Sent from:

To unsubscribe e-mail:

View raw message