spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lalwani, Jayesh" <>
Subject Union of 2 streaming data frames
Date Fri, 07 Jul 2017 18:27:34 GMT
In structured streaming, Is there a way to Union 2 streaming data frames? Are there any plans
to support Union of 2 streaming dataframes soon? I can understand the inherent complexity
in joining 2 streaming data frames. But, Union is  just concatenating 2 microbatches, innit?

The problem that we are trying to solve is that we have a Kafka stream that is receiving events.
Each event is assosciated with an account ID. We have a data store that stores historical
 events for hundreds of millions of accounts. What we want to do is for the events coming
in the input stream, we want to add in all the historical events from the data store and give
it to a model.

Initially, the way we were planning to do this is
a) read from Kafka into a streaming dataframe. Call this inputDF.
b) In a mapWithPartition method, get all the unique accounts in the partition. Look up all
the historical events for those unique accounts and return them. Let’s call this historicalDF
c) Union inputDF with historicalDF. Call this allDF
d) Call mapWithPartition on allDF and give the records to the model

Of course, this doesn’t work because both inputDF and historicalDF are streaming data frames.

What we ended up doing is in step b) we output the input records with the historical records,
which works but seems like a hacky way of doing things. The operation that does lookup does
union too. This works for now because the data from the data store doesn’t require any transformation
or aggregation. But, if it did, we would like to do that using Spark SQL, whereas this solution
forces us to doing any transformation of historical data in Scala

Is there a Sparky way of doing this?

The information contained in this e-mail is confidential and/or proprietary to Capital One
and/or its affiliates and may only be used solely in performance of work or services for Capital
One. The information transmitted herewith is intended only for use by the individual or entity
to which it is addressed. If the reader of this message is not the intended recipient, you
are hereby notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is strictly prohibited.
If you have received this communication in error, please contact the sender and delete the
material from your computer.
View raw message