spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Laeeq Ahmed <laeeqsp...@yahoo.com>
Subject Spark Streaming question batch size
Date Tue, 01 Jul 2014 09:34:43 GMT
Hi,

The window size in a spark streaming is time based which means we have different number of
elements in each window. For example if you have two streams (might be more) which are related
to each other and you want to compare them in a specific time interval. I am not clear how
it will work. Although they start running simultaneously, they might have different number
of elements in each time interval.

The following is output for two streams which have same number of elements and ran simultaneously.
The left most value is the number of elements in each window. If we add the number of elements
them, they are same for both streams but we can't compare both streams as they are different
in window size and number of windows.

Can we somehow make windows based on real time values for both streams? or Can we make windows
based on number of elements?

(n, (mean, varience, SD))

Stream 1

(7462,(1.0535658165371238,4242.001306434091,65.13064798107025))
(44826,(0.2546925855084064,5042.890184382894,71.0133099100647))
(245466,(0.2857731601728941,5014.411691661449,70.81251084138628))
(154852,(0.21907814309792514,3483.800160602281,59.023725404300606))
(156345,(0.3075668844414613,7449.528181550462,86.31064929399189))
(156603,(0.27785151491351234,5917.809892281489,76.9273026452994))
(156047,(0.18130350363672296,4019.0232843737017,63.39576708561623))


Stream 2

(10493,(0.5554953964547791,1254.883548218503,35.42433553672536))
(180649,(0.21684831234050583,1095.9634245399352,33.1053383087975))
(179994,(0.22048869512317407,1443.0566458182718,37.98758541705792))
(179455,(0.20473330254938552,1623.9538730448216,40.29831104456888))
(269817,(0.16987953223480945,3270.663944782799,57.18971887308766))
(101193,(0.21469292497504766,1263.0879032808723,35.53994799209577))


Regards,
Laeeq
Mime
View raw message