spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniela S" <daniela_4...@gmx.at>
Subject Aw: Re: Spark Streaming prediction
Date Mon, 02 Jan 2017 20:26:36 GMT
<html><head></head><body><div style="font-family: Verdana;font-size:
12.0px;"><div>
<div>Hi</div>

<div>&nbsp;</div>

<div>Thank you very much for your answer!</div>

<div>&nbsp;</div>

<div>My problem is that I know the values for the next 2-3 hours in advance but i do
not know the values from hour 2 or 3 to hour 24. How is it possible to combine the known values
with the predicted values as both are values in the future? And how can i ensure that there
are always 1440 values?</div>

<div>And I do not know how to map the values for 1440 minutes to a specific time on
the dashboard (e.g. how does the dashboard know that the value for minute 300 maps to time
15:05?</div>

<div>&nbsp;</div>

<div>Thank you in advance.</div>

<div>&nbsp;</div>

<div>Best regards,</div>

<div>Daniela</div>

<div>&nbsp;</div>

<div>&nbsp;</div>

<div>&nbsp;
<div name="quote" style="margin:10px 5px 5px 10px; padding: 10px 0 10px 10px; border-left:2px
solid #C3D9E5; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div style="margin:0 0 10px 0;"><b>Gesendet:</b>&nbsp;Montag, 02. Januar
2017 um 21:07 Uhr<br/>
<b>Von:</b>&nbsp;&quot;Marco Mistroni&quot; &lt;mmistroni@gmail.com&gt;<br/>
<b>An:</b>&nbsp;&quot;Daniela S&quot; &lt;daniela_4444@gmx.at&gt;<br/>
<b>Cc:</b>&nbsp;User &lt;user@spark.apache.org&gt;<br/>
<b>Betreff:</b>&nbsp;Re: Spark Streaming prediction</div>

<div name="quoted-content">
<div>
<div>Hi
<div>&nbsp;you&nbsp; might want to have a look at the Regression ML&nbsp;
algorithm and integrate it in your SparkStreaming application, i m sure someone on the list
has&nbsp; a similar use case</div>

<div>shortly, you&#39;d want to process all your events and feed it through a ML&nbsp;
model which,based on your inputs will predict output</div>

<div>You say that your events predict minutes values for next 2-3 hrs... gather data
for a day and train ur model based on that. Then save it somewhere and have your streaming
app load the module and have the module do the predictions based on incoming events from your
streaming app.<br/>
Save the results somewhere and have your dashboard poll periodically your data store to read
the predictions</div>

<div>I have seen ppl on the list doing ML over a Spark streaming app, i m sure someone
can reply back....</div>

<div>Hpefully i gave u a starting point....<br/>
&nbsp;</div>

<div>hth</div>

<div>&nbsp;marco</div>
</div>

<div class="gmail_extra">&nbsp;
<div class="gmail_quote">On 2 Jan 2017 4:03 pm, &quot;Daniela S&quot; &lt;<a
href="mailto:daniela_4444@gmx.at" onclick="parent.window.location.href=&#39;daniela_4444@gmx.at&#39;;
return false;" target="_blank">daniela_4444@gmx.at</a>&gt; wrote:

<blockquote class="gmail_quote" style="margin: 0 0 0 0.8ex;border-left: 1.0px rgb(204,204,204)
solid;padding-left: 1.0ex;">
<div>
<div style="font-family: Verdana;font-size: 12.0px;">
<div>Hi</div>

<div>&nbsp;</div>

<div>I am trying to solve the following problem with Spark Streaming.</div>

<div>I receive timestamped events&nbsp;from Kafka. Each event refers to a device
and contains values for every minute of the next 2 to 3 hours. What I would like to do is
to predict the minute values for the next 24 hours. So I would like to use the known values
and to predict the other values to achieve the 24 hours prediction. My thought was to use
arrays with a length of 1440 (1440 minutes = 24 hours). One for the known values and one for
the predicted values for each device. Then I would like to show the next 24 hours on a dashboard.
The dashboard should be updated automatically in realtime.&nbsp;</div>

<div>&nbsp;</div>

<div>My questions:</div>

<div>is this a possible solution?</div>

<div>how is it possible to combine known future values and predicted values?</div>

<div>how should I treat the timestamp as the length of 1440 does not correspond to a
timestamp?</div>

<div>how is it possible to update the dashboard automatically in realtime?</div>

<div>&nbsp;</div>

<div>Thank you in advance!</div>

<div>&nbsp;</div>

<div>Best regards,</div>

<div>Daniela</div>
</div>
</div>
--------------------------------------------------------------------- To unsubscribe e-mail:
<a href="mailto:user-unsubscribe@spark.apache.org" onclick="parent.window.location.href=&#39;user-unsubscribe@spark.apache.org&#39;;
return false;" target="_blank">user-unsubscribe@spark.apache.org</a></blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div></div></body></html>

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message