spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christiaan Ras <>
Subject Machine Learning with window data
Date Fri, 03 Aug 2018 10:01:27 GMT

I have a use case where I like to analyze windows of sensordata.
Currently I have a working case where I use Structured Streaming to process real-time streams
of sensordata. Now I like to analyse windows of sensordata and use classification to predict
the class of a whole window.
For instance, the application receives batches of sensordata (where each record holds: timestamp,
value, key). With the use of Machine learning I like to analyse windows of these streams and
classify the window as ‘warm’ or ‘cold’. A single record is not sufficient for classification,
a window of records shapes a pattern to be used for classification.

But how should you define features for a window of sensordata?
Each value (sensor) as a separate feature in the vector (for a window of x seconds, the vector
contains x sensor values)? Or is there a way a feature can hold multiple values (like an array)?
Or use some kind of encoding to fit x sensor values as a single feature?


View raw message