samoa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicolas Kourtellis <nkourtel...@gmail.com>
Subject Re: SAMOA stopping before the end of the dataset
Date Mon, 12 Sep 2016 07:51:49 GMT
Hi Eduardo,

Thanks for the tests and questions,

I can't tell if this is a problem of Storm / Nimbus or Samoa.
Have you tried a smaller stream, i.e., with a smaller number of events, or
smaller number of attributes?

Perhaps the DSPE is shutting down due to memory error or other resource
issue?

Try with different sizes and let us know the status of the machine before
it fails.

Thanks,

Nicolas




On Mon, Sep 12, 2016 at 4:25 AM, Eduardo Costa <eduardocosi@gmail.com>
wrote:

> Dear,
> I'm executanto the Samoa with the storm in local mode (command line:
> bin/samoa storm target/SAMOA-Storm-0.4.0-incubating-SNAPSHOT.jar
> "PrequentialEvaluation -i -1 -f 1000 -d /tmp/cenario10-2classes-
> ordenado-trat_entdv-sem-label-puro.csv -l (classifiers.ensemble.
> AdaptiveBagging
> -s 1) -s (org.apache.samoa.streams.ArffFileStream -s HDFSFileStreamSource
> -f /cenario10-2classes-ordenado-trat_entdv-sem-label-puro.csv.arff)").
>
> I am using a dataset of 1,300.00 instances, but this samoa stopping at
> 480,000 when I use Adaptive Bagging. After a while the following message
> appears:
>
> ------------------------------------------------------------
> ---------------------------------------------------
>
> org.apache.samoa.learners.classifiers.trees.ModelAggregatorProcessor -
> Change detected, resetting the classifier
> 66025 [Thread-22-org.apache.samoa.learners.classifiers.trees.
> ModelAggregatorProcessor_1]
> INFO  org.apache.samoa.learners.classifiers.trees.ModelAggregatorProcessor
> - Change detected, resetting the classifier
> 66025 [Thread-22-org.apache.samoa.learners.classifiers.trees.
> ModelAggregatorProcessor_1]
> INFO  org.apache.samoa.learners.classifiers.trees.ModelAggregatorProcessor
> - Change detected, resetting the classifier
> 66026 [Thread-22-org.apache.samoa.learners.classifiers.trees.
> ModelAggregatorProcessor_1]
> INFO  org.apache.samoa.learners.classifiers.trees.ModelAggregatorProcessor
> - Change detected, resetting the classifier
> 66027 [Thread-22-org.apache.samoa.learners.classifiers.trees.
> ModelAggregatorProcessor_1]
> INFO  org.apache.samoa.learners.classifiers.trees.ModelAggregatorProcessor
> - Change detected, resetting the classifier
> 66027 [Thread-22-org.apache.samoa.learners.classifiers.trees.
> ModelAggregatorProcessor_1]
> INFO  org.apache.samoa.learners.classifiers.trees.ModelAggregatorProcessor
> - Change detected, resetting the classifier
> 224126 [main] INFO  backtype.storm.daemon.nimbus - Delaying event :remove
> for 30 secs for Prequential_20160911213510-1-1473640527
> 224132 [main] INFO  backtype.storm.daemon.nimbus - Updated
> Prequential_20160911213510-1-1473640527 with status {:type :killed,
> :kill-time-secs 30}
> 224132 [main] INFO  backtype.storm.daemon.nimbus - Shutting down master
> 224136 [ProcessThread(sid:0 cport:-1):] INFO
> org.apache.storm.zookeeper.server.PrepRequestProcessor
> - Processed session termination for sessionid: 0x1571bd1c9c50001
> 224137 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO
>  org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket
> connection
> for client /0:0:0:0:0:0:0:1:50353 which had sessionid 0x1571bd1c9c50001
> 224137 [main] INFO  org.apache.storm.zookeeper.ZooKeeper - Session:
> 0x1571bd1c9c50001 closed
> 224137 [main-EventThread] INFO  org.apache.storm.zookeeper.ClientCnxn -
> EventThread shut down
> 224142 [main] INFO  backtype.storm.daemon.nimbus - Shut down master
> 224155 [ProcessThread(sid:0 cport:-1):] INFO
> org.apache.storm.zookeeper.server.PrepRequestProcessor
> - Processed session termination for sessionid: 0x1571bd1c9c50003
> 224159 [main] INFO  org.apache.storm.zookeeper.ZooKeeper - Session:
> 0x1571bd1c9c50003 closed
> 224159 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO
>  org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket
> connection
> for client /0:0:0:0:0:0:0:1:50355 which had sessionid 0x1571bd1c9c50003
> 224159 [main-EventThread] INFO  org.apache.storm.zookeeper.ClientCnxn -
> EventThread shut down
> 224163 [ProcessThread(sid:0 cport:-1):] INFO
> org.apache.storm.zookeeper.server.PrepRequestProcessor
> - Processed session termination for sessionid: 0x1571bd1c9c50005
> 224164 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO
>  org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket
> connection
> for client /127.0.0.1:49569 which had sessionid 0x1571bd1c9c50005
> 224165 [main-EventThread] INFO  org.apache.storm.zookeeper.ClientCnxn -
> EventThread shut down
> 224165 [main] INFO  org.apache.storm.zookeeper.ZooKeeper - Session:
> 0x1571bd1c9c50005 closed
> 224166 [main] INFO  backtype.storm.daemon.supervisor - Shutting down
> 9957d6b3-05c3-46d4-bb6f-c37751a40026:37ed88ae-36f0-4b87-b128-cd16c1fa21f7
> 224166 [main] INFO  backtype.storm.process-simulator - Killing process
> 3a39d80e-10b5-4db7-823d-a48ca5d85109
> 224166 [main] INFO  backtype.storm.daemon.worker - Shutting down worker
> Prequential_20160911213510-1-1473640527 9957d6b3-05c3-46d4-bb6f-
> c37751a40026
> 1024
> 224167 [main] INFO  backtype.storm.daemon.worker - Shutting down receive
> thread
> 224167 [main] INFO  backtype.storm.messaging.loader - Shutting down
> receiving-thread: [Prequential_20160911213510-1-1473640527, 1024]
> 224167 [main] INFO  backtype.storm.messaging.loader - Waiting for
> receiving-thread:[Prequential_20160911213510-1-1473640527, 1024] to die
> 224167 [Thread-10-worker-receiver-thread-0] INFO
> backtype.storm.messaging.loader
> - Receiving-thread:[Prequential_20160911213510-1-1473640527, 1024]
> received
> shutdown notice
> 224168 [main] INFO  backtype.storm.messaging.loader - Shutdown
> receiving-thread: [Prequential_20160911213510-1-1473640527, 1024]
> 224168 [main] INFO  backtype.storm.daemon.worker - Shut down receive thread
> 224169 [main] INFO  backtype.storm.daemon.worker - Terminating messaging
> context
> 224169 [main] INFO  backtype.storm.daemon.worker - Shutting down executors
> 224169 [main] INFO  backtype.storm.daemon.executor - Shutting down executor
> org.apache.samoa.evaluation.EvaluatorProcessor_1:[2 2]
> 224169 [Thread-12-org.apache.samoa.evaluation.EvaluatorProcessor_1] INFO
>  backtype.storm.util - Async loop interrupted!
> 224170 [Thread-11-disruptor-executor[2 2]-send-queue] INFO
>  backtype.storm.util - Async loop interrupted!
> 224170 [main] INFO  backtype.storm.daemon.executor - Shut down executor
> org.apache.samoa.evaluation.EvaluatorProcessor_1:[2 2]
> 224170 [main] INFO  backtype.storm.daemon.executor - Shutting down executor
> org.apache.samoa.learners.classifiers.ensemble.
> BaggingDistributorProcessor_1:[3 3]
> ------------------------------------------------------------
> --------------------------------------------------------
> What can it be?
>
> Regards,
> Eduardo Costa.
>



-- 
Nicolas Kourtellis

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message